{ "info": { "author": "Ralph Bean", "author_email": "rbean@redhat.com", "bugtrack_url": null, "classifiers": [ "Environment :: Web Environment", "Intended Audience :: Developers", "Programming Language :: Python", "Topic :: Software Development :: Libraries :: Python Modules" ], "description": "A daemon to build and keep fedmsg statistics.\n\n**Motivation**: we have a neat service called `datagrepper\n`_ with which you can query the\nhistory of the `fedmsg `_ bus. It is cool, but insufficient\nfor some more advanced reporting and analysis that we would like to do. Take,\nfor example, the `releng-dash `_.\nIn order to render the page, it has to make a dozen or more requests to\ndatagrepper to try and find the 'latest' events from large awkward pages of\nresults. Consequently, it takes a long time to load.\n\nEnter, statscache. It is a plugin to the fedmsg-hub that sits listening in our\ninfrastructure. When new messages arrive, it will pass them off to `plugins\n`_ that will calculate and\nstore various statistics. If we want a new kind of statistic to be kept, we\nwrite a new plugin for it. It will come with a tiny flask frontend, much like\ndatagrepper, that allows you to query for this or that stat in this or that\nformat (csv, json, maybe html or svg too but that might be overkill). The idea\nbeing that we can then build neater smarter frontends that can render\nfedmsg-based activity very quickly.. and perhaps later drill-down into the\n*details* kept in datagrepper.\n\nIt is kind of like a `data mart `_.\n\nHow to run it\n-------------\n\nCreate a virtualenv, and git clone this directory and the statscache_plugins\nrepo.\n\nRun ``python setup.py develop`` in the ``statscache`` dir first and then run it\nin ``statscache_plugins``.\n\nIn the main statscache repo directory, run ``fedmsg-hub`` to start the\ndaemon. You should see lots of fun stats being stored in stdout. To launch\nthe web interface (which currently only serves JSON and CSV responses), run\n``python statscache/app.py`` in the same directory. You can now view a list of\nthe available plugins in JSON by visiting\n`localhost:5000/api/ `_, and you can retrieve the\nstatistics recorded by a given plugin by appending its identifier to that same\nURL.\n\nYou can run the tests with ``python setup.py test``.\n\nHow it works\n------------\n\nWhen a message arrives, a *fedmsg consumer* receives it and hands a copy to\neach loaded plugin for processing. Each plugin internally caches the results\nof this message processing until a *polling producer* instructs it to update\nits database model and empty its cache. The frequency at which the polling\nproducer does so is configurable at the application level and is set to one\nsecond by default.\n\nThere are base sqlalchemy models that each of the plugins should use to store\ntheir results (and we can add more types of base models as we discover new\nneeds). But the important thing to know about the base models is that they are\nresponsible for knowing how to serialize themselves to different formats for\nthe REST API (e.g., render ``.to_csv()`` and ``.to_json()``).\n\nEven though statscache is intended to be a long-running service, the occasional\nreboot is inevitable. However, having breaks in the processed history of\nfedmsg data may lead some plugins to produce inaccurate statistics. Luckily,\nstatscache comes built-in with a mechanism to transparently handle this. On\nstart-up, statscache checks the timestamp of each plugin's most recent database\nupdate and queries datagrepper for the fedmsg data needed to fill in any gaps.\nOn the other hand, if a plugin specifically does not need a continuous view of\nthe fedmsg history, then it may specify a \"backlog delta,\" which is the\nmaximum backlog of fedmsg data that would be useful to it.", "description_content_type": null, "docs_url": null, "download_url": "https://pypi.python.org/pypi/statscache/", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "https://github.com/fedora-infra/statscache/", "keywords": null, "license": "LGPLv2+", "maintainer": null, "maintainer_email": null, "name": "statscache", "package_url": "https://pypi.org/project/statscache/", "platform": "UNKNOWN", "project_url": "https://pypi.org/project/statscache/", "project_urls": { "Download": "https://pypi.python.org/pypi/statscache/", "Homepage": "https://github.com/fedora-infra/statscache/" }, "release_url": "https://pypi.org/project/statscache/0.0.4/", "requires_dist": null, "requires_python": null, "summary": "Daemon to build and keep fedmsg statistics", "version": "0.0.4" }, "last_serial": 2045602, "releases": { "0.0.2": [ { "comment_text": "", "digests": { "md5": "fa89e27b6cdf6e6a62f3e2828714e4ef", "sha256": "8bbff5b19be95e1c0061788a79da5bcdd2f54327d422d76303d50b8ec73f2bfe" }, "downloads": -1, "filename": "statscache-0.0.2.tar.gz", "has_sig": true, "md5_digest": "fa89e27b6cdf6e6a62f3e2828714e4ef", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 1461650, "upload_time": "2015-10-30T19:16:04", "url": "https://files.pythonhosted.org/packages/a8/0b/28f9c79b1b3fc5ab68190ed423aa18abcd4aed35e626a0d3479179afa113/statscache-0.0.2.tar.gz" } ], "0.0.3": [ { "comment_text": "", "digests": { "md5": "aab9d2b3dea95f11651653279c837b20", "sha256": "c980a3303cb9c84f901aef1a9428f2579db5806d8af986ee9d3fd3ccf6346a97" }, "downloads": -1, "filename": "statscache-0.0.3.tar.gz", "has_sig": true, "md5_digest": "aab9d2b3dea95f11651653279c837b20", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 1467423, "upload_time": "2015-10-30T19:50:30", "url": "https://files.pythonhosted.org/packages/9e/ae/ad4f281c0cd20f227a4defb9977b09a96a3c4543f7117471c548a8d366b0/statscache-0.0.3.tar.gz" } ], "0.0.4": [ { "comment_text": "", "digests": { "md5": "a3b4df109298325a2563dcb2572f9112", "sha256": "8d54b85db4807732cc9ba510cf3b6729195dc063bb126994cc951f19971575cc" }, "downloads": -1, "filename": "statscache-0.0.4.tar.gz", "has_sig": true, "md5_digest": "a3b4df109298325a2563dcb2572f9112", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 1467771, "upload_time": "2015-11-05T15:25:23", "url": "https://files.pythonhosted.org/packages/6a/42/20ef95bc8f3d23ee34b8517a06f476466b373cad44a5229e434cc62f8636/statscache-0.0.4.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "a3b4df109298325a2563dcb2572f9112", "sha256": "8d54b85db4807732cc9ba510cf3b6729195dc063bb126994cc951f19971575cc" }, "downloads": -1, "filename": "statscache-0.0.4.tar.gz", "has_sig": true, "md5_digest": "a3b4df109298325a2563dcb2572f9112", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 1467771, "upload_time": "2015-11-05T15:25:23", "url": "https://files.pythonhosted.org/packages/6a/42/20ef95bc8f3d23ee34b8517a06f476466b373cad44a5229e434cc62f8636/statscache-0.0.4.tar.gz" } ] }