{ "info": { "author": "Jason Antman", "author_email": "jason@jasonantman.com", "bugtrack_url": null, "classifiers": [ "Development Status :: 3 - Alpha", "Environment :: Console", "Intended Audience :: Developers", "License :: OSI Approved :: GNU Affero General Public License v3 or later (AGPLv3+)", "Natural Language :: English", "Operating System :: OS Independent", "Programming Language :: Python", "Programming Language :: Python :: 2.7", "Programming Language :: Python :: 3", "Topic :: Internet :: Log Analysis", "Topic :: Software Development", "Topic :: Utilities" ], "description": "pypi-download-stats\n========================\n\n.. image:: https://img.shields.io/github/forks/jantman/pypi-download-stats.svg\n :alt: GitHub Forks\n :target: https://github.com/jantman/pypi-download-stats/network\n\n.. image:: https://img.shields.io/github/issues/jantman/pypi-download-stats.svg\n :alt: GitHub Open Issues\n :target: https://github.com/jantman/pypi-download-stats/issues\n\n.. image:: https://landscape.io/github/jantman/pypi-download-stats/master/landscape.svg\n :target: https://landscape.io/github/jantman/pypi-download-stats/master\n :alt: Code Health\n\n.. image:: https://secure.travis-ci.org/jantman/pypi-download-stats.png?branch=master\n :target: http://travis-ci.org/jantman/pypi-download-stats\n :alt: travis-ci for master branch\n\n.. image:: https://codecov.io/github/jantman/pypi-download-stats/coverage.svg?branch=master\n :target: https://codecov.io/github/jantman/pypi-download-stats?branch=master\n :alt: coverage report for master branch\n\n.. image:: https://readthedocs.org/projects/pypi-download-stats/badge/?version=latest\n :target: https://readthedocs.org/projects/pypi-download-stats/?badge=latest\n :alt: sphinx documentation for latest release\n\n.. image:: http://www.repostatus.org/badges/latest/active.svg\n :alt: Project Status: Active - The project has reached a stable, usable state and is being actively developed.\n :target: http://www.repostatus.org/#active\n\nIntroduction\n------------\n\nThis package retrieves download statistics from Google BigQuery for one or more\n`PyPI `_ packages, caches them locally, and then\ngenerates download count badges as well as an HTML page of raw data and graphs\n(generated by `bokeh `_ ). It's intended to\nbe run on a schedule (i.e. daily) and have the results uploaded somewhere.\n\nIt would certainly be nice to make this into a real service (and some extension\npoints for that have been included), but at the moment\nI have neither the time to dedicate to that, the money to cover some sort\nof hosting and bandwidth, nor the desire to handle how to architect this for\nover 85,000 projects as opposed to my few.\n\nHopefully stats like these will eventually end up in the official PyPI; see\nwarehouse `#699 `_,\n`#188 `_ and\n`#787 `_ for reference on that work.\nFor the time being, I want to (a) give myself a way to get simple download stats\nand badges like the old PyPI legacy (downloads per day, week and month) as well\nas (b) enable some higher-granularity analysis.\n\n**Note** this package is *very* young; I wrote it as an evening/weekend project,\nhoping to only take a few days on it. Though writing this makes me want to bathe\nimmediately, it has no tests. If people start using it, I'll change that.\n\nFor a live example of exactly how the output looks, you can see the download\nstats page for my awslimitchecker project, generated by a cronjob on my desktop,\nat: `http://jantman-personal-public.s3-website-us-east-1.amazonaws.com/pypi-stats/awslimitchecker/index.html `_.\n\nBackground\n----------\n\nSometime in February 2016, `download stats `_\nstopped working on pypi.python.org. As I later learned, what we currently (August 2016)\nknow as pypi is really the `pypi-legacy `_ codebase,\nand is far from a stable hands-off service. The `small team of interpid souls `_\nwho keep it running have their hands full simply keeping it online, while also working\non its replacement, `warehouse `_ (which as of August 2016 is available online\nat `https://pypi.io/ `_). While the actual pypi.python.org web UI hasn't been\nswitched over to the warehouse code yet (it's still under development), the current Warehouse\nservice does provide full access to pypi. It's completely understandable that, given all this\nand the \"life support\" status of the legacy pypi codebase, download stats in a legacy codebase\nare their last concern.\n\nHowever, current download statistics (actually the raw log information) since January 22, 2016\nare `available in a Google BigQuery public dataset `_\nand being updated in near-real-time. There may be download statistics functionality\n\nRequirements\n------------\n\n* Python 2.7+ (currently tested with 2.7, 3.2, 3.3, 3.4)\n* Python `VirtualEnv `_ and ``pip`` (recommended installation method; your OS/distribution should have packages for these)\n\npypi-download-stats relies on `bokeh `_ to generate\npretty SVG charts that work offline, and\n`google-api-python-client `_\nfor querying BigQuery. Each of those have additional dependencies.\n\nInstallation\n------------\n\nIt's recommended that you install into a virtual environment (virtualenv /\nvenv). See the `virtualenv usage documentation `_\nfor information on how to create a venv.\n\nThis isn't on pypi yet, ironically. Until it is:\n\n.. code-block:: bash\n\n $ pip install git+https://github.com/jantman/pypi-download-stats.git\n\nConfiguration\n-------------\n\nYou'll need Google Cloud credentials for a project that has the BigQuery API\nenabled. The recommended method is to generate system account credentials;\ndownload the JSON file for the credentials and export the path to it as the\n``GOOGLE_APPLICATION_CREDENTIALS`` environment variable. The system account\nwill need to be added as a Project Member.\n\nUsage\n-----\n\nRun with ``-h`` for command-line help::\n\n usage: pypi-download-stats [-h] [-V] [-v] [-Q | -G] [-o OUT_DIR]\n [-p PROJECT_ID] [-c CACHE_DIR] [-B BACKFILL_DAYS]\n [-P PROJECT | -U USER]\n\n pypi-download-stats - Calculate detailed download stats and generate HTML and\n badges for PyPI packages - \n\n optional arguments:\n -h, --help show this help message and exit\n -V, --version show program's version number and exit\n -v, --verbose verbose output. specify twice for debug-level output.\n -Q, --no-query do not query; just generate output from cached data\n -G, --no-generate do not generate output; just query data and cache\n results\n -o OUT_DIR, --out-dir OUT_DIR\n output directory (default: ./pypi-stats\n -p PROJECT_ID, --project-id PROJECT_ID\n ProjectID for your Google Cloud user, if not using\n service account credentials JSON file\n -c CACHE_DIR, --cache-dir CACHE_DIR\n stats cache directory (default: ./pypi-stats-cache)\n -B BACKFILL_DAYS, --backfill-num-days BACKFILL_DAYS\n number of days of historical data to backfill, if\n missing (defaut: 7). Note this may incur BigQuery\n charges. Set to -1 to backfill all available history.\n -P PROJECT, --project PROJECT\n project name to query/generate stats for (can be\n specified more than once; this will reduce query cost\n for multiple projects)\n -U USER, --user USER Run for all PyPI projects owned by the specifieduser.\n\nTo run queries and generate reports for PyPI projects \"foo\" and \"bar\", using a\nGoogle Cloud credentials JSON file at ``foo.json``:\n\n.. code-block:: bash\n\n $ export GOOGLE_APPLICATION_CREDENTIALS=/foo.json\n $ pypi-download-stats -P foo -P bar\n\nTo run queries but *not* generate reports for all PyPI projects owned by user\n\"myname\":\n\n.. code-block:: bash\n\n $ export GOOGLE_APPLICATION_CREDENTIALS=/foo.json\n $ pypi-download-stats -G -U myname\n\nTo generate reports against cached query data for the project \"foo\":\n\n.. code-block:: bash\n\n $ export GOOGLE_APPLICATION_CREDENTIALS=/foo.json\n $ pypi-download-stats -Q -P foo\n\nTo run nightly and upload results to a website-hosting S3 bucket, I use the\nfollowing script via cron (note the paths are specific to my purpose; also note\nthe two commands, as ``s3cmd`` does not seem to set the MIME type for the SVG\nimages correctly):\n\n.. code-block:: bash\n\n #!/bin/bash -x\n\n export GOOGLE_APPLICATION_CREDENTIALS=/home/jantman/.ssh/pypi-bigquery.json\n cd /home/jantman/GIT/pypi-download-stats\n bin/pypi-download-stats -vv -U jantman\n\n # sync html files\n ~/venvs/foo/bin/s3cmd -r --delete-removed --stats --exclude='*.svg' sync pypi-stats s3://jantman-personal-public/\n # sync SVG and set mime-type, since s3cmd gets it wrong\n ~/venvs/foo/bin/s3cmd -r --delete-removed --stats --exclude='*.html' --mime-type='image/svg+xml' sync pypi-stats s3://jantman-personal-public/\n\nCost\n++++\n\nAt this point... I have no idea. Some of the download tables are 3+ GB per day.\nI imagine that backfilling historical data from the beginning of what's currently\nthere (20160122) might incur quite a bit of data cost.\n\nBugs and Feature Requests\n-------------------------\n\nBug reports and feature requests are happily accepted via the `GitHub Issue Tracker `_. Pull requests are\nwelcome. Issues that don't have an accompanying pull request will be worked on\nas my time and priority allows.\n\nDevelopment\n===========\n\nTo install for development:\n\n1. Fork the `pypi-download-stats `_ repository on GitHub\n2. Create a new branch off of master in your fork.\n\n.. code-block:: bash\n\n $ virtualenv pypi-download-stats\n $ cd pypi-download-stats && source bin/activate\n $ pip install -e git+git@github.com:YOURNAME/pypi-download-stats.git@BRANCHNAME#egg=pypi-download-stats\n $ cd src/pypi-download-stats\n\nThe git clone you're now in will probably be checked out to a specific commit,\nso you may want to ``git checkout BRANCHNAME``.\n\nGuidelines\n----------\n\n* pep8 compliant with some exceptions (see pytest.ini)\n\nTesting\n-------\n\nThere isn't any right now. I'm bad. If people actually start using this, I'll\nrefactor and add tests, but for now this started as a one-night project.\n\nRelease Checklist\n-----------------\n\n1. Open an issue for the release; cut a branch off master for that issue.\n2. Confirm that there are CHANGES.rst entries for all major changes.\n3. Ensure that Travis tests passing in all environments.\n4. Ensure that test coverage is no less than the last release (ideally, 100%).\n5. Increment the version number in pypi-download-stats/version.py and add version and release date to CHANGES.rst, then push to GitHub.\n6. Confirm that README.rst renders correctly on GitHub.\n7. Upload package to testpypi:\n\n * Make sure your ~/.pypirc file is correct (a repo called ``test`` for https://testpypi.python.org/pypi)\n * ``rm -Rf dist``\n * ``python setup.py register -r https://testpypi.python.org/pypi``\n * ``python setup.py sdist bdist_wheel``\n * ``twine upload -r test dist/*``\n * Check that the README renders at https://testpypi.python.org/pypi/pypi-download-stats\n\n8. Create a pull request for the release to be merged into master. Upon successful Travis build, merge it.\n9. Tag the release in Git, push tag to GitHub:\n\n * tag the release. for now the message is quite simple: ``git tag -a X.Y.Z -m 'X.Y.Z released YYYY-MM-DD'``\n * push the tag to GitHub: ``git push origin X.Y.Z``\n\n11. Upload package to live pypi:\n\n * ``twine upload dist/*``\n\n10. make sure any GH issues fixed in the release were closed.\n\n\n", "description_content_type": null, "docs_url": null, "download_url": "", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "https://github.com/jantman/pypi-download-stats", "keywords": "pypi warehouse download stats badge", "license": "", "maintainer": "", "maintainer_email": "", "name": "pypi-download-stats", "package_url": "https://pypi.org/project/pypi-download-stats/", "platform": "", "project_url": "https://pypi.org/project/pypi-download-stats/", "project_urls": { "Homepage": "https://github.com/jantman/pypi-download-stats" }, "release_url": "https://pypi.org/project/pypi-download-stats/0.2.1/", "requires_dist": [ "iso3166", "bokeh (==0.12.1)", "google-api-python-client (>=1.5.0)", "oauth2client (>=3.0.0)", "pandas (<1.0,>=0.18)", "pytz", "requests (<3.0,>2.0)", "tzlocal" ], "requires_python": "", "summary": "Calculate detailed download stats and generate HTML and badges for PyPI packages", "version": "0.2.1" }, "last_serial": 2349338, "releases": { "0.1.0": [ { "comment_text": "", "digests": { "md5": "61fa2f935c04ba2f40bbe744e97fc866", "sha256": "ac115391b957403d0c7ec3ccaf373a214e893db8e3313453117eae7601fd8856" }, "downloads": -1, "filename": "pypi_download_stats-0.1.0-py2.py3-none-any.whl", "has_sig": false, "md5_digest": "61fa2f935c04ba2f40bbe744e97fc866", "packagetype": "bdist_wheel", "python_version": "py2.py3", "requires_python": null, "size": 40137, "upload_time": "2016-09-02T21:13:31", "url": "https://files.pythonhosted.org/packages/13/e7/712bc8ca27c11848d42b3d3478eb5fc76f3532df860433e8d289e627d893/pypi_download_stats-0.1.0-py2.py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "65aafaaac3064ae3a56e18c63b9fbbd7", "sha256": "39d86e147348d3fed4d0dc822845eebd40069f4b45ed22f9c9343b28b3eb0a31" }, "downloads": -1, "filename": "pypi-download-stats-0.1.0.tar.gz", "has_sig": false, "md5_digest": "65aafaaac3064ae3a56e18c63b9fbbd7", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 40100, "upload_time": "2016-09-02T21:13:35", "url": "https://files.pythonhosted.org/packages/8f/ba/475e19f8e4c544ba291ce076eb2442de5d8a70cfd517f39c0984436861af/pypi-download-stats-0.1.0.tar.gz" } ], "0.2.0": [ { "comment_text": "", "digests": { "md5": "0ccd01b1a9e2ff1474b11195b14cbd0e", "sha256": "8f76d86c17c0a8200b3600b07027d0cedf3e2ba82aeb8fcd3a9e2b5f1c4fe926" }, "downloads": -1, "filename": "pypi_download_stats-0.2.0-py2.py3-none-any.whl", "has_sig": false, "md5_digest": "0ccd01b1a9e2ff1474b11195b14cbd0e", "packagetype": "bdist_wheel", "python_version": "py2.py3", "requires_python": null, "size": 44559, "upload_time": "2016-09-18T11:15:38", "url": "https://files.pythonhosted.org/packages/05/0f/02b2fd88f78f1bdfd183aad3aea49a1ad155110fa2a664224dc38293d0ee/pypi_download_stats-0.2.0-py2.py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "72dda801ec714d634da349391e844c5f", "sha256": "c47f10b8cb782341b5cca6706260492d20ac144e16b7f56efef399cbdd697cce" }, "downloads": -1, "filename": "pypi-download-stats-0.2.0.tar.gz", "has_sig": false, "md5_digest": "72dda801ec714d634da349391e844c5f", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 43743, "upload_time": "2016-09-18T11:15:40", "url": "https://files.pythonhosted.org/packages/61/74/d5aa85a42611c27d9eb0c2d2d13c9b19f4ceebb8f99b434b41b8490791d4/pypi-download-stats-0.2.0.tar.gz" } ], "0.2.1": [ { "comment_text": "", "digests": { "md5": "77641092ef870155fac851505d3bca12", "sha256": "bdc2ae538899710d694be604128e746d9f582a94af57751a46c4a207f9c936c1" }, "downloads": -1, "filename": "pypi_download_stats-0.2.1-py2.py3-none-any.whl", "has_sig": false, "md5_digest": "77641092ef870155fac851505d3bca12", "packagetype": "bdist_wheel", "python_version": "py2.py3", "requires_python": null, "size": 44854, "upload_time": "2016-09-18T16:32:03", "url": "https://files.pythonhosted.org/packages/6e/5a/e3ab4a74fa9f32cf876cbb380500727008f963769ece8f165918f4e2fb65/pypi_download_stats-0.2.1-py2.py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "a1f1edfa738c463ccb9e910fa13631ea", "sha256": "ad625ffcff140374a9467dc0ea1a00947271433cc43130fb294336f5dd7f40a2" }, "downloads": -1, "filename": "pypi-download-stats-0.2.1.tar.gz", "has_sig": false, "md5_digest": "a1f1edfa738c463ccb9e910fa13631ea", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 44074, "upload_time": "2016-09-18T16:32:05", "url": "https://files.pythonhosted.org/packages/15/1c/4650876b03388ad2fc9a530036e820974b36a0f601a04eef68318c39a1f5/pypi-download-stats-0.2.1.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "77641092ef870155fac851505d3bca12", "sha256": "bdc2ae538899710d694be604128e746d9f582a94af57751a46c4a207f9c936c1" }, "downloads": -1, "filename": "pypi_download_stats-0.2.1-py2.py3-none-any.whl", "has_sig": false, "md5_digest": "77641092ef870155fac851505d3bca12", "packagetype": "bdist_wheel", "python_version": "py2.py3", "requires_python": null, "size": 44854, "upload_time": "2016-09-18T16:32:03", "url": "https://files.pythonhosted.org/packages/6e/5a/e3ab4a74fa9f32cf876cbb380500727008f963769ece8f165918f4e2fb65/pypi_download_stats-0.2.1-py2.py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "a1f1edfa738c463ccb9e910fa13631ea", "sha256": "ad625ffcff140374a9467dc0ea1a00947271433cc43130fb294336f5dd7f40a2" }, "downloads": -1, "filename": "pypi-download-stats-0.2.1.tar.gz", "has_sig": false, "md5_digest": "a1f1edfa738c463ccb9e910fa13631ea", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 44074, "upload_time": "2016-09-18T16:32:05", "url": "https://files.pythonhosted.org/packages/15/1c/4650876b03388ad2fc9a530036e820974b36a0f601a04eef68318c39a1f5/pypi-download-stats-0.2.1.tar.gz" } ] }