{
    "info": {
        "author": "Software Heritage developers",
        "author_email": "swh-devel@inria.fr",
        "bugtrack_url": null,
        "classifiers": [
            "Development Status :: 5 - Production/Stable",
            "Intended Audience :: Developers",
            "License :: OSI Approved :: GNU General Public License v3 (GPLv3)",
            "Operating System :: OS Independent",
            "Programming Language :: Python :: 3"
        ],
        "description": "swh-loader-pypi\n====================\n\nSWH PyPI loader's source code repository\n\n# What does the loader do?\n\nThe PyPI loader visits and loads a PyPI project [1].\n\nEach visit will result in:\n- 1 snapshot (which targets n revisions ; 1 per release artifact)\n- 1 revision (which targets 1 directory ; the release artifact uncompressed)\n\n[1] https://pypi.org/help/#packages\n\n## First visit\n\nGiven a PyPI project (origin), the loader, for the first visit:\n\n- retrieves information for the given project (including releases)\n- then for each associated release\n- for each associated source distribution (type 'sdist') release\n  artifact (possibly many per release)\n- retrieves the associated artifact archive (with checks)\n- uncompresses locally the archive\n- computes the hashes of the uncompressed directory\n- then creates a revision (using PKG-INFO metadata file) targeting\n  such directory\n- finally, creates a snapshot targeting all seen revisions\n  (uncompressed PyPI artifact and metadata).\n\n## Next visit\n\nThe loader starts by checking if something changed since the last\nvisit.  If nothing changed, the visit's snapshot is left\nunchanged. The new visit targets the same snapshot.\n\nIf something changed, the already seen release artifacts are skipped.\nOnly the new ones are loaded. In the end, the loader creates a new\nsnapshot based on the previous one. Thus, the new snapshot targets\nboth the old and new PyPI release artifacts.\n\n## Terminology\n\n- 1 project: a PyPI project (used as swh origin). This is a collection\n             of releases.\n\n- 1 release: a specific version of the (PyPi) project. It's a\n             collection of information and associated source release\n             artifacts (type 'sdist')\n\n- 1 release artifact: a source release artifact (distributed by a PyPI\n                      maintainer). In swh, we are specifically\n                      interested by the 'sdist' type (source code).\n\n## Edge cases\n\n- If no release provides release artifacts, those are skipped\n\n- If a release artifact holds no PKG-INFO file (root at the archive),\n  the release artifact is skipped.\n\n- If a problem occurs during a fetch action (e.g. release artifact\n  download), the load fails and the visit is marked as 'partial'.\n\n# Development\n\n## Configuration file\n\n### Location\n\nEither:\n- /etc/softwareheritage/\n- ~/.config/swh/\n- ~/.swh/\n\nNote: Will call that location $SWH_CONFIG_PATH\n\n### Configuration sample\n\n$SWH_CONFIG_PATH/loader/pypi.yml:\n```\nstorage:\n  cls: remote\n  args:\n    url: http://localhost:5002/\n\n```\n\n## Local run\n\nThe built-in command-line will run the loader for a project in the\nmain PyPI archive.\n\nFor instance, to load arrow:\n``` sh\npython3 -m swh.loader.pypi.loader arrow\n```\n\nIf you need more control, you can use the loader directly. It expects\nthree arguments:\n- project: a PyPI project name (f.e.: arrow)\n- project_url: URL of the PyPI project (human-readable html page)\n- project_metadata_url: URL of the PyPI metadata information\n  (machine-parsable json document)\n\n``` python\nimport logging\nlogging.basicConfig(level=logging.DEBUG)\n\nfrom swh.loader.pypi.tasks import LoadPyPI\n\nproject='arrow'\n\nLoadPyPI().run(project, 'https://pypi.org/pypi/%s/' % project, 'https://pypi.org/pypi/%s/json' % project)\n```\n\n\n",
        "description_content_type": "text/markdown",
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://forge.softwareheritage.org/source/swh-loader-pypi",
        "keywords": "",
        "license": "",
        "maintainer": "",
        "maintainer_email": "",
        "name": "swh.loader.pypi",
        "package_url": "https://pypi.org/project/swh.loader.pypi/",
        "platform": "",
        "project_url": "https://pypi.org/project/swh.loader.pypi/",
        "project_urls": {
            "Bug Reports": "https://forge.softwareheritage.org/maniphest",
            "Funding": "https://www.softwareheritage.org/donate",
            "Homepage": "https://forge.softwareheritage.org/source/swh-loader-pypi",
            "Source": "https://forge.softwareheritage.org/source/swh-loader-pypi"
        },
        "release_url": "https://pypi.org/project/swh.loader.pypi/0.0.9/",
        "requires_dist": [
            "setuptools",
            "vcversioner",
            "requests",
            "arrow",
            "pkginfo",
            "swh.core",
            "swh.model (>=0.0.28)",
            "swh.storage (>=0.0.114)",
            "swh.scheduler (>=0.0.39)",
            "swh.loader.core (>=0.0.37)",
            "pytest ; extra == 'testing'"
        ],
        "requires_python": "",
        "summary": "Software Heritage PyPI Loader",
        "version": "0.0.9"
    },
    "last_serial": 4791921,
    "releases": {
        "0.0.3": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "668ce6d126cf9cfb48fd09109a6bedc8",
                    "sha256": "529fb32888705286f8d42718adc07312e67fcfa72570887de458a79f81fb87ed"
                },
                "downloads": -1,
                "filename": "swh.loader.pypi-0.0.3-py3-none-any.whl",
                "has_sig": true,
                "md5_digest": "668ce6d126cf9cfb48fd09109a6bedc8",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 32801,
                "upload_time": "2018-10-18T09:26:21",
                "url": "https://files.pythonhosted.org/packages/0f/6b/93d94d71abc2049fc5cb55a154563e3358bc1631f85a551518789b042014/swh.loader.pypi-0.0.3-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "7b96e50c0f8113d4dbad1fd78d04d9a3",
                    "sha256": "8142857b86adc63ba99d98752012a31d0f18d032135f2f2f0a87ea1bb158d320"
                },
                "downloads": -1,
                "filename": "swh.loader.pypi-0.0.3.tar.gz",
                "has_sig": true,
                "md5_digest": "7b96e50c0f8113d4dbad1fd78d04d9a3",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 17131,
                "upload_time": "2018-10-18T09:26:23",
                "url": "https://files.pythonhosted.org/packages/60/14/0a8a351a15fb0badbbaff7f400e1e62f88fdfae8a0ca08b0548997cadee7/swh.loader.pypi-0.0.3.tar.gz"
            }
        ],
        "0.0.4": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "30403b1e9061d3ba7a8da72c21f0f2bf",
                    "sha256": "c1cbf7053d890b8370484d03a93f15f133c00e2c312d1139f3633ea39c00275c"
                },
                "downloads": -1,
                "filename": "swh.loader.pypi-0.0.4-py3-none-any.whl",
                "has_sig": true,
                "md5_digest": "30403b1e9061d3ba7a8da72c21f0f2bf",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 32858,
                "upload_time": "2018-10-23T17:19:38",
                "url": "https://files.pythonhosted.org/packages/35/47/ff2ddad66c19f8ab64cf4e26f8640ab75d15b9638275c605c56f3a974146/swh.loader.pypi-0.0.4-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "7f243a37421b727d5e1cd2d589d36680",
                    "sha256": "48c94b6d66b10a8f81bae5b7a7e6f6b8e98b32a9e4a63b4f500e3ba5ce74d4b7"
                },
                "downloads": -1,
                "filename": "swh.loader.pypi-0.0.4.tar.gz",
                "has_sig": true,
                "md5_digest": "7f243a37421b727d5e1cd2d589d36680",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 17151,
                "upload_time": "2018-10-23T17:19:39",
                "url": "https://files.pythonhosted.org/packages/f3/d5/03996f8d417894af97adefb0de95045073aab6826f022786d9b72942c7f0/swh.loader.pypi-0.0.4.tar.gz"
            }
        ],
        "0.0.6": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "052a90f2e8b4ddf2de3f22a7501506b5",
                    "sha256": "d894d6eea8bea98599eef2d14d138ea63ed3abe9d8755e78b117fd59b0effa49"
                },
                "downloads": -1,
                "filename": "swh.loader.pypi-0.0.6-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "052a90f2e8b4ddf2de3f22a7501506b5",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 32907,
                "upload_time": "2018-11-30T08:58:42",
                "url": "https://files.pythonhosted.org/packages/88/17/dab1bd786d264dc6a5bd32f4be83f56c1e2623eb8513decfc7086f4474ee/swh.loader.pypi-0.0.6-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "88af9836254c9121b7768a551e6fde1f",
                    "sha256": "b12a2d81a3b0158972d80b16f9d1250f6a93979632af4805ad2f10fc182b6e94"
                },
                "downloads": -1,
                "filename": "swh.loader.pypi-0.0.6.tar.gz",
                "has_sig": false,
                "md5_digest": "88af9836254c9121b7768a551e6fde1f",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 16298,
                "upload_time": "2018-11-30T08:58:43",
                "url": "https://files.pythonhosted.org/packages/dd/8f/a1ec2d4c925c74e103f6c35a04638d6f93980e4070ee3df1470738459fa2/swh.loader.pypi-0.0.6.tar.gz"
            }
        ],
        "0.0.7": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "5d4ff98897801c0f9e5a2a7fa355b693",
                    "sha256": "4624a72f629dd847356974fd40b0b82d079e9e1f6e5ba7c691442644bca645b0"
                },
                "downloads": -1,
                "filename": "swh.loader.pypi-0.0.7-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "5d4ff98897801c0f9e5a2a7fa355b693",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 32914,
                "upload_time": "2018-11-30T10:03:06",
                "url": "https://files.pythonhosted.org/packages/f3/92/1e76342e7c3cde8c576eb7db1f2d63ef8f7acabe04a35a9ec124619d7a95/swh.loader.pypi-0.0.7-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "7d8e3ec6edac8ba5dd1252d0568da04f",
                    "sha256": "ca5f08dbd57821eb5aa8dd5fddea7f7dae4791a6bb855ea056260d861f24255f"
                },
                "downloads": -1,
                "filename": "swh.loader.pypi-0.0.7.tar.gz",
                "has_sig": false,
                "md5_digest": "7d8e3ec6edac8ba5dd1252d0568da04f",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 16309,
                "upload_time": "2018-11-30T10:03:08",
                "url": "https://files.pythonhosted.org/packages/8b/61/6d4f78a59a1d526cc990a7a2c4bd0d07c1cd5de851061069c33335501904/swh.loader.pypi-0.0.7.tar.gz"
            }
        ],
        "0.0.8": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "b92286c5cc53434270405a6890f24bed",
                    "sha256": "c6d912fd24d3879090070ab0c314fe99b388f27a3c907543bdaf9cc5eab1fbf1"
                },
                "downloads": -1,
                "filename": "swh.loader.pypi-0.0.8-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "b92286c5cc53434270405a6890f24bed",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 33167,
                "upload_time": "2018-12-20T10:42:04",
                "url": "https://files.pythonhosted.org/packages/d7/db/48f6bc4449ebdcc538659d0fc2a60e9c6c178a3fc75a446ab681d5058926/swh.loader.pypi-0.0.8-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "32eff0516320a77dd116e88a902fe790",
                    "sha256": "acb270895b4e163c0dc2aa04e8548b7e69a7bb93f4d42d58d0764fc7cd75a89c"
                },
                "downloads": -1,
                "filename": "swh.loader.pypi-0.0.8.tar.gz",
                "has_sig": false,
                "md5_digest": "32eff0516320a77dd116e88a902fe790",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 16543,
                "upload_time": "2018-12-20T10:42:05",
                "url": "https://files.pythonhosted.org/packages/7c/42/e5bb80c27c08b603ef178569f4320f5baf1a20683113af73a5c8522e90ba/swh.loader.pypi-0.0.8.tar.gz"
            }
        ],
        "0.0.9": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "534e162fe9de1a1f64c8d9142b5fd34e",
                    "sha256": "f750f3ce75de9d340d5c57cad16db06db47e8f8745f5f7cd381dd3a8df0dd989"
                },
                "downloads": -1,
                "filename": "swh.loader.pypi-0.0.9-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "534e162fe9de1a1f64c8d9142b5fd34e",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 33142,
                "upload_time": "2019-02-07T16:37:49",
                "url": "https://files.pythonhosted.org/packages/a6/f1/66f5c5dd8dc5be1377deaf26883546ad3a404742e5cdb5c9345d9b11bd1f/swh.loader.pypi-0.0.9-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "6ebaba24d1418ab9501308ad8a859817",
                    "sha256": "4c6e71fb287fddc92bf59b1698dad1b402ca5ac11c30507527e9252883265747"
                },
                "downloads": -1,
                "filename": "swh.loader.pypi-0.0.9.tar.gz",
                "has_sig": false,
                "md5_digest": "6ebaba24d1418ab9501308ad8a859817",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 16534,
                "upload_time": "2019-02-07T16:37:50",
                "url": "https://files.pythonhosted.org/packages/ba/1f/eb3fccc001ab581fe0edb3540a687a253051dbba7cb9f200cfb72eea1b91/swh.loader.pypi-0.0.9.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "534e162fe9de1a1f64c8d9142b5fd34e",
                "sha256": "f750f3ce75de9d340d5c57cad16db06db47e8f8745f5f7cd381dd3a8df0dd989"
            },
            "downloads": -1,
            "filename": "swh.loader.pypi-0.0.9-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "534e162fe9de1a1f64c8d9142b5fd34e",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": null,
            "size": 33142,
            "upload_time": "2019-02-07T16:37:49",
            "url": "https://files.pythonhosted.org/packages/a6/f1/66f5c5dd8dc5be1377deaf26883546ad3a404742e5cdb5c9345d9b11bd1f/swh.loader.pypi-0.0.9-py3-none-any.whl"
        },
        {
            "comment_text": "",
            "digests": {
                "md5": "6ebaba24d1418ab9501308ad8a859817",
                "sha256": "4c6e71fb287fddc92bf59b1698dad1b402ca5ac11c30507527e9252883265747"
            },
            "downloads": -1,
            "filename": "swh.loader.pypi-0.0.9.tar.gz",
            "has_sig": false,
            "md5_digest": "6ebaba24d1418ab9501308ad8a859817",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 16534,
            "upload_time": "2019-02-07T16:37:50",
            "url": "https://files.pythonhosted.org/packages/ba/1f/eb3fccc001ab581fe0edb3540a687a253051dbba7cb9f200cfb72eea1b91/swh.loader.pypi-0.0.9.tar.gz"
        }
    ]
}