{
    "info": {
        "author": "Ian Lee",
        "author_email": "lee1001@llnl.gov",
        "bugtrack_url": null,
        "classifiers": [
            "Development Status :: 4 - Beta",
            "Intended Audience :: Developers",
            "Operating System :: OS Independent",
            "Programming Language :: Python",
            "Programming Language :: Python :: 3"
        ],
        "description": "# Scraper\n\nScraper is a tool for scraping and visualizing open source data from various\ncode hosting platforms, such as: GitHub.com, GitHub Enterprise, GitLab.com,\nhosted GitLab, and Bitbucket Server.\n\n## Getting Started: Code.gov\n\n[Code.gov](https://code.gov) is a newly launched website of the US Federal\nGovernment to allow the People to access metadata from the governments custom\ndeveloped software. This site requires metadata to function, and this Python\nlibrary can help with that!\n\nTo get started, you will need a [GitHub Personal Auth\nToken](https://help.github.com/articles/creating-a-personal-access-token-for-the-command-line/)\nto make requests to the GitHub API. This should be set in your environment or\nshell ``rc`` file with the name ``GITHUB_API_TOKEN``:\n\n    $ export GITHUB_API_TOKEN=XYZ\n\n    $ echo \"export GITHUB_API_TOKEN=XYZ\" >> ~/.bashrc\n\nAdditionally, to perform the labor hours estimation, you will need to install\n``cloc`` into your environment. This is typically done with a [Package\nManager](https://github.com/AlDanial/cloc#install-via-package-manager) such as\n``npm`` or ``homebrew``.\n\nThen to generate a ``code.json`` file for your agency, you will need a\n``config.json`` file to coordinate the platforms you will connect to and scrape\ndata from. An example config file can be found in [demo.json](/demo.json). Once\nyou have your config file, you are ready to install and run the scraper!\n\n    # Install Scraper\n    $ pip install -e .\n\n    # Run Scraper with your config file ``config.json``\n    $ scraper --config config.json\n\nA full example of the resulting ``code.json`` file can be [found\nhere](https://gist.github.com/IanLee1521/b7d7c0c2d8c24b10dd04edd5e8cab6c4).\n\n## Config File Options\n\nThe configuration file is a json file that specifies what repository platforms\nto pull projects from  as well as some settings that can be used to override\nincomplete or inaccurate data returned via the scraping.\n\nThe basic structure is:\n\n```json\n{\n    # REQUIRED\n    \"contact_email\": \"...\", # Used when the contact email cannot be found otherwise\n\n    # OPTIONAL\n    \"agency\": \"...\",        # Your agency abbreviation here\n    \"organization\": \"...\",  # The organization within the agency\n    \"permissions\": { ... }, # Object containing default values for usageType and exemptionText\n\n    # Platform configurations, described in more detail below\n    \"GitHub\": [ ... ],\n    \"GitLab\": [ ... ],\n    \"Bitbucket\": [ ... ],\n}\n```\n\n```json\n\"GitHub\": [\n    {\n        \"url\": \"https://github.com\",    # GitHub.com or GitHub Enterprise URL to inventory\n        \"token\": null,                  # Private token for accessing this GitHub instance\n        \"public_only\": true,            # Only inventory public repositories\n\n        \"orgs\": [ ... ],    # List of organizations to inventory\n        \"repos\": [ ... ],   # List of single repositories to inventory\n        \"exclude\": [ ... ]  # List of organizations / repositories to exclude from inventory\n    }\n],\n```\n\n```json\n\"GitLab\": [\n    {\n        \"url\": \"https://gitlab.com\",    # GitLab.com or hosted GitLab instance URL to inventory\n        \"token\": null,                  # Private token for accessing this GitHub instance\n        \"fetch_languages\": false,       # Include individual calls to API for language metadata. Very slow, so defaults to false. (eg, for 191 projects on internal server, 5 seconds for False, 12 minutes, 38 seconds for True)\n\n        \"orgs\": [ ... ],    # List of organizations to inventory\n        \"repos\": [ ... ],   # List of single repositories to inventory\n        \"exclude\": [ ... ]  # List of groups / repositories to exclude from inventory\n    }\n]\n```\n\n```json\n\"Bitbucket\": [\n    {\n        \"url\": \"https://bitbucket.internal\",    # Base URL for a Bitbucket Server instance\n        \"username\": \"\",                         # Username to authenticate with\n        \"password\": \"\",                         # Password to authenticate with\n\n        \"exclude\": [ ... ]  # List of projects / repositories to exclude from inventory\n    }\n]\n```\n\n```json\n\"TFS\": [\n    {\n        \"url\": \"https://tfs.internal\",  # Base URL for a Team Foundation Server (TFS) or Visual Studio Team Services (VSTS) or Azure DevOps instance\n        \"token\": null,                  # Private token for accessing this TFS instance\n        \"exclude\": [ ... ]  # List of projects / repositories to exclude from inventory\n    }\n]\n```\n\n## License\n\nScraper is released under an MIT license. For more details see the\n[LICENSE](/LICENSE) file.\n\nLLNL-CODE-705597\n\n\n",
        "description_content_type": "text/markdown",
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://github.com/llnl/scraper",
        "keywords": "",
        "license": "",
        "maintainer": "",
        "maintainer_email": "",
        "name": "llnl-scraper",
        "package_url": "https://pypi.org/project/llnl-scraper/",
        "platform": "",
        "project_url": "https://pypi.org/project/llnl-scraper/",
        "project_urls": {
            "Homepage": "https://github.com/llnl/scraper"
        },
        "release_url": "https://pypi.org/project/llnl-scraper/0.8.1/",
        "requires_dist": [
            "github3.py (==1.2.0)",
            "python-gitlab (>=1.6.0)",
            "stashy (>=0.3)",
            "python-dateutil (>=2.7.3)",
            "vsts (>=0.1.25)",
            "requests (>=2.16)",
            "msrest (>=0.6.4)",
            "pytz (>=2017.3)"
        ],
        "requires_python": "",
        "summary": "Package for extracting software repository metadata",
        "version": "0.8.1"
    },
    "last_serial": 5620984,
    "releases": {
        "0.2.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "126220e2545c381f8c422bb191e570bc",
                    "sha256": "cc2a55f71ad5dc14ca56ea6d0d2db2622d1782b61b0733aa66f65358188f2994"
                },
                "downloads": -1,
                "filename": "llnl_scraper-0.2.1-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "126220e2545c381f8c422bb191e570bc",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 15107,
                "upload_time": "2018-03-23T23:34:16",
                "url": "https://files.pythonhosted.org/packages/95/90/7e87b590be0653131a4aac9070db727284005f422abf3b12c97f20ce60cf/llnl_scraper-0.2.1-py2.py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "ae59f5850105d121da2d68a00891cc5f",
                    "sha256": "9b720d3138012f47c55339664521b5c7171fa857a85cfc630d4210a1e48f908f"
                },
                "downloads": -1,
                "filename": "llnl-scraper-0.2.1.tar.gz",
                "has_sig": false,
                "md5_digest": "ae59f5850105d121da2d68a00891cc5f",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 12518,
                "upload_time": "2018-03-23T23:34:18",
                "url": "https://files.pythonhosted.org/packages/9d/4f/a2e91fa2d480cf53c7a0147dbf48ee3c4fb907d8f156d23c842f6c6609b9/llnl-scraper-0.2.1.tar.gz"
            }
        ],
        "0.3.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "f00c33bc54c4b91541379fd8b61e2391",
                    "sha256": "a23676765c9fccc94959081a6148a868a696d7f1184d239c99279f7e13931fd1"
                },
                "downloads": -1,
                "filename": "llnl_scraper-0.3.0-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "f00c33bc54c4b91541379fd8b61e2391",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 20335,
                "upload_time": "2018-08-22T22:13:42",
                "url": "https://files.pythonhosted.org/packages/5f/2e/e22dfaba01998c443627a72d184ca01e21960aed36bec6991ca36b84403d/llnl_scraper-0.3.0-py2.py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "3738a4a1f20b86e4532ed1d8bf1da57b",
                    "sha256": "d3ca59d0cd71262ae07801f67dcf25334d65b41b9955da3d249c061927f2801a"
                },
                "downloads": -1,
                "filename": "llnl-scraper-0.3.0.tar.gz",
                "has_sig": false,
                "md5_digest": "3738a4a1f20b86e4532ed1d8bf1da57b",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 18237,
                "upload_time": "2018-08-22T22:13:44",
                "url": "https://files.pythonhosted.org/packages/b8/d4/943636d7e38e35d578dab9798d166dc1859f7bbd6b6a37579ba1e6e76989/llnl-scraper-0.3.0.tar.gz"
            }
        ],
        "0.4.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "215c306bdd43bc699d3390da3f3f54cd",
                    "sha256": "1fbda67ea679a283b51f85eb913c90e35c1397de5b5c5d21041160b80e4caec7"
                },
                "downloads": -1,
                "filename": "llnl_scraper-0.4.0-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "215c306bdd43bc699d3390da3f3f54cd",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 23107,
                "upload_time": "2018-08-26T06:49:40",
                "url": "https://files.pythonhosted.org/packages/cf/ba/e3326b2d827c6f8e89a1d4fd01e40771d54ff68c23e7354b604ae1e4ae06/llnl_scraper-0.4.0-py2.py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "30417321ee8ec48bd986765c55dd8029",
                    "sha256": "c4db9d3531a0b22372425472b714a452e849e247c5fca79b0be6745c1a108f1e"
                },
                "downloads": -1,
                "filename": "llnl-scraper-0.4.0.tar.gz",
                "has_sig": false,
                "md5_digest": "30417321ee8ec48bd986765c55dd8029",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 19770,
                "upload_time": "2018-08-26T06:49:41",
                "url": "https://files.pythonhosted.org/packages/ab/50/204298eba5d8b31f14b7b4e2f03d8040e37817fed79ee10354561067fd40/llnl-scraper-0.4.0.tar.gz"
            }
        ],
        "0.5.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "a1a5ad37b145b53019758e8d5db31b86",
                    "sha256": "ea38bb1b9abe64bfbaac65c9d1dbdb2b17db0bab7222eeab06c0e90e5f6e335f"
                },
                "downloads": -1,
                "filename": "llnl_scraper-0.5.0-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "a1a5ad37b145b53019758e8d5db31b86",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 24892,
                "upload_time": "2018-08-28T23:57:44",
                "url": "https://files.pythonhosted.org/packages/d8/90/02f9dab48090923b184e77fd182c4681cbef26878feb0448adb35ee8d33f/llnl_scraper-0.5.0-py2.py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "35ac4f78dec4d29dfd13af18a60103ae",
                    "sha256": "03d505527265dfad2a014a1c15728839edf852b2e92f4a919f41cada85fb0317"
                },
                "downloads": -1,
                "filename": "llnl-scraper-0.5.0.tar.gz",
                "has_sig": false,
                "md5_digest": "35ac4f78dec4d29dfd13af18a60103ae",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 21744,
                "upload_time": "2018-08-29T00:02:57",
                "url": "https://files.pythonhosted.org/packages/d4/f0/37c9c43954304309a6ecb2beaac1d53385ef008dd92ad171115091ce2145/llnl-scraper-0.5.0.tar.gz"
            }
        ],
        "0.5.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "4459075e196d9612cb5f701aba58f8c3",
                    "sha256": "8760220d26e831e4317bea0b770e5a95c909fa02d8edbf5726ca7cfa437c2d6a"
                },
                "downloads": -1,
                "filename": "llnl_scraper-0.5.1-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "4459075e196d9612cb5f701aba58f8c3",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 24944,
                "upload_time": "2018-08-30T04:38:18",
                "url": "https://files.pythonhosted.org/packages/98/4f/178cc2dde3719089637f8a7ac93126902ce6d55836bdf8eb550dd8f6e1a1/llnl_scraper-0.5.1-py2.py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "fe10d9fa0bf8dd02b50663ee9bf6c90c",
                    "sha256": "31eb3bca5a6e87fdfe978aae286afc5cf3d198f9a9d38285a792d13cb3460b74"
                },
                "downloads": -1,
                "filename": "llnl-scraper-0.5.1.tar.gz",
                "has_sig": false,
                "md5_digest": "fe10d9fa0bf8dd02b50663ee9bf6c90c",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 21777,
                "upload_time": "2018-08-30T04:38:19",
                "url": "https://files.pythonhosted.org/packages/90/d4/06723590690affdf0af7b0149277a83d45d14120b9e1d5a2fb2cadab9d07/llnl-scraper-0.5.1.tar.gz"
            }
        ],
        "0.6.0.dev0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "853cba99601883c0a59dccb0a448dfec",
                    "sha256": "326e4a79a3ab600e58bdd17476cd60bfdc7369a5ded085d6925043bb16536c38"
                },
                "downloads": -1,
                "filename": "llnl_scraper-0.6.0.dev0-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "853cba99601883c0a59dccb0a448dfec",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 25005,
                "upload_time": "2019-02-06T20:57:39",
                "url": "https://files.pythonhosted.org/packages/f0/81/3a61d35057783df52f49bf0e4cda61a0e95a9807cd8438ffbbbb13c35e65/llnl_scraper-0.6.0.dev0-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "c054de404013ab0f4d2ca609c8d95ce1",
                    "sha256": "cf236a8f92b6ab75d4ebd52a56207b0da1da85bc47843603eb9f9eb486c3c942"
                },
                "downloads": -1,
                "filename": "llnl-scraper-0.6.0.dev0.tar.gz",
                "has_sig": false,
                "md5_digest": "c054de404013ab0f4d2ca609c8d95ce1",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 21772,
                "upload_time": "2019-02-06T20:57:41",
                "url": "https://files.pythonhosted.org/packages/0a/31/68c8f5ab02cb75cf40b6b0246220161c489c4da8f874ac771d070a91d28a/llnl-scraper-0.6.0.dev0.tar.gz"
            }
        ],
        "0.6.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "f14dae088df2b4cf0f21b5c9ba043d2e",
                    "sha256": "0aa3f08aa665c169641b2e5aa53a15cc1b697a5fe24e4cd8f65fd1db31d1bec6"
                },
                "downloads": -1,
                "filename": "llnl_scraper-0.6.1-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "f14dae088df2b4cf0f21b5c9ba043d2e",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 27082,
                "upload_time": "2019-02-06T20:57:40",
                "url": "https://files.pythonhosted.org/packages/71/69/aba80616aa6d484c2bf2c4b64253dfc158e8f90570f9ad6a68b50e3dc0e9/llnl_scraper-0.6.1-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "978b0aa3e06343d155584f7f51caf658",
                    "sha256": "71c8a005acc38b1bae9346b295459ca1e077edbe582d8556c0caf0452d0b04d7"
                },
                "downloads": -1,
                "filename": "llnl-scraper-0.6.1.tar.gz",
                "has_sig": false,
                "md5_digest": "978b0aa3e06343d155584f7f51caf658",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 20672,
                "upload_time": "2019-02-06T20:57:43",
                "url": "https://files.pythonhosted.org/packages/b5/94/bf5d62b98711f7250c587c0518cdad172f0528b881e0b06966eb0f516192/llnl-scraper-0.6.1.tar.gz"
            }
        ],
        "0.7.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "dbf29a43f18db64ad088a77a2424dacc",
                    "sha256": "34bb9710532362707ccce38b8d4d745677ff15667a249149659c24a243e2adba"
                },
                "downloads": -1,
                "filename": "llnl_scraper-0.7.0-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "dbf29a43f18db64ad088a77a2424dacc",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 29848,
                "upload_time": "2019-03-19T03:05:49",
                "url": "https://files.pythonhosted.org/packages/1e/b5/e2c0da506acd143ab98f3213d149bf9c9b42dddb550ca2f155592b302a99/llnl_scraper-0.7.0-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "cb2ee5b51aaa7b67113cd45d3ee1ea14",
                    "sha256": "aa7e51867a6c3f9cd63a4251d6d489483baedd7ecee445721624ef800b23d3ad"
                },
                "downloads": -1,
                "filename": "llnl-scraper-0.7.0.tar.gz",
                "has_sig": false,
                "md5_digest": "cb2ee5b51aaa7b67113cd45d3ee1ea14",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 22949,
                "upload_time": "2019-03-19T03:05:51",
                "url": "https://files.pythonhosted.org/packages/25/a0/7950654b1cc6c7d130f3745ea010962982a7f7a5112e2003bc10995b5e46/llnl-scraper-0.7.0.tar.gz"
            }
        ],
        "0.8.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "67f117201d8de7b50f1fabd6808867ac",
                    "sha256": "17d0fd2cde898dcd3d630a2c7cbc115b642e4381a39c0be221bdf67baf287296"
                },
                "downloads": -1,
                "filename": "llnl_scraper-0.8.1-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "67f117201d8de7b50f1fabd6808867ac",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 29106,
                "upload_time": "2019-08-01T21:46:48",
                "url": "https://files.pythonhosted.org/packages/52/d3/d6f210ecbc6a65d63010277b98590f6f19f3fd6a327891986ba1c57a00e4/llnl_scraper-0.8.1-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "8a7b8e31e0c6a011b405d606a0f878d0",
                    "sha256": "e26782a0d6ef4e016d930684dfe23753670e57303676e52ed2e2db06e8c1d198"
                },
                "downloads": -1,
                "filename": "llnl-scraper-0.8.1.tar.gz",
                "has_sig": false,
                "md5_digest": "8a7b8e31e0c6a011b405d606a0f878d0",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 23042,
                "upload_time": "2019-08-01T21:46:56",
                "url": "https://files.pythonhosted.org/packages/2d/4d/ad16a4932b2ff0f0cdad19c6a23418a53d1456c76423493c8753c5e3fb9a/llnl-scraper-0.8.1.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "67f117201d8de7b50f1fabd6808867ac",
                "sha256": "17d0fd2cde898dcd3d630a2c7cbc115b642e4381a39c0be221bdf67baf287296"
            },
            "downloads": -1,
            "filename": "llnl_scraper-0.8.1-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "67f117201d8de7b50f1fabd6808867ac",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": null,
            "size": 29106,
            "upload_time": "2019-08-01T21:46:48",
            "url": "https://files.pythonhosted.org/packages/52/d3/d6f210ecbc6a65d63010277b98590f6f19f3fd6a327891986ba1c57a00e4/llnl_scraper-0.8.1-py3-none-any.whl"
        },
        {
            "comment_text": "",
            "digests": {
                "md5": "8a7b8e31e0c6a011b405d606a0f878d0",
                "sha256": "e26782a0d6ef4e016d930684dfe23753670e57303676e52ed2e2db06e8c1d198"
            },
            "downloads": -1,
            "filename": "llnl-scraper-0.8.1.tar.gz",
            "has_sig": false,
            "md5_digest": "8a7b8e31e0c6a011b405d606a0f878d0",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 23042,
            "upload_time": "2019-08-01T21:46:56",
            "url": "https://files.pythonhosted.org/packages/2d/4d/ad16a4932b2ff0f0cdad19c6a23418a53d1456c76423493c8753c5e3fb9a/llnl-scraper-0.8.1.tar.gz"
        }
    ]
}