{ "info": { "author": "Ben Morris, Shivam Negi, Akash Goel, Andrew Zhang, Henry Senyondo, Ethan White", "author_email": "ethan@weecology.org", "bugtrack_url": null, "classifiers": [ "Intended Audience :: Science/Research", "License :: OSI Approved :: MIT License", "Operating System :: Microsoft :: Windows", "Operating System :: POSIX", "Operating System :: Unix", "Programming Language :: Python", "Programming Language :: Python :: 3", "Programming Language :: Python :: 3.4", "Programming Language :: Python :: 3.5", "Programming Language :: Python :: 3.6", "Programming Language :: Python :: 3.7", "Programming Language :: Python :: Implementation :: PyPy", "Topic :: Database", "Topic :: Scientific/Engineering :: GIS", "Topic :: Scientific/Engineering :: Information Analysis", "Topic :: Software Development :: Libraries :: Python Modules" ], "description": "![Retriever logo](http://i.imgur.com/se7TtrK.png)\n\n\n[![Build Status](https://travis-ci.org/weecology/retriever.svg?branch=master)](https://travis-ci.org/weecology/retriever)\n[![Build Status (windows)](https://ci.appveyor.com/api/projects/status/qetgo4jxa5769qtb/branch/master?svg=true)](https://ci.appveyor.com/project/ethanwhite/retriever/branch/master)\n[![Research software impact](http://depsy.org/api/package/pypi/retriever/badge.svg)](http://depsy.org/package/python/retriever)\n[![codecov.io](https://codecov.io/github/weecology/retriever/coverage.svg?branch=master)](https://codecov.io/github/weecology/retriever?branch=master)\n[![Documentation Status](https://readthedocs.org/projects/retriever/badge/?version=latest)](http://retriever.readthedocs.io/en/latest/?badge=latest)\n[![License](http://img.shields.io/badge/license-MIT-blue.svg)](https://raw.githubusercontent.com/weecology/retriever/master/LICENSE)\n[![Join the chat at https://gitter.im/weecology/retriever](https://badges.gitter.im/weecology/retriever.svg)](https://gitter.im/weecology/retriever?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)\n[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.1038272.svg)](https://doi.org/10.5281/zenodo.1038272)\n[![JOSS Publication](http://joss.theoj.org/papers/10.21105/joss.00451/status.svg)](https://doi.org/10.21105/joss.00451)\n[![Anaconda-Server Badge](https://anaconda.org/conda-forge/retriever/badges/version.svg)](https://anaconda.org/conda-forge/retriever)\n[![Version](https://img.shields.io/pypi/v/retriever.svg)](https://pypi.python.org/pypi/retriever)\n\n\nFinding data is one thing. Getting it ready for analysis is another. Acquiring,\ncleaning, standardizing and importing publicly available data is time consuming\nbecause many datasets lack machine readable metadata and do not conform to\nestablished data structures and formats. The Data Retriever automates the first\nsteps in the data analysis pipeline by downloading, cleaning, and standardizing\ndatasets, and importing them into relational databases, flat files, or\nprogramming languages. The automation of this process reduces the time for a\nuser to get most large datasets up and running by hours, and in some cases days.\n\n## Installing the Current Release\n\nIf you have Python installed you can install the current release using either `pip`:\n\n```bash\npip install retriever\n```\n\nor `conda` after adding the `conda-forge` channel (`conda config --add channels conda-forge`):\n\n```bash\nconda install retriever\n```\n\nDepending on your system configuration this may require `sudo` for `pip`:\n\n```bash\nsudo pip install retriever\n```\n\nPrecompiled binary installers are also available for Windows, OS X, and\nUbuntu/Debian on\nthe [releases page](https://github.com/weecology/retriever/releases). These do\nnot require a Python installation. Download the installer for your operating\nsystem and follow the instructions at on\nthe [download page](http://www.data-retriever.org/download.html).\n\n\nInstalling From Source\n----------------------\n\nTo install the Data Retriever from source, you'll need Python 2.7+ or 3.3+ with the following packages installed:\n\n* xlrd\n\nThe following packages are optionally needed to interact with associated\ndatabase management systems:\n\n* PyMySQL (for MySQL)\n* sqlite3 (for SQLite)\n* psycopg2-binary (for PostgreSQL), previously psycopg2.\n* pyodbc (for MS Access - this option is only available on Windows)\n* Microsoft Access Driver (ODBC for windows)\n\n### To install from source\n\nEither use pip to install directly from GitHub:\n\n```shell\npip install git+https://git@github.com/weecology/retriever.git\n```\n\nor:\n\n1. Clone the repository\n2. From the directory containing setup.py, run the following command: `pip\n install .`. You may need to include `sudo` at the beginning of the\n command depending on your system (i.e., `sudo pip install .`).\n\nMore extensive documentation for those that are interested in developing can be found [here](http://retriever.readthedocs.io/en/latest/?badge=latest)\n\nUsing the Command Line\n----------------------\nAfter installing, run `retriever update` to download all of the available dataset scripts.\nTo see the full list of command line options and datasets run `retriever --help`.\nThe output will look like this:\n\n```shell\nusage: retriever [-h] [-v] [-q]\n {download,install,defaults,update,new,new_json,edit_json,delete_json,ls,citation,reset,help}\n ...\n\npositional arguments:\n {download,install,defaults,update,new,new_json,edit_json,delete_json,ls,citation,reset,help}\n sub-command help\n download download raw data files for a dataset\n install download and install dataset\n defaults displays default options\n update download updated versions of scripts\n new create a new sample retriever script\n new_json CLI to create retriever datapackage.json script\n edit_json CLI to edit retriever datapackage.json script\n delete_json CLI to remove retriever datapackage.json script\n ls display a list all available dataset scripts\n citation view citation\n reset reset retriever: removes configuration settings,\n scripts, and cached data\n help\n\noptional arguments:\n -h, --help show this help message and exit\n -v, --version show program's version number and exit\n -q, --quiet suppress command-line output\n```\n\nTo install datasets, use `retriever install`:\n\n```shell\nusage: retriever install [-h] [--compile] [--debug]\n {mysql,postgres,sqlite,msaccess,csv,json,xml} ...\n\npositional arguments:\n {mysql,postgres,sqlite,msaccess,csv,json,xml}\n engine-specific help\n mysql MySQL\n postgres PostgreSQL\n sqlite SQLite\n msaccess Microsoft Access\n csv CSV\n json JSON\n xml XML\n\noptional arguments:\n -h, --help show this help message and exit\n --compile force re-compile of script before downloading\n --debug run in debug mode\n```\n\n\n### Examples\n\nThese examples are using the [*Iris* flower dataset](https://en.wikipedia.org/wiki/Iris_flower_data_set).\nMore examples can be found in the Data Retriever documentation.\n\nUsing Install\n\n```shell\nretriever install -h (gives install options)\n```\n\nUsing specific database engine, retriever install {Engine}\n\n```shell\nretriever install mysql -h (gives install mysql options)\nretriever install mysql --user myuser --password ******** --host localhost --port 8888 --database_name testdbase iris\n```\ninstall data into an sqlite database named iris.db you would use:\n\n```shell\nretriever install sqlite iris -f iris.db\n```\n\nUsing download\n\n```shell\nretriever download -h (gives you help options)\nretriever download iris\nretriever download iris --path C:\\Users\\Documents\n```\n\nUsing citation\n\n```shell\nretriever citation (citation of the retriever engine)\nretriever citation iris (citation for the iris data)\n```\n\nSpatial Dataset Installation\n----------------------------\n\n**Set up Spatial support**\n\nTo set up spatial support for Postgres using Postgis please\nrefer to the [spatial set-up docs](https://retriever.readthedocs.io/en/latest/spatial_dbms.html).\n\n```shell\nretriever install postgres harvard-forest # Vector data\nretriever install postgres bioclim # Raster data\n# Install only the data of USGS elevation in the given extent\nretriever install postgres usgs-elevation -b -94.98704597353938 39.027001800158615 -94.3599408119917 40.69577051867074\n\n```\n\nWebsite\n-------\n\nFor more information see the\n[Data Retriever website](http://www.data-retriever.org/).\n\nAcknowledgments\n---------------\n\nDevelopment of this software was funded by [the Gordon and Betty Moore\nFoundation's Data-Driven Discovery\nInitiative](http://www.moore.org/programs/science/data-driven-discovery) through\n[Grant GBMF4563](http://www.moore.org/grants/list/GBMF4563) to Ethan White and\nthe [National Science Foundation](http://nsf.gov/) as part of a [CAREER award to\nEthan White](http://nsf.gov/awardsearch/showAward.do?AwardNumber=0953694).\n\n\n", "description_content_type": "text/markdown", "docs_url": null, "download_url": "", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "https://github.com/weecology/retriever", "keywords": "", "license": "", "maintainer": "", "maintainer_email": "", "name": "retriever", "package_url": "https://pypi.org/project/retriever/", "platform": "", "project_url": "https://pypi.org/project/retriever/", "project_urls": { "Homepage": "https://github.com/weecology/retriever" }, "release_url": "https://pypi.org/project/retriever/2.4.0/", "requires_dist": [ "xlrd", "future", "argcomplete", "tqdm", "requests", "pandas" ], "requires_python": "", "summary": "Data Retriever", "version": "2.4.0" }, "last_serial": 5384767, "releases": { "1.4": [ { "comment_text": "", "digests": { "md5": "d563e16bdf4f178368c8c8b685a466e2", "sha256": "f268e0f8a7cbdc7d8a1f141ddcec5781a9e0c98b643c652910036daaf2e1e7d9" }, "downloads": -1, "filename": "retriever-1.4.tar.gz", "has_sig": false, "md5_digest": "d563e16bdf4f178368c8c8b685a466e2", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 102081, "upload_time": "2012-12-03T16:21:23", "url": "https://files.pythonhosted.org/packages/35/ce/04232d8d138dcf85869cb9836df3d29394bd973fd3cf7742a0217d670073/retriever-1.4.tar.gz" } ], "1.6": [ { "comment_text": "", "digests": { "md5": "0ce417dd576bb3a7897748bcc2567403", "sha256": "57605de869603d4ca2ccb50215966db37cd7a4fc659cec5690dffc9c308c6ab5" }, "downloads": -1, "filename": "retriever-1.6.tar.gz", "has_sig": false, "md5_digest": "0ce417dd576bb3a7897748bcc2567403", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 105925, "upload_time": "2014-03-23T17:53:49", "url": "https://files.pythonhosted.org/packages/ab/79/267d259ce444f63b2075fd2a6bda4c54ac279b6ee2ef84e686b292fd24b9/retriever-1.6.tar.gz" } ], "1.7.0": [ { "comment_text": "", "digests": { "md5": "c943cd228bc5f0072e3a192af105fd9a", "sha256": "ac81dee8ce7bf4e41344841d88bd3fa089558d55368ab6b01c5076d55c4d44a5" }, "downloads": -1, "filename": "retriever-1.7.0.tar.gz", "has_sig": false, "md5_digest": "c943cd228bc5f0072e3a192af105fd9a", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 107790, "upload_time": "2014-10-05T12:38:07", "url": "https://files.pythonhosted.org/packages/c9/84/f83754c400c8b59044b26df7238aa0c603ecb6f941f55c3e8a230d0f0561/retriever-1.7.0.tar.gz" } ], "1.8.3": [ { "comment_text": "", "digests": { "md5": "e14017c7bfc6791013d1ddcd62355c72", "sha256": "61d6e9c6e1a92ccd439c7e035bc713eb8d53fb8d9b381fb5abdadcdf67906ac2" }, "downloads": -1, "filename": "retriever-1.8.3.zip", "has_sig": false, "md5_digest": "e14017c7bfc6791013d1ddcd62355c72", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 123066, "upload_time": "2016-02-12T16:35:03", "url": "https://files.pythonhosted.org/packages/68/43/c83803dce8c6a416c0c2c096410a7c9e038bb7d0ceba55e34fb4ed250856/retriever-1.8.3.zip" } ], "2.0.0": [ { "comment_text": "", "digests": { "md5": "967cada3e7e44d4b69f036597558129e", "sha256": "508eb6e93503c8359d695235cf1a59a96fa76a9765c8c6f6563af2505b8de090" }, "downloads": -1, "filename": "retriever-2.0.0-py2.py3-none-any.whl", "has_sig": false, "md5_digest": "967cada3e7e44d4b69f036597558129e", "packagetype": "bdist_wheel", "python_version": "2.7", "requires_python": null, "size": 159823, "upload_time": "2017-02-24T17:02:09", "url": "https://files.pythonhosted.org/packages/bb/e4/3c019e12c587b23d64b73353bc4fa860336640e5a694b07d745abc9207af/retriever-2.0.0-py2.py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "ee62df97dc5ef88a6ab048eb96f87347", "sha256": "5c82f7b11a0c1825876e1e7463e456dc7990e8d1f93af7d4f641f9108b31e6ff" }, "downloads": -1, "filename": "retriever-2.0.0.tar.gz", "has_sig": false, "md5_digest": "ee62df97dc5ef88a6ab048eb96f87347", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 43840, "upload_time": "2017-02-24T17:02:07", "url": "https://files.pythonhosted.org/packages/e3/c4/e5c13600e0613a9a4c347391dab0b1a03b0ee0e086af758fb0bf2d115bcd/retriever-2.0.0.tar.gz" } ], "2.1.0": [ { "comment_text": "", "digests": { "md5": "f258a92863efaf16d2e6532bd6c08350", "sha256": "c45cede0e672d9b13298dd881ac59cf1664804fb47a59fd7c0e6c8f27d5431c1" }, "downloads": -1, "filename": "retriever-2.1.0-py2.py3-none-any.whl", "has_sig": false, "md5_digest": "f258a92863efaf16d2e6532bd6c08350", "packagetype": "bdist_wheel", "python_version": "3.5", "requires_python": null, "size": 55574, "upload_time": "2017-10-27T21:10:16", "url": "https://files.pythonhosted.org/packages/49/1c/97242bd26e5d109addcc954e945746c171c2a5914d73d5fcd6e058e7d6f0/retriever-2.1.0-py2.py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "4604318a559e9be6b889f9169001510f", "sha256": "a1f3c9a50369246fab6faa9915981e5b85e57745c52c6277ff5a2eab24cd2483" }, "downloads": -1, "filename": "retriever-2.1.0.tar.gz", "has_sig": false, "md5_digest": "4604318a559e9be6b889f9169001510f", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 50650, "upload_time": "2017-10-27T21:10:15", "url": "https://files.pythonhosted.org/packages/1e/72/48f31950c4226d2787c8269264856e6e3cff7b80f93a01b2754e89706571/retriever-2.1.0.tar.gz" } ], "2.2.0": [ { "comment_text": "", "digests": { "md5": "4a310c8027c7b62cf5b186297e4e37a7", "sha256": "315790d4a2e738883b8b2b8ea61b020cf46c6e59ef404e39b9633164a015defe" }, "downloads": -1, "filename": "retriever-2.2.0-py2.py3-none-any.whl", "has_sig": false, "md5_digest": "4a310c8027c7b62cf5b186297e4e37a7", "packagetype": "bdist_wheel", "python_version": "3.6", "requires_python": null, "size": 59723, "upload_time": "2018-11-06T07:33:48", "url": "https://files.pythonhosted.org/packages/14/68/a214aee1dd3733998b84ed4adb6c44ee1a4aeffe524268ad525809069bbb/retriever-2.2.0-py2.py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "7cbdd34eebbe914c78d288c628c31a94", "sha256": "8b195f70b04e89e7e3b1423e165271339e8dcc712c7f6d38daebb294d978f0a1" }, "downloads": -1, "filename": "retriever-2.2.0.tar.gz", "has_sig": false, "md5_digest": "7cbdd34eebbe914c78d288c628c31a94", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 57696, "upload_time": "2018-11-06T07:33:46", "url": "https://files.pythonhosted.org/packages/66/ca/cc53486779fb13f48e7b6a2a03e01487f4cb58d79568c32ceb52349016cb/retriever-2.2.0.tar.gz" } ], "2.3.0": [ { "comment_text": "", "digests": { "md5": "3eb3933b9e3c8672167b4f4f49b20589", "sha256": "6391cf29484bab97f4fe7176829e6a6790f8a453a1e1b0ce06212c6250b61b34" }, "downloads": -1, "filename": "retriever-2.3.0-py2.py3-none-any.whl", "has_sig": false, "md5_digest": "3eb3933b9e3c8672167b4f4f49b20589", "packagetype": "bdist_wheel", "python_version": "2.7", "requires_python": null, "size": 77413, "upload_time": "2019-05-01T17:14:07", "url": "https://files.pythonhosted.org/packages/09/94/b96c25fdc74afa3bff29984d0eb2e6aef5808178b118fd75b2cd99c1e3fe/retriever-2.3.0-py2.py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "a7c68828b7be0a61d4a3458377293ce4", "sha256": "711c731feef837bc6aa08d209d86df35cd5486e6632e53ae0a2c41a1cb123de9" }, "downloads": -1, "filename": "retriever-2.3.0.tar.gz", "has_sig": false, "md5_digest": "a7c68828b7be0a61d4a3458377293ce4", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 72040, "upload_time": "2019-05-01T17:14:05", "url": "https://files.pythonhosted.org/packages/4a/8e/1065db8922ce5d7328ea373225d8f4fa7c1f64a5934432ab991654240d23/retriever-2.3.0.tar.gz" } ], "2.3.1": [ { "comment_text": "", "digests": { "md5": "13e0e97cd6ef2a56dc0fdde8f69f243d", "sha256": "c64da642bed69948257f4c0c841dd15dcec6167f6d7b8b5415b0d1a4bd7932f5" }, "downloads": -1, "filename": "retriever-2.3.1-py2.py3-none-any.whl", "has_sig": false, "md5_digest": "13e0e97cd6ef2a56dc0fdde8f69f243d", "packagetype": "bdist_wheel", "python_version": "py2.py3", "requires_python": null, "size": 77462, "upload_time": "2019-05-01T17:59:32", "url": "https://files.pythonhosted.org/packages/b5/2d/e8d5928e688431e8125983d0a3ce33849893457877cc0f5138af1ca79cfc/retriever-2.3.1-py2.py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "b2c02a8cff1c40010ff48bb3e2959449", "sha256": "b8a774a45db31e5ba4b693aabfa8c6ef958ea82187d5f97fd1d98d02da7ca540" }, "downloads": -1, "filename": "retriever-2.3.1.tar.gz", "has_sig": false, "md5_digest": "b2c02a8cff1c40010ff48bb3e2959449", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 72090, "upload_time": "2019-05-01T17:59:33", "url": "https://files.pythonhosted.org/packages/18/21/159f927c4b6c46273937140070e7ee00c79051b684713eb7ec153c9304d6/retriever-2.3.1.tar.gz" } ], "2.4.0": [ { "comment_text": "", "digests": { "md5": "470e92c23ad62748c9d3108c54193b39", "sha256": "5989413e8cbca2a97c74e46d473f0c2695e9599026f2d5920400d4870df76be7" }, "downloads": -1, "filename": "retriever-2.4.0-py2.py3-none-any.whl", "has_sig": false, "md5_digest": "470e92c23ad62748c9d3108c54193b39", "packagetype": "bdist_wheel", "python_version": "py2.py3", "requires_python": null, "size": 65942, "upload_time": "2019-06-11T05:52:05", "url": "https://files.pythonhosted.org/packages/53/c1/80576fe892ed950ee2c3bf126b5c6b3f3420bf315b1f5fe6f3cdddbf1ad1/retriever-2.4.0-py2.py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "d87ac185c02c5e7c8c9afd15bb5da1ca", "sha256": "f59045647bca50036a5cef0bd6167b079c635295160ab29000a0694944e64837" }, "downloads": -1, "filename": "retriever-2.4.0.tar.gz", "has_sig": false, "md5_digest": "d87ac185c02c5e7c8c9afd15bb5da1ca", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 65828, "upload_time": "2019-06-11T05:52:07", "url": "https://files.pythonhosted.org/packages/8f/ad/456f788f4b9704570080a9a84710dd67eaa509bd8f34ef6611576d7b587f/retriever-2.4.0.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "470e92c23ad62748c9d3108c54193b39", "sha256": "5989413e8cbca2a97c74e46d473f0c2695e9599026f2d5920400d4870df76be7" }, "downloads": -1, "filename": "retriever-2.4.0-py2.py3-none-any.whl", "has_sig": false, "md5_digest": "470e92c23ad62748c9d3108c54193b39", "packagetype": "bdist_wheel", "python_version": "py2.py3", "requires_python": null, "size": 65942, "upload_time": "2019-06-11T05:52:05", "url": "https://files.pythonhosted.org/packages/53/c1/80576fe892ed950ee2c3bf126b5c6b3f3420bf315b1f5fe6f3cdddbf1ad1/retriever-2.4.0-py2.py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "d87ac185c02c5e7c8c9afd15bb5da1ca", "sha256": "f59045647bca50036a5cef0bd6167b079c635295160ab29000a0694944e64837" }, "downloads": -1, "filename": "retriever-2.4.0.tar.gz", "has_sig": false, "md5_digest": "d87ac185c02c5e7c8c9afd15bb5da1ca", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 65828, "upload_time": "2019-06-11T05:52:07", "url": "https://files.pythonhosted.org/packages/8f/ad/456f788f4b9704570080a9a84710dd67eaa509bd8f34ef6611576d7b587f/retriever-2.4.0.tar.gz" } ] }