{ "info": { "author": "Catalyst Cooperative", "author_email": "pudl@catalyst.coop", "bugtrack_url": null, "classifiers": [ "Development Status :: 3 - Alpha", "Environment :: Console", "Intended Audience :: Developers", "Intended Audience :: Science/Research", "License :: OSI Approved :: MIT License", "Natural Language :: English", "Operating System :: OS Independent", "Programming Language :: Python :: 3.7", "Topic :: Scientific/Engineering" ], "description": "===============================================================================\nThe Public Utility Data Liberation Project (PUDL)\n===============================================================================\n\n.. readme-intro\n\n.. image:: https://www.repostatus.org/badges/latest/active.svg\n :target: https://www.repostatus.org/#active\n :alt: Project Status: Active \u2013 The project has reached a stable, usable state and is being actively developed.\n\n.. image:: https://img.shields.io/travis/catalyst-cooperative/pudl\n :target: https://travis-ci.org/catalyst-cooperative/pudl\n :alt: Travis CI Build Status\n\n.. image:: https://img.shields.io/readthedocs/catalystcoop-pudl\n :target: https://catalystcoop-pudl.readthedocs.io/en/latest/\n :alt: Read the Docs Build Status\n\n.. image:: https://img.shields.io/codecov/c/github/catalyst-cooperative/pudl\n :target: https://codecov.io/gh/catalyst-cooperative/pudl\n :alt: Codecov Test Coverage\n\n.. image:: https://img.shields.io/codacy/grade/2fead07adef249c08288d0bafae7cbb5\n :target: https://app.codacy.com/app/zaneselvans/pudl\n :alt: Codacy Grade\n\n.. image:: https://img.shields.io/pypi/v/catalystcoop.pudl\n :target: https://pypi.org/project/catalystcoop.pudl/\n :alt: PyPI Version\n\n.. image:: https://img.shields.io/conda/vn/conda-forge/catalystcoop.pudl\n :target: https://anaconda.org/conda-forge/catalystcoop.pudl\n :alt: conda-forge Version\n\n.. image:: https://zenodo.org/badge/80646423.svg\n :target: https://zenodo.org/badge/latestdoi/80646423\n :alt: Zenodo DOI\n\n`PUDL `__ makes US energy data easier to access\nand work with. Hundreds of gigabytes of public information is published\nby government agencies, but in many different formats that make it hard to\nwork with and combine. PUDL takes these spreadsheets, CSV files, and databases\nand turns them into easy use\n`tabular data packages `__\nthat can populate a database, or be used directly with Python, R, Microsoft\nAccess, and many other tools.\n\nThe project currently integrates data from:\n\n* `EIA Form 860 `__\n* `EIA Form 923 `__\n* `The EPA Continuous Emissions Monitoring System (CEMS) `__\n* `The EPA Integrated Planning Model (IPM) `__\n* `FERC Form 1 `__\n\nThe project is especially meant to serve researchers, activists, journalists,\nand policy makers that might not otherwise be able to afford access to this\ndata from existing commercial data providers.\n\nGetting Started\n---------------\n\nJust want to play with some example data? Install\n`Anaconda `__\n(or `miniconda `__) with at\nleast Python 3.7. Then work through the following commands.\n\nFirst, we create and activate conda environment named ``pudl``. All the\nrequired packages are available from the community maintained ``conda-forge``\nchannel, and that channel is given priority, to simplify satisfying\ndependencies. Note that PUDL currently requires Python 3.7, the most recently\navailable major version of Python. In addition to the ``catalystcoop.pudl``\npackage, we'll also install JupyterLab so we can work with the PUDL data\ninteractively.\n\n.. code-block:: console\n\n $ conda create -y -n pudl -c conda-forge --strict-channel-priority python=3.7 catalystcoop.pudl jupyter jupyterlab pip\n $ conda activate pudl\n\nNow we create a data management workspace called ``pudl-work`` and download\nsome data. The workspace is a well defined directory structure that PUDL uses\nto organize the data it downloads, processes, and outputs. You can run\n``pudl_setup --help`` and ``pudl_data --help`` for more information.\n\n.. code-block:: console\n\n $ mkdir pudl-work\n $ pudl_setup pudl-work\n $ pudl_data --sources eia923 eia860 ferc1 epacems epaipm --years 2017 --states id\n\nNow that we have the original data as published by the federal agencies, we can\nrun the ETL (Extract, Transform, Load) pipeline, that turns the raw data into\nan well organized, standardized bundle of data packages. This involves a couple\nof steps: cloning the FERC Form 1 into an SQLite database, extracting data from\nthat database and all the other sources and cleaning it up, outputting that\ndata into well organized CSV/JSON based data packages, and finally loading\nthose data packages into a local database.\n\nPUDL provides a script to clone the FERC Form 1 database, controlled by a YAML\nfile which you can find in the settings folder. Run it like this:\n\n.. code-block:: console\n\n $ ferc1_to_sqlite pudl-work/settings/ferc1_to_sqlite_example.yml\n\nThe main ETL process is controlled by the YAML file ``etl_example.yml`` which\ndefines what data will be processed. It is well commented -- if you want to\nunderstand what the options are, open it in a text editor, and create your own\nversion.\n\nThe data packages will be generated in a sub-directory in\n``pudl-work/datapackage`` named ``pudl-example`` (you can change this by\nchanging the value of ``pkg_bundle_name`` in the ETL settings file you're\nusing. Run the ETL pipeline with this command:\n\n.. code-block:: console\n\n $ pudl_etl pudl-work/settings/etl_example.yml\n\nThe generated data packages are made up of CSV and JSON files. They're both\neasy to parse programmatically, and readable by humans. They are also well\nsuited to archiving, citation, and bulk distribution. However, to make the\ndata easier to query and work with interactively, we typically load it into a\nlocal SQLite database using this script, which combines several data packages\nfrom the same bundle into a single unified structure:\n\n.. code-block:: console\n\n $ datapkg_to_sqlite --pkg_bundle_name pudl-example\n\nNow that we have a live database, we can easily work with it using a variety\nof tools, including Python, pandas dataframes, and\n`Jupyter notebooks `__. This command will start up a local\nJupyter notebook server, and open a notebook of PUDL usage examples:\n\n.. code-block:: console\n\n $ jupyter lab pudl-work/notebook/pudl_intro.ipynb\n\nFor more details, see `the full PUDL documentation\n`__ on Read The Docs.\n\nContributing to PUDL\n--------------------\n\nFind PUDL useful? Want to help make it better? There are lots of ways to\ncontribute!\n\n* Please be sure to read our `Code of Conduct `__\n* You can file a bug report, make a feature request, or ask questions in the\n `Github issue tracker\n `__.\n* Feel free to fork the project and make a pull request with new code,\n better documentation, or example notebooks.\n* `Make a recurring financial contribution `__ to support\n our work liberating public energy data.\n* Hire us to do some custom analysis, and let us add the code the project.\n* For more information check out our `Contribution Guidelines `__\n\nLicensing\n---------\n\nThe PUDL software is released under the\n`MIT License `__.\n`The PUDL documentation `__\nand the data packages we distribute are released under the\n`CC-BY-4.0 `__ license.\n\nContact Us\n----------\n\nFor help with initial setup, usage questions, bug reports, suggestions to make\nPUDL better and anything else that could conceivably be of use or interest to\nthe broader community of users, use the\n`PUDL issue tracker `__.\non Github. For private communication about the project, you can email the\nteam: `pudl@catalyst.coop `__\n\nAbout Catalyst Cooperative\n--------------------------\n\n`Catalyst Cooperative `__ is a small group of data\nscientists and policy wonks. We\u2019re organized as a worker-owned cooperative\nconsultancy. Our goal is a more just, livable, and sustainable world. We\nintegrate public data and perform custom analyses to inform public policy\nmaking. Our focus is primarily on mitigating climate change and improving\nelectric utility regulation in the United States.\n\nDo you work on renewable energy or climate policy? Have you found yourself\nscraping data from government PDFs, spreadsheets, websites, and databases,\nwithout getting something reusable? We build tools to pull this kind of\ninformation together reliably and automatically so you can focus on your real\nwork instead \u2014 whether that\u2019s political advocacy, energy journalism, academic\nresearch, or public policy making.\n\n* Web: https://catalyst.coop\n* Newsletter: https://catalyst.coop/updates/\n* Email: `hello@catalyst.coop `__\n* Twitter: `@CatalystCoop `__\n\n\n", "description_content_type": "text/x-rst", "docs_url": null, "download_url": "", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "https://catalyst.coop/pudl", "keywords": "electricity,energy,data,analysis,mcoe,climate change,finance,eia 923,eia 860,ferc,form 1,epa ampd,epa cems,coal,natural gas", "license": "MIT", "maintainer": "Zane A. Selvans", "maintainer_email": "zane.selvans@catalyst.coop", "name": "catalystcoop.pudl", "package_url": "https://pypi.org/project/catalystcoop.pudl/", "platform": "", "project_url": "https://pypi.org/project/catalystcoop.pudl/", "project_urls": { "Documentation": "https://catalystcoop-pudl.readthedocs.io", "Homepage": "https://catalyst.coop/pudl", "Issue Tracker": "https://github.com/catalyst-cooperative/pudl/issues", "Source": "https://github.com/catalyst-cooperative/pudl" }, "release_url": "https://pypi.org/project/catalystcoop.pudl/0.2.0/", "requires_dist": [ "coloredlogs", "datapackage (>=1.9.0)", "dbfread", "goodtables", "matplotlib", "networkx (>=2.2)", "numpy", "pandas (>=0.24)", "pyarrow (>=0.14.0)", "pyyaml", "scikit-learn (>=0.20)", "scipy", "sqlalchemy (>=1.3.0)", "tableschema", "tableschema-sql (==1.1.0)", "timezonefinder", "xlsxwriter", "python-snappy", "doc8 ; extra == 'doc'", "sphinx ; extra == 'doc'", "sphinx-rtd-theme ; extra == 'doc'", "bandit ; extra == 'test'", "coverage ; extra == 'test'", "doc8 ; extra == 'test'", "flake8 ; extra == 'test'", "flake8-docstrings ; extra == 'test'", "flake8-builtins ; extra == 'test'", "pep8-naming ; extra == 'test'", "pre-commit ; extra == 'test'", "pydocstyle ; extra == 'test'", "pytest ; extra == 'test'", "pytest-cov ; extra == 'test'", "nbval ; extra == 'test'" ], "requires_python": ">=3.7, <3.8.0a0", "summary": "An open data processing pipeline for public US utility data.", "version": "0.2.0" }, "last_serial": 5846638, "releases": { "0.1.0": [ { "comment_text": "", "digests": { "md5": "00f6ab375fb943c64bfc4814876dab0d", "sha256": "ad37a75f0e336f48df63be16ec39ef3400fe9ff013fc33caf613f0a774f7de0a" }, "downloads": -1, "filename": "catalystcoop.pudl-0.1.0-py3-none-any.whl", "has_sig": true, "md5_digest": "00f6ab375fb943c64bfc4814876dab0d", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": ">=3.7, <3.8.0a0", "size": 5931034, "upload_time": "2019-09-12T23:35:55", "url": "https://files.pythonhosted.org/packages/1d/1e/e5fea7af6f29453ed36f9c24a0a18c398ebad9127d3a1df5b2319f1b8da0/catalystcoop.pudl-0.1.0-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "8befb4377f01c839a6899c84aae395fc", "sha256": "ab6925701b57625b67027fefa00639e12aad476dd7493383af527415a4115289" }, "downloads": -1, "filename": "catalystcoop.pudl-0.1.0.tar.gz", "has_sig": true, "md5_digest": "8befb4377f01c839a6899c84aae395fc", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.7, <3.8.0a0", "size": 6041409, "upload_time": "2019-09-12T23:36:16", "url": "https://files.pythonhosted.org/packages/c6/5c/20e309ced26764c215f72806b2f0e8a09b9310aea2f9ea0681c53bfd696d/catalystcoop.pudl-0.1.0.tar.gz" } ], "0.1.0a1": [ { "comment_text": "", "digests": { "md5": "e7b3b05505d7d30bdbaf5aee22ef6c61", "sha256": "01a2db43b2dd5387c4cc54047a183575388901583dd996b49a8c6decc681ae9b" }, "downloads": -1, "filename": "catalystcoop.pudl-0.1.0a1-py3-none-any.whl", "has_sig": true, "md5_digest": "e7b3b05505d7d30bdbaf5aee22ef6c61", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": ">=3.7, <3.8.0a0", "size": 5939582, "upload_time": "2019-09-10T16:38:12", "url": "https://files.pythonhosted.org/packages/c3/57/cbb73bb309f80e6a5f89ae24e0aa3ab3b8c36519e91ec7ae19e2bc3e03da/catalystcoop.pudl-0.1.0a1-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "59d664d916781bcd06778b63d4e6fe76", "sha256": "1bd672d4914d0f38c0a44fb433b4efad1c62367b2a5d87fdfc3251303ec070a7" }, "downloads": -1, "filename": "catalystcoop.pudl-0.1.0a1.tar.gz", "has_sig": true, "md5_digest": "59d664d916781bcd06778b63d4e6fe76", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.7, <3.8.0a0", "size": 6050716, "upload_time": "2019-09-10T16:38:34", "url": "https://files.pythonhosted.org/packages/6d/1f/c020312b562ca8a28567d360744863541426a9ef556a4121ef9d4835eb66/catalystcoop.pudl-0.1.0a1.tar.gz" } ], "0.1.0a2": [ { "comment_text": "", "digests": { "md5": "0b8055dbc59d75386dfd3001218215ab", "sha256": "485cbf7ff94478202c02d44ee7195893d3773fb6fb723d320c643d5240685f6c" }, "downloads": -1, "filename": "catalystcoop.pudl-0.1.0a2-py3-none-any.whl", "has_sig": true, "md5_digest": "0b8055dbc59d75386dfd3001218215ab", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": ">=3.7, <3.8.0a0", "size": 5939600, "upload_time": "2019-09-11T00:40:56", "url": "https://files.pythonhosted.org/packages/88/cf/017b31000a349235030bc373cbb6b3e3a6142e3422f5b5cd7838a5ccd8d1/catalystcoop.pudl-0.1.0a2-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "b19d901872d06da443d02fbfdd179e7e", "sha256": "b6912239152a7c942aa55d2a87473151d3137015ab4992414164b46d5f343f53" }, "downloads": -1, "filename": "catalystcoop.pudl-0.1.0a2.tar.gz", "has_sig": true, "md5_digest": "b19d901872d06da443d02fbfdd179e7e", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.7, <3.8.0a0", "size": 6050761, "upload_time": "2019-09-11T00:44:07", "url": "https://files.pythonhosted.org/packages/63/93/4db93ef55dc7cc89fd7a42c0187a5ead4cf3a157ded0d080b2eb26f419c0/catalystcoop.pudl-0.1.0a2.tar.gz" } ], "0.1.0a3": [ { "comment_text": "", "digests": { "md5": "e1bd87bdd85ff28067a532aefc512fa7", "sha256": "b070a41eff63fb5ce878c46ea8deb7694797a47b6fca9cae2b38d758f70c38c0" }, "downloads": -1, "filename": "catalystcoop.pudl-0.1.0a3-py3-none-any.whl", "has_sig": true, "md5_digest": "e1bd87bdd85ff28067a532aefc512fa7", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": ">=3.7, <3.8.0a0", "size": 5939926, "upload_time": "2019-09-11T23:10:18", "url": "https://files.pythonhosted.org/packages/87/6f/e7aa09464943281155f0c15370a22ced1487148f6e2ff00387d2df2512ef/catalystcoop.pudl-0.1.0a3-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "ec30b69034463720a70207de3af39763", "sha256": "8b90fe49a3ed4915545c098ddc6fe09dcdf2d567b99bb8cf007a7f22ae70ae60" }, "downloads": -1, "filename": "catalystcoop.pudl-0.1.0a3.tar.gz", "has_sig": true, "md5_digest": "ec30b69034463720a70207de3af39763", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.7, <3.8.0a0", "size": 6051117, "upload_time": "2019-09-11T23:10:39", "url": "https://files.pythonhosted.org/packages/94/29/af511cdc3b8d3f95dc07486cad12321f0d267f8fba5771533b7871463f62/catalystcoop.pudl-0.1.0a3.tar.gz" } ], "0.1.0a4": [ { "comment_text": "", "digests": { "md5": "159b5fef77385530a3505b79706d1f02", "sha256": "10feca4738056cf7e5945a561bb79a65996052cbc7d6f25b80aafa5738fabf4f" }, "downloads": -1, "filename": "catalystcoop.pudl-0.1.0a4-py3-none-any.whl", "has_sig": true, "md5_digest": "159b5fef77385530a3505b79706d1f02", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": ">=3.7, <3.8.0a0", "size": 5940041, "upload_time": "2019-09-12T02:40:35", "url": "https://files.pythonhosted.org/packages/9d/30/effaaaff0774cc4c5f1962dc5168c5827890070305e54571be950a644fb6/catalystcoop.pudl-0.1.0a4-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "e7fbd7d85d3c521a9aa8b6c081550ee9", "sha256": "d3bee543db42a7f3f75e623c0195368ea014a3047d51125e957d48c799107307" }, "downloads": -1, "filename": "catalystcoop.pudl-0.1.0a4.tar.gz", "has_sig": true, "md5_digest": "e7fbd7d85d3c521a9aa8b6c081550ee9", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.7, <3.8.0a0", "size": 6051254, "upload_time": "2019-09-12T02:43:35", "url": "https://files.pythonhosted.org/packages/54/a3/d4e75542df27d5cea853cb2db3a3a196b3b8582ca5a32bc4570c04dda6a8/catalystcoop.pudl-0.1.0a4.tar.gz" } ], "0.1.0rc1": [ { "comment_text": "", "digests": { "md5": "c8ebd79595d845a07d1a978b27b14e63", "sha256": "1b19c87a55895da304471aac2dc4c140f1ea96a28ca213643f3057c8669ec328" }, "downloads": -1, "filename": "catalystcoop.pudl-0.1.0rc1-py3-none-any.whl", "has_sig": true, "md5_digest": "c8ebd79595d845a07d1a978b27b14e63", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": ">=3.7, <3.8.0a0", "size": 5931045, "upload_time": "2019-09-12T20:08:01", "url": "https://files.pythonhosted.org/packages/74/7e/e703f63c60dea4961756864d03151b105bd397da2621a3169d2c73d462b3/catalystcoop.pudl-0.1.0rc1-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "962e7c50f8dfb23cfb951124efb33eac", "sha256": "5afcbf6f0b2084e1342303f6fc82042668aa108c1e183ea3efc0586e14001602" }, "downloads": -1, "filename": "catalystcoop.pudl-0.1.0rc1.tar.gz", "has_sig": true, "md5_digest": "962e7c50f8dfb23cfb951124efb33eac", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.7, <3.8.0a0", "size": 6041466, "upload_time": "2019-09-12T20:08:23", "url": "https://files.pythonhosted.org/packages/a7/e9/f9e61c73c515e3392ee80b0d85a6ccb07248d3be2e03e4c48f094d0e377f/catalystcoop.pudl-0.1.0rc1.tar.gz" } ], "0.2.0": [ { "comment_text": "", "digests": { "md5": "0f408bec597372002c0f8a71cd461178", "sha256": "2e8257e0809f9bb9af6e01809427cd9b734ee58aa29f1b29624750854d849777" }, "downloads": -1, "filename": "catalystcoop.pudl-0.2.0-py3-none-any.whl", "has_sig": true, "md5_digest": "0f408bec597372002c0f8a71cd461178", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": ">=3.7, <3.8.0a0", "size": 5923222, "upload_time": "2019-09-18T02:27:49", "url": "https://files.pythonhosted.org/packages/1a/04/0746f5115618ad0dea8cb2a72f8b023a335347cb9dec217d63193837d095/catalystcoop.pudl-0.2.0-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "367defa1b5d315f9d5277fc5efe803eb", "sha256": "b388d9c18066250dd4018b57b26f0e4d0ef4f142319905b6f70d73f5c52fa347" }, "downloads": -1, "filename": "catalystcoop.pudl-0.2.0.tar.gz", "has_sig": true, "md5_digest": "367defa1b5d315f9d5277fc5efe803eb", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.7, <3.8.0a0", "size": 6039489, "upload_time": "2019-09-18T02:29:37", "url": "https://files.pythonhosted.org/packages/42/96/380bcff91b3adb3131b1034bc3bc4b5c40e1e70ac9e6d426e93ff8b70f0d/catalystcoop.pudl-0.2.0.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "0f408bec597372002c0f8a71cd461178", "sha256": "2e8257e0809f9bb9af6e01809427cd9b734ee58aa29f1b29624750854d849777" }, "downloads": -1, "filename": "catalystcoop.pudl-0.2.0-py3-none-any.whl", "has_sig": true, "md5_digest": "0f408bec597372002c0f8a71cd461178", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": ">=3.7, <3.8.0a0", "size": 5923222, "upload_time": "2019-09-18T02:27:49", "url": "https://files.pythonhosted.org/packages/1a/04/0746f5115618ad0dea8cb2a72f8b023a335347cb9dec217d63193837d095/catalystcoop.pudl-0.2.0-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "367defa1b5d315f9d5277fc5efe803eb", "sha256": "b388d9c18066250dd4018b57b26f0e4d0ef4f142319905b6f70d73f5c52fa347" }, "downloads": -1, "filename": "catalystcoop.pudl-0.2.0.tar.gz", "has_sig": true, "md5_digest": "367defa1b5d315f9d5277fc5efe803eb", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.7, <3.8.0a0", "size": 6039489, "upload_time": "2019-09-18T02:29:37", "url": "https://files.pythonhosted.org/packages/42/96/380bcff91b3adb3131b1034bc3bc4b5c40e1e70ac9e6d426e93ff8b70f0d/catalystcoop.pudl-0.2.0.tar.gz" } ] }