{ "info": { "author": "Operation Pluto contributors", "author_email": "", "bugtrack_url": null, "classifiers": [ "Development Status :: 2 - Pre-Alpha", "Programming Language :: Python :: 3.5", "Topic :: Office/Business :: Financial :: Investment" ], "description": "Operation Pluto\n===============\n\n|PyPI version| |Codacy Badge|\n\n`Operation Pluto `__ is a\npipeline set-up. It plumbs financial and economic data. Focused markets\nare *Hong Kong*, *U.S.* and *China*.\n\nThis data pipeline is organized in `Luigi\nframework `__ with Python.\n\nAvailable Data\n--------------\n\nCurrently connected data sources :\n\nHong Kong\n~~~~~~~~~\n\n- `Census and Statistics Department `__\n- `The Hong Kong Association of Banks `__\n- `Hong Kong Government Bond Programme `__\n- `Hong Kong Monetary Authority `__\n- `Hang Seng Indexes Company `__\n- `Rating and Valuation Department `__\n\nUnited States\n~~~~~~~~~~~~~\n\n- `U.S. Bureau of Labor Statistics `__\n- `Federal Reserve System `__\n- `St.\u00a0Louis Federal Reserve Economic\n Data `__\n\nChina\n~~~~~\n\n- ?\n\nMaster Data\n~~~~~~~~~~~\n\n- `Holiday API `__\n\nPipeline Organization\n---------------------\n\n- Crawl websites, back-fill past data, and construct file directories.\n All done as code.\n- One table in data source corresponds to one target file.\n- Pipeline task is stateful. Overwrite source file the least possible.\n\nPrerequisites\n-------------\n\n- `Python 3.5 `__\n- `Luigi 2.7 `__\n\nGetting Started\n---------------\n\nHave Python 3.5 installed and clone this repository :\n\n::\n\n # Clone this repository\n $ git clone https://github.com/hydra-lab/operation-pluto\n\nInstall Python dependencies :\n\n::\n\n # Installing with Conda may not work\n $ pip install -r requirements.txt\n\nSet up Luigi configuration file :\n\n::\n\n # Rename luigi.cfg.sample to luigi.cfg\n $ mv luigi.cfg.sample luigi.cfg\n\nConfigure proxies in ``luigi.cfg`` if you\u2019re behind any :\n\n::\n\n [proxies]\n https = https://username:password@hostname:port/\n\nTest the installation. New data should be extracted and parsed into\nfolder ``test/data`` :\n\n::\n\n $ python -m luigi --module main RunMock --local-scheduler\n $ ls test/data\n\nHigh-level job orchestration is done in ``main.py``. e.g. ``RunAll()``\nis the wrapper class to initialize whole ``data`` directory and trigger\nall processing tasks. In production, tasks should be run on Luigi\nserver. Because Luigi daemon will not run on Windows, simply run :\n\n::\n\n # Run Luigi server on http://localhost:8082\n $ luigid\n # Run task on Luigi server\n $ python -m luigi --module main RunAll\n\nSchedule pipeline to run periodically in Task Scheduler or cron. Set up\n``run.sh`` on Windows :\n\n::\n\n # Script on Windows\n start luigid\n python -m luigi --module main RunAll\n cmd \"/c taskkill /IM \"luigid.exe\" /T /F\"\n\nLicense\n-------\n\n|License: AGPL v3|\n\nThis project is licensed under GNU Affero General Public License,\nVersion 3.0. See LICENSE for full license text.\n\n.. |PyPI version| image:: https://badge.fury.io/py/Operation-Pluto.svg\n :target: https://pypi.python.org/pypi/Operation-Pluto\n.. |Codacy Badge| image:: https://api.codacy.com/project/badge/Coverage/ae24c1a0b93a45bb972c40af136a01b2\n :target: https://www.codacy.com/app/tc-ying/Operation-Pluto-upstream?utm_source=github.com&utm_medium=referral&utm_content=hydra-lab/Operation-Pluto&utm_campaign=Badge_Coverage\n.. |License: AGPL v3| image:: https://img.shields.io/badge/License-AGPL%20v3-blue.svg\n :target: https://www.gnu.org/licenses/agpl-3.0", "description_content_type": null, "docs_url": null, "download_url": "", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "https://github.com/hydra-lab/Operation-Pluto", "keywords": "data-pipeline finance market", "license": "AGPL-3.0", "maintainer": "", "maintainer_email": "", "name": "Operation-Pluto", "package_url": "https://pypi.org/project/Operation-Pluto/", "platform": "", "project_url": "https://pypi.org/project/Operation-Pluto/", "project_urls": { "Homepage": "https://github.com/hydra-lab/Operation-Pluto" }, "release_url": "https://pypi.org/project/Operation-Pluto/0.1.1/", "requires_dist": null, "requires_python": ">=3.5", "summary": "Grab and rinse financial and economic data.", "version": "0.1.1" }, "last_serial": 3550684, "releases": { "0.1.0": [ { "comment_text": "", "digests": { "md5": "aa81c27173bca39ac32bd215ce7b1879", "sha256": "869d82c9ad6e871770b36d46d4fd4993fafb2afafcb91977322dc14df930af87" }, "downloads": -1, "filename": "Operation-Pluto-0.1.0.tar.gz", "has_sig": false, "md5_digest": "aa81c27173bca39ac32bd215ce7b1879", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.5", "size": 34780, "upload_time": "2018-02-04T05:33:15", "url": "https://files.pythonhosted.org/packages/df/0b/c7aa1fcc5f2560f963073661be45c5b979fa45285a25402be491b297f050/Operation-Pluto-0.1.0.tar.gz" } ], "0.1.1": [ { "comment_text": "", "digests": { "md5": "d22c3a4657d1785306c9573ad41e030a", "sha256": "f36742d2f2e356930834baa49c01752caab7feba02db5a0c78adcc72b9e361ea" }, "downloads": -1, "filename": "Operation-Pluto-0.1.1.tar.gz", "has_sig": false, "md5_digest": "d22c3a4657d1785306c9573ad41e030a", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.5", "size": 35158, "upload_time": "2018-02-04T16:27:45", "url": "https://files.pythonhosted.org/packages/4c/ab/a31bf07f801d4596cfecec1144b425b0a3125d4de0c1bb3f5eabbf974d4e/Operation-Pluto-0.1.1.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "d22c3a4657d1785306c9573ad41e030a", "sha256": "f36742d2f2e356930834baa49c01752caab7feba02db5a0c78adcc72b9e361ea" }, "downloads": -1, "filename": "Operation-Pluto-0.1.1.tar.gz", "has_sig": false, "md5_digest": "d22c3a4657d1785306c9573ad41e030a", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.5", "size": 35158, "upload_time": "2018-02-04T16:27:45", "url": "https://files.pythonhosted.org/packages/4c/ab/a31bf07f801d4596cfecec1144b425b0a3125d4de0c1bb3f5eabbf974d4e/Operation-Pluto-0.1.1.tar.gz" } ] }