{ "info": { "author": "Faisal Dosani", "author_email": "faisal.dosani@capitalone.com", "bugtrack_url": null, "classifiers": [], "description": ".. image:: https://travis-ci.org/capitalone/Data-Load-and-Copy-using-Python.svg?branch=master\n :target: https://travis-ci.org/capitalone/Data-Load-and-Copy-using-Python\n.. image:: https://img.shields.io/badge/code%20style-black-000000.svg\n :target: https://github.com/ambv/black\n\nlocopy: Data Load and Copy using Python\n========================================\n\nA Python library to assist with ETL processing for:\n\n- Amazon Redshift (``COPY``, ``UNLOAD``)\n- Snowflake (``COPY INTO ``, ``COPY INTO ``)\n\nIn addition:\n\n- The library supports Python 3.5+\n- DB Driver (Adapter) agnostic. Use your favourite driver that complies with\n `DB-API 2.0 `_\n- It provides functionality to download and upload data to S3 buckets, and internal stages (Snowflake)\n\n\nQuick Installation\n==================\n\n.. code-block:: bash\n\n pip install locopy\n\n\nInstallation instructions\n-------------------------\n\nA virtual environment is highly recommended\n\n.. code-block:: bash\n\n $ virtualenv locopy\n $ source locopy/bin/activate\n $ pip install --upgrade setuptools pip\n $ pip install locopy\n\n\nPython Database API Specification 2.0\n-------------------------------------\n\nRather than using a specific Python DB Driver / Adapter for Postgres (which should supports Amazon\nRedshift or Snowflake), ``locopy`` prefers to be agnostic. As an end user you can use any Python\nDatabase API Specification 2.0 package.\n\nThe following packages have been tested:\n\n- ``psycopg2``\n- ``pg8000``\n- ``snowflake-connector-python``\n\nYou can use which ever one you prefer by importing the package and passing it\ninto the constructor input ``dbapi``.\n\n\n\nUsage\n-----\n\nYou need to store your connection parameters in a YAML file (or pass them in directly).\nThe YAML would consist of the following items:\n\n.. code-block:: yaml\n\n # required to connect to redshift\n host: my.redshift.cluster.com\n port: 5439\n database: db\n user: userid\n password: password\n ## optional extras for the dbapi connector\n sslmode: require\n another_option: 123\n\n\n\nIf you aren't loading data, you don't need to have AWS tokens set up.\nThe Redshift connection (``Redshift``) can be used like this:\n\n.. code-block:: python\n\n import pg8000\n import locopy\n\n with locopy.Redshift(dbapi=pg8000, config_yaml=\"config.yml\") as redshift:\n redshift.execute(\"SELECT * FROM schema.table\")\n df = redshift.to_dataframe()\n print(df)\n\n\nIf you want to load data to Redshift via S3, the ``Redshift`` class inherits from ``S3``:\n\n.. code-block:: python\n\n import pg8000\n import locopy\n\n with locopy.Redshift(dbapi=pg8000, config_yaml=\"config.yml\") as redshift:\n redshift.execute(\"SET query_group TO quick\")\n redshift.execute(\"CREATE TABLE schema.table (variable VARCHAR(20)) DISTKEY(variable)\")\n redshift.load_and_copy(\n local_file=\"example/example_data.csv\",\n s3_bucket=\"my_s3_bucket\",\n table_name=\"schema.table\",\n delim=\",\")\n redshift.execute(\"SELECT * FROM schema.table\")\n res = redshift.cursor.fetchall()\n\n print(res)\n\n\nIf you want to download data from Redshift to a CSV, or read it into Python\n\n.. code-block:: python\n\n my_profile = \"some_profile_with_valid_tokens\"\n with locopy.Redshift(dbapi=pg8000, config_yaml=\"config.yml\", profile=my_profile) as redshift:\n ##Optionally provide export if you ALSO want the exported data copied to a flat file\n redshift.unload_and_copy(\n query=\"SELECT * FROM schema.table\",\n s3_bucket=\"my_s3_bucket\",\n export_path=\"my_output_destination.csv\")\n\n\n\nNote on tokens\n^^^^^^^^^^^^^^\n\nTo load data to S3, you will need to be able to generate AWS tokens, or assume the IAM role on a EC2\ninstance. There are a few options for doing this, depending on where you're running your script and\nhow you want to handle tokens. Once you have your tokens, they need to be accessible to the AWS\ncommand line interface. See\nhttp://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html#config-settings-and-precedence\nfor more information, but you can:\n\n- Populate environment variables ``AWS_ACCESS_KEY_ID``, ``AWS_SECRET_ACCESS_KEY``,\n etc.\n- Leverage the AWS credentials file. If you have multiple profiles configured\n you can either call ``locopy.Redshift(profile=\"my-profile\")``, or set up an\n environment variable ``AWS_DEFAULT_PROFILE``.\n- If you are on a EC2 instance you can assume the credentials associated with the IAM role attached.\n\n\nAdvanced Usage\n--------------\n\nSee the `docs `_ for\nmore detailed usage instructions and examples including Snowflake.\n\n\nContributors\n------------\n\nWe welcome your interest in Capital One\u2019s Open Source Projects (the \"Project\").\nAny Contributor to the project must accept and sign a CLA indicating agreement to\nthe license terms. Except for the license granted in this CLA to Capital One and\nto recipients of software distributed by Capital One, you reserve all right, title,\nand interest in and to your contributions; this CLA does not impact your rights to\nuse your own contributions for any other purpose.\n\n- `Link to Individual CLA `_\n- `Link to Corporate CLA `_\n\nThis project adheres to the `Open Source Code of Conduct `_.\nBy participating, you are expected to honor this code.\n\n\n", "description_content_type": "", "docs_url": null, "download_url": "", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "https://github.com/capitalone/Data-Load-and-Copy-using-Python", "keywords": "", "license": "Apache Software License", "maintainer": "", "maintainer_email": "", "name": "locopy", "package_url": "https://pypi.org/project/locopy/", "platform": "", "project_url": "https://pypi.org/project/locopy/", "project_urls": { "Homepage": "https://github.com/capitalone/Data-Load-and-Copy-using-Python" }, "release_url": "https://pypi.org/project/locopy/0.3.1/", "requires_dist": [ "boto3 (==1.9.92)", "PyYAML (>=5.1)", "pandas (>=0.19.0)", "loguru (==0.2.5)", "pg8000 (==1.13.1) ; extra == 'pg8000'", "psycopg2 (==2.7.7) ; extra == 'psycopg2'", "snowflake-connector-python (==1.7.6) ; extra == 'snowflake'" ], "requires_python": "", "summary": "Loading/Unloading to Amazon Redshift using Python", "version": "0.3.1" }, "last_serial": 5248471, "releases": { "0.1.1": [ { "comment_text": "", "digests": { "md5": "2c424e63e17cfc4b99a748b786ed54ee", "sha256": "65fc22382fb5c115d73ed1adf0547914edcb85994a60fdddf57ff22f31a19cfc" }, "downloads": -1, "filename": "locopy-0.1.1-py3-none-any.whl", "has_sig": false, "md5_digest": "2c424e63e17cfc4b99a748b786ed54ee", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": null, "size": 18268, "upload_time": "2018-07-31T19:48:05", "url": "https://files.pythonhosted.org/packages/dc/0b/768f6c47d738a09fb2818b49378fec7cc39000d25278d694139aba1238e7/locopy-0.1.1-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "b47fd68c437df52aabaca1feb5910f1d", "sha256": "10c62bf8f4d6ca0c93fde4d4886fb817535a2a4c1597d14c0dee6082d14d47b2" }, "downloads": -1, "filename": "locopy-0.1.1.tar.gz", "has_sig": false, "md5_digest": "b47fd68c437df52aabaca1feb5910f1d", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 15624, "upload_time": "2018-07-31T19:48:06", "url": "https://files.pythonhosted.org/packages/b7/46/812679d2dbdcf66b8459e83aa359772a407a6c11e54ef46f430a42374b22/locopy-0.1.1.tar.gz" } ], "0.2.0": [ { "comment_text": "", "digests": { "md5": "db513b16dceff1cab36249a5d807727d", "sha256": "5ab1f16ae080c1693d0c65853dfb355898e43c7616203cb7261a800aeb55cf28" }, "downloads": -1, "filename": "locopy-0.2.0-py3-none-any.whl", "has_sig": false, "md5_digest": "db513b16dceff1cab36249a5d807727d", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": null, "size": 22607, "upload_time": "2018-11-21T13:46:23", "url": "https://files.pythonhosted.org/packages/60/8f/6f1cb9fd80844030855dba3ced8227621e54b41168373a7e0b2fb71e2bba/locopy-0.2.0-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "692980566456e7c92d640e7de275f21a", "sha256": "d0eda8737ac1627c04ef2388312755d56d06a44be30402eaa5b9378da647d53a" }, "downloads": -1, "filename": "locopy-0.2.0.tar.gz", "has_sig": false, "md5_digest": "692980566456e7c92d640e7de275f21a", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 16208, "upload_time": "2018-11-21T13:46:24", "url": "https://files.pythonhosted.org/packages/ee/40/4f9cd4e823d72454f2388fc23927f714a56fd10b0e1ccf1952665b05b4d2/locopy-0.2.0.tar.gz" } ], "0.3.0": [ { "comment_text": "", "digests": { "md5": "45459d4183732bd799f2ae29fc0bec7c", "sha256": "3f6aa3d7f43aa3828a3b5725d01c8bb80dd400090457319eb5654e4304d7c3f6" }, "downloads": -1, "filename": "locopy-0.3.0-py3-none-any.whl", "has_sig": false, "md5_digest": "45459d4183732bd799f2ae29fc0bec7c", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": null, "size": 31316, "upload_time": "2019-02-12T15:31:31", "url": "https://files.pythonhosted.org/packages/d7/84/476798cc09bfc3827e36bb51004b6150dee405d6e46d3711a936a527e392/locopy-0.3.0-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "8bef4478e8a965c613617314a930529d", "sha256": "e7095bbc8d9d16d2bf4730c9b3225211d03a5def65e09c98deeacde9551cdb44" }, "downloads": -1, "filename": "locopy-0.3.0.tar.gz", "has_sig": false, "md5_digest": "8bef4478e8a965c613617314a930529d", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 19190, "upload_time": "2019-02-12T15:31:33", "url": "https://files.pythonhosted.org/packages/40/81/26143cc1749113599ff5ef6ae20094a01e1531da43af4b9d98a61ba75407/locopy-0.3.0.tar.gz" } ], "0.3.1": [ { "comment_text": "", "digests": { "md5": "42fd39ba10ff8965303b0426f336914e", "sha256": "68a04dadc16eb0db0dabf014acd91e17d5d6a81d4161f7a2bac45ef653367ef3" }, "downloads": -1, "filename": "locopy-0.3.1-py3-none-any.whl", "has_sig": false, "md5_digest": "42fd39ba10ff8965303b0426f336914e", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": null, "size": 31302, "upload_time": "2019-04-03T02:29:23", "url": "https://files.pythonhosted.org/packages/d2/b2/ee3ba0dd6417682d45f05fd85a86ba3a16c0ac45d0095b790ea671be6fc5/locopy-0.3.1-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "bd89bddb2c3eee89dcc9b7b5ef5da5a4", "sha256": "ff8ed145f5281787919bd59b1dd6492065a1b13238e17a6f9ac3f8fd03e6c277" }, "downloads": -1, "filename": "locopy-0.3.1.tar.gz", "has_sig": false, "md5_digest": "bd89bddb2c3eee89dcc9b7b5ef5da5a4", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 19403, "upload_time": "2019-04-03T02:29:25", "url": "https://files.pythonhosted.org/packages/94/96/d8744b8d3d78cd480e2833e9b7d6aeee5752f74ab28f46b7a5585efc3115/locopy-0.3.1.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "42fd39ba10ff8965303b0426f336914e", "sha256": "68a04dadc16eb0db0dabf014acd91e17d5d6a81d4161f7a2bac45ef653367ef3" }, "downloads": -1, "filename": "locopy-0.3.1-py3-none-any.whl", "has_sig": false, "md5_digest": "42fd39ba10ff8965303b0426f336914e", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": null, "size": 31302, "upload_time": "2019-04-03T02:29:23", "url": "https://files.pythonhosted.org/packages/d2/b2/ee3ba0dd6417682d45f05fd85a86ba3a16c0ac45d0095b790ea671be6fc5/locopy-0.3.1-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "bd89bddb2c3eee89dcc9b7b5ef5da5a4", "sha256": "ff8ed145f5281787919bd59b1dd6492065a1b13238e17a6f9ac3f8fd03e6c277" }, "downloads": -1, "filename": "locopy-0.3.1.tar.gz", "has_sig": false, "md5_digest": "bd89bddb2c3eee89dcc9b7b5ef5da5a4", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 19403, "upload_time": "2019-04-03T02:29:25", "url": "https://files.pythonhosted.org/packages/94/96/d8744b8d3d78cd480e2833e9b7d6aeee5752f74ab28f46b7a5585efc3115/locopy-0.3.1.tar.gz" } ] }