{
    "info": {
        "author": "The Narrativ Company, Inc.",
        "author_email": "engineering@narrativ.com",
        "bugtrack_url": null,
        "classifiers": [
            "Development Status :: 4 - Beta",
            "Intended Audience :: Developers",
            "License :: OSI Approved :: MIT License",
            "Natural Language :: English",
            "Programming Language :: Python :: 2",
            "Programming Language :: Python :: 2.7",
            "Programming Language :: Python :: 3",
            "Programming Language :: Python :: 3.5",
            "Programming Language :: Python :: 3.6"
        ],
        "description": "=========\nSpectrify\n=========\n\n\n.. image:: https://img.shields.io/pypi/v/spectrify.svg\n    :target: https://pypi.python.org/pypi/spectrify\n\n.. image:: https://img.shields.io/travis/hellonarrativ/spectrify.svg\n    :target: https://travis-ci.org/hellonarrativ/spectrify\n\n.. image:: https://readthedocs.org/projects/spectrify/badge/?version=latest\n    :target: https://spectrify.readthedocs.io/en/latest/?badge=latest\n    :alt: Documentation Status\n\n\nA simple yet powerful tool to move your data from Redshift to Redshift Spectrum.\n\n\n* Free software: MIT license\n* Documentation: https://spectrify.readthedocs.io.\n\n\nFeatures\n--------\n\nOne-liners to:\n\n* Export a Redshift table to S3 (CSV)\n* Convert exported CSVs to Parquet files in parallel\n* Create the Spectrum table on your Redshift cluster\n* **Perform all 3 steps in sequence**, essentially \"copying\" a Redshift table Spectrum in one command.\n\nS3 credentials are specified using boto3. See http://boto3.readthedocs.io/en/latest/guide/configuration.html\n\nRedshift credentials are supplied via environment variables, command-line parameters, or interactive prompt.\n\nInstall\n--------\n\n.. code-block:: bash\n\n    $ pip install spectrify\n\n\nCommand-line Usage\n------------------\n\nExport Redshift table `my_table` to a folder of CSV files on S3:\n\n.. code-block::\n\n    $ spectrify --host=example-url.redshift.aws.com --user=myuser --db=mydb export my_table \\\n        's3://example-bucket/my_table'\n\nConvert exported CSVs to Parquet:\n\n.. code-block::\n\n    $ spectrify --host=example-url.redshift.aws.com --user=myuser --db=mydb convert my_table \\\n        's3://example-bucket/my_table'\n\nCreate Spectrum table from S3 folder:\n\n.. code-block::\n\n    $ spectrify --host=example-url.redshift.aws.com --user=myuser --db=mydb create_table \\\n        's3://example-bucket/my_table' my_table my_spectrum_table\n\nTransform Redshift table by performing all 3 steps in sequence:\n\n.. code-block::\n\n    $ spectrify --host=example-url.redshift.aws.com --user=myuser --db=mydb transform my_table \\\n        's3://example-bucket/my_table'\n\n\nPython Usage\n------------\n\nExport to S3:\n\n.. code-block:: python\n\n\n    from spectrify.export import RedshiftDataExporter\n    RedshiftDataExporter(sa_engine, s3_config).export_to_csv('my_table')\n\nConvert exported CSVs to Parquet:\n\n.. code-block:: python\n\n    from spectrify.convert import ConcurrentManifestConverter\n    from spectrify.utils.schema import SqlAlchemySchemaReader\n    sa_table = SqlAlchemySchemaReader(engine).get_table_schema('my_table')\n    ConcurrentManifestConverter(sa_table, s3_config).convert_manifest()\n\nCreate Spectrum table from S3 parquet folder:\n\n.. code-block:: python\n\n    from spectrify.create import SpectrumTableCreator\n    from spectrify.utils.schema import SqlAlchemySchemaReader\n    sa_table = SqlAlchemySchemaReader(engine).get_table_schema('my_table')\n    SpectrumTableCreator(sa_engine, dest_schema, dest_table_name, sa_table, s3_config).create()\n\nTransform Redshift table by performing all 3 steps in sequence:\n\n.. code-block:: python\n\n    from spectrify.transform import TableTransformer\n    transformer = TableTransformer(engine, 'my_table', s3_config, dest_schema, dest_table_name)\n    transformer.transform()\n\nContribute\n----------\nContributions always welcome! Read our guide on contributing here: http://spectrify.readthedocs.io/en/latest/contributing.html\n\nLicense\n-------\nMIT License. Copyright (c) 2017, The Narrativ Company, Inc.\n\n\n=======\nHistory\n=======\n\n2.0.0 (2019-03-09)\n------------------\n\n* Default to 256MB files\n* Flag for unicode support on Python 2.7 (performance implications)\n* Drop support for Python 3.4\n* Support for additional CSV format parameters\n* Support for REAL data type\n\n\n1.0.1 (2018-07-12)\n------------------\n\n* Loosen version requirement for PyArrow\n* Add example script\n* Update documentation\n\n\n1.0.0 (2018-04-20)\n------------------\n\n* Move functionality into classes to make customizing behavior easier\n* Add support for DATE columns\n* Add support for DECIMAL/NUMERIC columns\n* Upgrade to pyarrow v0.9.0\n\n\n0.4.1 (2018-03-25)\n------------------\n\n* Fix exception when source table is not in schema public\n\n\n0.4.0 (2018-02-25)\n------------------\n\n* Upgrade to pyarrow v0.8.0\n* Verify Redshift column types are supported before attempting conversion\n* Bugfix: Properly clean up multiprocessing.pool resource\n\n\n0.3.0 (2017-10-30)\n------------------\n\n* Support 16- and 32-bit integers\n* Packaging updates\n\n\n0.2.1 (2017-09-27)\n------------------\n\n* Fix Readme\n\n\n0.2.0 (2017-09-27)\n------------------\n\n* First release on PyPI.\n\n\n0.1.0 (2017-09-13)\n------------------\n\n* Didn't even make it to PyPI.\n\n\n",
        "description_content_type": "",
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://github.com/hellonarrativ/spectrify",
        "keywords": "spectrify",
        "license": "MIT license",
        "maintainer": "",
        "maintainer_email": "",
        "name": "spectrify",
        "package_url": "https://pypi.org/project/spectrify/",
        "platform": "",
        "project_url": "https://pypi.org/project/spectrify/",
        "project_urls": {
            "Homepage": "https://github.com/hellonarrativ/spectrify"
        },
        "release_url": "https://pypi.org/project/spectrify/2.0.0/",
        "requires_dist": [
            "boto3",
            "ciso8601",
            "Click",
            "future",
            "pandas",
            "psycopg2",
            "pyarrow (>=0.9.0)",
            "python-dateutil (<2.7.0,>=2.1)",
            "s3fs",
            "sqlalchemy",
            "sqlalchemy-redshift (>=0.7.1)",
            "unicodecsv"
        ],
        "requires_python": "",
        "summary": "Tools for working with Redshift Spectrum.",
        "version": "2.0.0"
    },
    "last_serial": 4919658,
    "releases": {
        "0.2.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "57c66dc91fb777bc06c28326a82e29a5",
                    "sha256": "f13a7d4875a1f2827c3b71dee80bab4df51b80d4b35b69f485ac4cd692ad4cc6"
                },
                "downloads": -1,
                "filename": "spectrify-0.2.0-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "57c66dc91fb777bc06c28326a82e29a5",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 9971,
                "upload_time": "2017-09-27T22:00:49",
                "url": "https://files.pythonhosted.org/packages/50/b3/c73bcfeed6cb1fb147d6667dd1cdfec00da7a8818fbf6a961b0ce4a70f3f/spectrify-0.2.0-py2.py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "20d7bfcec68c72582322a24629920f2f",
                    "sha256": "6fc1beb16a52ec40043e8771939e23cf2847b9d772837c3c84a563b047cf48cb"
                },
                "downloads": -1,
                "filename": "spectrify-0.2.0.tar.gz",
                "has_sig": false,
                "md5_digest": "20d7bfcec68c72582322a24629920f2f",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 16835,
                "upload_time": "2017-09-27T22:00:51",
                "url": "https://files.pythonhosted.org/packages/53/8c/7378c2bfbcdb516c7d2bdb62dd81dbcd09749931f484dc98f48ac6e057c6/spectrify-0.2.0.tar.gz"
            }
        ],
        "0.2.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "ae2b103a37434fd65345e21f19231f3d",
                    "sha256": "1ea949be54880d8abdf4e80ddcd72585410521f1ce55dad57baa01fa7657f838"
                },
                "downloads": -1,
                "filename": "spectrify-0.2.1-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "ae2b103a37434fd65345e21f19231f3d",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 11614,
                "upload_time": "2017-09-27T22:33:53",
                "url": "https://files.pythonhosted.org/packages/b9/2a/3e6bdb43c5fcff49137ae63fc34cae3a31f02ebb77e18b158c87afd6ff3d/spectrify-0.2.1-py2.py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "495ac0c470549fb56e949a523380a7b4",
                    "sha256": "92133ff5d07636702c3a3b4c548f43d9db395c39214c2c43487146a35c37fa77"
                },
                "downloads": -1,
                "filename": "spectrify-0.2.1.tar.gz",
                "has_sig": false,
                "md5_digest": "495ac0c470549fb56e949a523380a7b4",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 18181,
                "upload_time": "2017-09-27T22:33:54",
                "url": "https://files.pythonhosted.org/packages/b2/2b/26bee0abaa581621dad0a90ef36fda4beb875485903882942cceaae2bb20/spectrify-0.2.1.tar.gz"
            }
        ],
        "0.3.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "4c2f8cb4641bc86c0abc4c4a39b14dbb",
                    "sha256": "60b7afda308edefdcd8b6cc9df191f750f66ef1d715597e3f078ee46b97a51ac"
                },
                "downloads": -1,
                "filename": "spectrify-0.3.0-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "4c2f8cb4641bc86c0abc4c4a39b14dbb",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 15839,
                "upload_time": "2017-10-31T02:44:06",
                "url": "https://files.pythonhosted.org/packages/27/c2/1a1962cae890a38d2add4c40268fdf39f6ec4ce65b8117d3c5be915e0921/spectrify-0.3.0-py2.py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "08347a36e9782a87ca7b3603b950edcd",
                    "sha256": "aeb4fe485f3737247c85e57c68624cf4fca041a1584160c58acc76b2c0e9836c"
                },
                "downloads": -1,
                "filename": "spectrify-0.3.0.tar.gz",
                "has_sig": false,
                "md5_digest": "08347a36e9782a87ca7b3603b950edcd",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 20801,
                "upload_time": "2017-10-31T02:44:07",
                "url": "https://files.pythonhosted.org/packages/8a/c7/88c213b8d7b98018991d114fe696c58635ff45dabcca8f5021798cc5eb1f/spectrify-0.3.0.tar.gz"
            }
        ],
        "0.4.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "6619e4fb93ea00f2c2cc7ee5a628173f",
                    "sha256": "c3f9c8561504c7556c8260fe31c9d00b63a4c6395fb77c17c5c4b8dbe1c3897c"
                },
                "downloads": -1,
                "filename": "spectrify-0.4.0-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "6619e4fb93ea00f2c2cc7ee5a628173f",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 15969,
                "upload_time": "2018-02-25T20:17:45",
                "url": "https://files.pythonhosted.org/packages/46/41/40ff28c3677038cf74c1f1fff979f9c32a623ae9c350c1e7c4e7216a7c77/spectrify-0.4.0-py2.py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "dab2586b45b7b5936525ce6b221363b9",
                    "sha256": "d946032c7bbeb5c4b852b489bd82afa8c6845cf71b8aabdf14165f47a1705cb0"
                },
                "downloads": -1,
                "filename": "spectrify-0.4.0.tar.gz",
                "has_sig": false,
                "md5_digest": "dab2586b45b7b5936525ce6b221363b9",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 20669,
                "upload_time": "2018-02-25T20:17:47",
                "url": "https://files.pythonhosted.org/packages/d7/ba/c5daf7c75f01c6a510ab3f7fff8732bb6e95dcb27370f3448f66a1c9e499/spectrify-0.4.0.tar.gz"
            }
        ],
        "0.4.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "86cf627ca8712d10fece1e5eeec5876e",
                    "sha256": "e5e28e854fe2c24398033b60840bdfd12d42095066b417836b97a7a848d38282"
                },
                "downloads": -1,
                "filename": "spectrify-0.4.1-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "86cf627ca8712d10fece1e5eeec5876e",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 16204,
                "upload_time": "2018-03-25T22:24:22",
                "url": "https://files.pythonhosted.org/packages/99/be/9fe7283a88bea18c2ab13ba754933471e2ab7ab974d450393a626721df24/spectrify-0.4.1-py2.py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "9e47606e056c53e69616336c9edbce2b",
                    "sha256": "117b04d0b51fb111c39792705af0d3a1d9cf2699b9235ab36e065a4c8e3a07d2"
                },
                "downloads": -1,
                "filename": "spectrify-0.4.1.tar.gz",
                "has_sig": false,
                "md5_digest": "9e47606e056c53e69616336c9edbce2b",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 20885,
                "upload_time": "2018-03-25T22:24:23",
                "url": "https://files.pythonhosted.org/packages/e3/9e/633d607393bbe6075fd77f3c2b5af557fae8605a207ed56fd8a98badae03/spectrify-0.4.1.tar.gz"
            }
        ],
        "1.0.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "6d4487650ff60c4a5907f5b422ba2ef2",
                    "sha256": "5b66de2ed2f685d863c045f9ef2d7dc9385b26921c9ca390889b9becc209ee3c"
                },
                "downloads": -1,
                "filename": "spectrify-1.0.0-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "6d4487650ff60c4a5907f5b422ba2ef2",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 15116,
                "upload_time": "2018-04-20T18:57:51",
                "url": "https://files.pythonhosted.org/packages/bb/b4/b4b182b2fbc5189b6e8fcd5de3c954892a9c2e97ad68a329796cf9b36a5f/spectrify-1.0.0-py2.py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "f5a7f1ac88e4135ecf3c0228981914ee",
                    "sha256": "4c14396409b741de2b459f61038c2929cec0aff62c9affb124eab3e95ad5f45e"
                },
                "downloads": -1,
                "filename": "spectrify-1.0.0.tar.gz",
                "has_sig": false,
                "md5_digest": "f5a7f1ac88e4135ecf3c0228981914ee",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 22064,
                "upload_time": "2018-04-20T18:57:53",
                "url": "https://files.pythonhosted.org/packages/a5/00/f38e1546a46c2e171411f7d47517a433db4311850d5d7c8de3d0af69f371/spectrify-1.0.0.tar.gz"
            }
        ],
        "1.0.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "08c5f9ead1da6ff9a9e28c99b5bb87ef",
                    "sha256": "fa165cac630e610588e4a6c39e04d9ebbbfced229d92331adf465c976f7a196f"
                },
                "downloads": -1,
                "filename": "spectrify-1.0.1-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "08c5f9ead1da6ff9a9e28c99b5bb87ef",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 15161,
                "upload_time": "2018-07-12T21:10:08",
                "url": "https://files.pythonhosted.org/packages/ce/09/f016b38d020a85d82e07cc523d79b87857d54775f1be294aaf1331bd19cc/spectrify-1.0.1-py2.py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "3c7a376b7e7ba963f5ce5c976d26e7e2",
                    "sha256": "8ad95d30170c29ccb2043d61db0e35eb0f8be6bf472f2d51c2b2f10bdf205ed1"
                },
                "downloads": -1,
                "filename": "spectrify-1.0.1.tar.gz",
                "has_sig": false,
                "md5_digest": "3c7a376b7e7ba963f5ce5c976d26e7e2",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 23982,
                "upload_time": "2018-07-12T21:10:09",
                "url": "https://files.pythonhosted.org/packages/b5/e5/079ba9a7f7a5bdac1cd60fc4fc529d581a4e6e0dd0c7ed387501c812d230/spectrify-1.0.1.tar.gz"
            }
        ],
        "2.0.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "4d0c443cdf09dd6bd10b559dcd0c5a4a",
                    "sha256": "bf4362878e88dfbd94e1935ceda5fc6c36c48a605eeca0c4540b0487db382c24"
                },
                "downloads": -1,
                "filename": "spectrify-2.0.0-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "4d0c443cdf09dd6bd10b559dcd0c5a4a",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 17317,
                "upload_time": "2019-03-09T21:22:28",
                "url": "https://files.pythonhosted.org/packages/e7/b4/0352949296646bd302daa49f7ba7c5d6c2d97ce7bc32886dd0d1b9c54223/spectrify-2.0.0-py2.py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "6fcd579182ebb25ebea8ac1a78a71ba2",
                    "sha256": "6417ef3fae1540288c166b337fc3a68274b09f85d57d5100e566d2ca3c0b88ac"
                },
                "downloads": -1,
                "filename": "spectrify-2.0.0.tar.gz",
                "has_sig": false,
                "md5_digest": "6fcd579182ebb25ebea8ac1a78a71ba2",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 26198,
                "upload_time": "2019-03-09T21:22:30",
                "url": "https://files.pythonhosted.org/packages/88/2c/df7fe1d4be89f29e2b30178cabf7db0bf1762d767663b3ac842a01247152/spectrify-2.0.0.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "4d0c443cdf09dd6bd10b559dcd0c5a4a",
                "sha256": "bf4362878e88dfbd94e1935ceda5fc6c36c48a605eeca0c4540b0487db382c24"
            },
            "downloads": -1,
            "filename": "spectrify-2.0.0-py2.py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "4d0c443cdf09dd6bd10b559dcd0c5a4a",
            "packagetype": "bdist_wheel",
            "python_version": "py2.py3",
            "requires_python": null,
            "size": 17317,
            "upload_time": "2019-03-09T21:22:28",
            "url": "https://files.pythonhosted.org/packages/e7/b4/0352949296646bd302daa49f7ba7c5d6c2d97ce7bc32886dd0d1b9c54223/spectrify-2.0.0-py2.py3-none-any.whl"
        },
        {
            "comment_text": "",
            "digests": {
                "md5": "6fcd579182ebb25ebea8ac1a78a71ba2",
                "sha256": "6417ef3fae1540288c166b337fc3a68274b09f85d57d5100e566d2ca3c0b88ac"
            },
            "downloads": -1,
            "filename": "spectrify-2.0.0.tar.gz",
            "has_sig": false,
            "md5_digest": "6fcd579182ebb25ebea8ac1a78a71ba2",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 26198,
            "upload_time": "2019-03-09T21:22:30",
            "url": "https://files.pythonhosted.org/packages/88/2c/df7fe1d4be89f29e2b30178cabf7db0bf1762d767663b3ac842a01247152/spectrify-2.0.0.tar.gz"
        }
    ]
}