{ "info": { "author": "data.world", "author_email": "help@data.world", "bugtrack_url": null, "classifiers": [ "Programming Language :: Python :: 3 :: Only" ], "description": "============\ntap-redshift\n============\n\n\n`Singer `_ tap that extracts data from a `Redshift `_ database and produces JSON-formatted data following the Singer spec.\n\n\nUsage\n=====\ntap-redshift assumes you have a connection to Redshift and requires Python 3.6+.\n\nStep 1: Create a configuration file\n-----------------------------------\nWhen you install tap-redshift, you need to create a ``config.json`` file for the database connection.\n\nThe json file requires the following attributes;\n\n* ``host``\n* ``port``\n* ``dbname``\n* ``user``\n* ``password``\n* ``start_date`` (Notation: yyyy-mm-ddThh:mm:ssZ)\n\nAnd an optional attribute;\n\n* ``schema``\n\nExample:\n\n.. code-block:: json\n\n {\n \"host\": \"REDSHIFT_HOST\",\n \"port\": \"REDSHIFT_PORT\",\n \"dbname\": \"REDSHIFT_DBNAME\",\n \"user\": \"REDSHIFT_USER\",\n \"password\": \"REDSHIFT_PASSWORD\",\n \"start_date\": \"REDSHIFT_START_DATE\",\n \"schema\": \"REDSHIFT_SCHEMA\"\n }\n\n\nStep 2: Discover what can be extracted from Redshift\n----------------------------------------------------\nThe tap can be invoked in discovery mode to get the available tables and columns in the database.\nIt points to the config file created to connect to redshift:\n\n.. code-block:: shell\n\n $ tap-redshift --config config.json -d\n\nA full catalog tap is written to stdout, with a JSON-schema description of each table. A source\ntable directly corresponds to a Singer stream.\n\nRedirect output from the tap's discovery mode to a file so that it can be modified when the tap is\nto be invoked in sync mode.\n\n.. code-block:: shell\n\n $ tap-redshift -c config.json -d > catalog.json\n\nThis runs the tap in discovery mode and copies the output into a ``catalog.json`` file.\n\nA catalog contains a list of stream objects, one for each table available in your Redshift schema.\n\nExample:\n\n.. code-block:: json\n\n {\n \"streams\": [\n {\n \"tap_stream_id\": \"sample-dbname.public.sample-name\",\n \"stream\": \"sample-stream\",\n \"database_name\": \"sample-dbname\",\n \"table_name\": \"public.sample-name\"\n \"schema\": {\n \"properties\": {\n \"id\": {\n \"minimum\": -2147483648,\n \"inclusion\": \"automatic\",\n \"maximum\": 2147483647,\n \"type\": [\n \"null\",\n \"integer\"\n ]\n },\n \"name\": {\n \"maxLength\": 255,\n \"inclusion\": \"available\",\n \"type\": [\n \"null\",\n \"string\"\n ]\n },\n \"updated_at\": {\n \"inclusion\": \"available\",\n \"type\": [\n \"string\"\n ],\n \"format\": \"date-time\"\n },\n },\n \"type\": \"object\"\n },\n \"metadata\": [\n {\n \"metadata\": {\n \"selected-by-default\": false,\n \"selected\": true,\n \"is-view\": false,\n \"table-key-properties\": [\"id\"],\n \"schema-name\": \"sample-stream\",\n \"valid-replication-keys\": [\n \"updated_at\"\n ]\n },\n \"breadcrumb\": [],\n },\n {\n \"metadata\": {\n \"selected-by-default\": true,\n \"sql-datatype\": \"int2\",\n \"inclusion\": \"automatic\"\n },\n \"breadcrumb\": [\n \"properties\",\n \"id\"\n ]\n },\n {\n \"metadata\": {\n \"selected-by-default\": true,\n \"sql-datatype\": \"varchar\",\n \"inclusion\": \"available\"\n },\n \"breadcrumb\": [\n \"properties\",\n \"name\"\n ]\n },\n {\n \"metadata\": {\n \"selected-by-default\": true,\n \"sql-datatype\": \"datetime\",\n \"inclusion\": \"available\",\n },\n \"breadcrumb\": [\n \"properties\",\n \"updated_at\"\n ]\n }\n ]\n }\n ]\n }\n\n\nStep 3: Select the tables you want to sync\n------------------------------------------\nIn sync mode, ``tap-redshift`` requires a catalog file to be supplied, where the user must\nhave selected which streams (tables) should be transferred. Streams are not selected by default.\n\nFor each stream in the catalog, find the ``metadata`` section. That is the section you will modify\nto select the stream and, optionally, individual properties too.\n\nThe stream itself is represented by an empty breadcrumb.\n\nExample:\n\n.. code-block:: json\n\n \"metadata\": [\n {\n \"breadcrumb\": [],\n \"metadata\": {\n \"selected-by-default\": false,\n ...\n }\n }\n ]\n\nYou can select it by adding ``\"selected\": true`` to its metadata.\n\nExample:\n\n.. code-block:: json\n\n \"metadata\": [\n {\n \"breadcrumb\": [],\n \"metadata\": {\n \"selected\": true,\n \"selected-by-default\": false,\n ...\n }\n }\n ]\n\nThe tap can then be invoked in sync mode with the properties catalog argument:\n\nExample (paired with ``target-datadotworld``)\n\n.. code-block:: shell\n\n tap-redshift -c config.json --catalog catalog.json | target-datadotworld -c config-dw.json\n\n\nStep 4: Sync your data\n----------------------\nThere are two ways to replicate a given table. FULL_TABLE and INCREMENTAL.\nFULL_TABLE replication is used by default.\n\nFull Table\n++++++++++\nFull-table replication extracts all data from the source table each time the tap is invoked without\na state file.\n\nIncremental\n+++++++++++\nIncremental replication works in conjunction with a state file to only extract new records each\ntime the tap is invoked i.e continue from the last synced data.\n\nTo use incremental replication, we need to add the ``replication_method`` and ``replication_key``\nto the streams (tables) metadata in the ``catalog.json`` file.\n\nExample:\n\n.. code-block:: json\n\n \"metadata\": [\n {\n \"breadcrumb\": [],\n \"metadata\": {\n \"selected\": true,\n \"selected-by-default\": false,\n \"replication-method\": \"INCREMENTAL\",\n \"replication-key\": \"updated_at\",\n ...\n }\n }\n ]\n\nWe can then invoke the tap again in sync mode. This time the output will have ``STATE`` messages\nthat contains a ``replication_key_value`` and ``bookmark`` for data that were extracted.\n\nRedirect the output to a ``state.json`` file. Normally, the target will echo the last STATE after\nit has finished processing data.\n\nRun the code below to pass the state into a ``state.json`` file.\n\nExample:\n\n.. code-block:: shell\n\n tap-redshift -c config.json --catalog catalog.json | \\\n target-datadotworld -c config-dw.json > state.json\n\nThe ``state.json`` file should look like;\n\n.. code-block:: json\n\n {\n \"currently_syncing\": null,\n \"bookmarks\": {\n \"sample-dbname.public.sample-name\": {\n \"replication_key\": \"updated_at\",\n \"version\": 1516304171710,\n \"replication_key_value\": \"2013-10-29T09:38:41.341Z\"\n }\n }\n }\n\nFor subsequent runs, you can then invoke the incremental replication passing the latest state in order to limit data only to what has been modified since the last execution.\n\n.. code-block:: shell\n\n tail -1 state.json > latest-state.json; \\\n tap-redshift \\\n -c config-redshift.json \\\n --catalog catalog.json \\\n\t -s latest-state.json | \\\n\t target-datadotworld -c config-dw.json > state.json\n\n\nAll steps in one Makefile\n=========================\n\nFor your convenience, all the steps mentioned above are captured in the ``Makefile`` below.\nThis example uses ``target-datadotworld`` but can be modified to use any other Singer target.\n\n.. code-block:: Makefile\n\n # Requires python 3.6\n install:\n pip3 install tap-redshift; \\\n pip3 install target-datadotworld\n\n # Catalog discovery\n discover:\n tap-redshift \\\n -c config-redshift.json -d > catalog.json\n\n # Full sync\n fullsync:\n tap-redshift \\\n -c config-redshift.json \\\n --catalog catalog.json | \\\n target-datadotworld -c config-dw.json > state.json\n\n # Incremental sync\n sync:\n tail -1 state.json > latest-state.json; \\\n tap-redshift \\\n -c config-redshift.json \\\n --catalog catalog.json \\\n -s latest-state.json | \\\n target-datadotworld -c config-dw.json > state.json\n\n\n", "description_content_type": "", "docs_url": null, "download_url": "", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "", "keywords": "", "license": "", "maintainer": "", "maintainer_email": "", "name": "tap-redshift", "package_url": "https://pypi.org/project/tap-redshift/", "platform": "", "project_url": "https://pypi.org/project/tap-redshift/", "project_urls": null, "release_url": "https://pypi.org/project/tap-redshift/1.0.0b9/", "requires_dist": [ "attrs (==18.2.0)", "pendulum (==1.2.0)", "singer-python (==5.0.4)", "backoff (==1.3.2)", "psycopg2 (==2.7.3.2)" ], "requires_python": "", "summary": "Singer.io tap for extracting data from redshift", "version": "1.0.0b9" }, "last_serial": 4317604, "releases": { "1.0.0b1": [ { "comment_text": "", "digests": { "md5": "386b09874dd28b9efa926ffd31560d19", "sha256": "91691d7dd35d37d7322086b79a5f7f883ef637810297c1a362666f323369812e" }, "downloads": -1, "filename": "tap_redshift-1.0.0b1-py2.py3-none-any.whl", "has_sig": false, "md5_digest": "386b09874dd28b9efa926ffd31560d19", "packagetype": "bdist_wheel", "python_version": "py2.py3", "requires_python": null, "size": 11776, "upload_time": "2018-01-19T21:52:32", "url": "https://files.pythonhosted.org/packages/c0/c4/39ccff3aa01b057cd70d23f431cc883c018ddeafb91418e4e2ceaafffed5/tap_redshift-1.0.0b1-py2.py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "4d66433048ce179afb6aec730de442b9", "sha256": "efcceb17e041c5c257a1320c5c98866289f6b1056c99b667b98b61adf3166d77" }, "downloads": -1, "filename": "tap-redshift-1.0.0b1.tar.gz", "has_sig": false, "md5_digest": "4d66433048ce179afb6aec730de442b9", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 13134, "upload_time": "2018-01-19T21:52:59", "url": "https://files.pythonhosted.org/packages/c2/6d/020e8139c7b8e7ea49b8a7551dc2fe85cfb42915d2a222fb7e11f93fb8c7/tap-redshift-1.0.0b1.tar.gz" } ], "1.0.0b2": [ { "comment_text": "", "digests": { "md5": "8acf8907cfbe974b90ed9dce642fbe53", "sha256": "ca365c02bba05f11845869bfbcfc012b56ae2da9b325a65f849e3e5fdd7b7c4a" }, "downloads": -1, "filename": "tap_redshift-1.0.0b2-py2.py3-none-any.whl", "has_sig": false, "md5_digest": "8acf8907cfbe974b90ed9dce642fbe53", "packagetype": "bdist_wheel", "python_version": "py2.py3", "requires_python": null, "size": 12589, "upload_time": "2018-02-16T03:25:44", "url": "https://files.pythonhosted.org/packages/70/c0/5f07f06579c5369a800f2fa5b5b9c5341826496b5607819d2b639f71566b/tap_redshift-1.0.0b2-py2.py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "6c91da8742d0d8673066782871b966ea", "sha256": "bcf86723b5e650b88287b5a466af99d61579c18705c21b808b28a55e13b05891" }, "downloads": -1, "filename": "tap-redshift-1.0.0b2.tar.gz", "has_sig": false, "md5_digest": "6c91da8742d0d8673066782871b966ea", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 12710, "upload_time": "2018-02-16T03:25:45", "url": "https://files.pythonhosted.org/packages/59/bb/82ac238d1ec710fff5e6a74e742f2daabe2dbe611382a6fb331772e23959/tap-redshift-1.0.0b2.tar.gz" } ], "1.0.0b3": [ { "comment_text": "", "digests": { "md5": "08ee26310e3e2a277e19754dc7e9c539", "sha256": "8e454ecbf808c12d95139c7e1c5d52ea47b19c1b1f87f25cb7000795f48abf3a" }, "downloads": -1, "filename": "tap_redshift-1.0.0b3-py2.py3-none-any.whl", "has_sig": false, "md5_digest": "08ee26310e3e2a277e19754dc7e9c539", "packagetype": "bdist_wheel", "python_version": "py2.py3", "requires_python": null, "size": 13451, "upload_time": "2018-02-16T21:37:14", "url": "https://files.pythonhosted.org/packages/ad/c5/6c42ffd304cf355b4a2758be81c48e15250dcb1a89c19092aba83699210c/tap_redshift-1.0.0b3-py2.py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "c5a3d197098af6c879f8e3d93e1f88cb", "sha256": "f40dec0abbfa568e165710136bd0ed2d1426b2793618ad316a510a309358148e" }, "downloads": -1, "filename": "tap-redshift-1.0.0b3.tar.gz", "has_sig": false, "md5_digest": "c5a3d197098af6c879f8e3d93e1f88cb", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 15798, "upload_time": "2018-02-16T21:37:15", "url": "https://files.pythonhosted.org/packages/72/a8/835e3b931298660f5745b40390196a99062297bc4d4f9fe8338556ead1c5/tap-redshift-1.0.0b3.tar.gz" } ], "1.0.0b4": [ { "comment_text": "", "digests": { "md5": "deab23a1a3c458b8ddeb457cae22708c", "sha256": "42e2994ab1352590da9bdd65f655a1e968dbccbc8ae8e41906c5fb6d4d9ad846" }, "downloads": -1, "filename": "tap_redshift-1.0.0b4-py2.py3-none-any.whl", "has_sig": false, "md5_digest": "deab23a1a3c458b8ddeb457cae22708c", "packagetype": "bdist_wheel", "python_version": "py2.py3", "requires_python": null, "size": 13576, "upload_time": "2018-04-02T22:26:48", "url": "https://files.pythonhosted.org/packages/ff/0a/3190bb7b01fca2d0ce886c95969c6228a99ec845aa03eab19c79f9661d05/tap_redshift-1.0.0b4-py2.py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "79636e62fbf306a036393e2dde1fe605", "sha256": "fb801471c1121298728b363df1fccc5e26eab3233d4606c55234d4dcbb6e3f96" }, "downloads": -1, "filename": "tap-redshift-1.0.0b4.tar.gz", "has_sig": false, "md5_digest": "79636e62fbf306a036393e2dde1fe605", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 13562, "upload_time": "2018-04-02T22:26:49", "url": "https://files.pythonhosted.org/packages/da/84/13900c556dedfa5f8f2623d4315c04aaa898cdef061ac34e0e18c22afdc3/tap-redshift-1.0.0b4.tar.gz" } ], "1.0.0b5": [ { "comment_text": "", "digests": { "md5": "539d6df59b6a4f925a06a4753d9c57f8", "sha256": "36a36e86c85fddc6c572985ce9e658ab8f5b910b1a0130b0a9ed93285d99b31b" }, "downloads": -1, "filename": "tap_redshift-1.0.0b5-py2.py3-none-any.whl", "has_sig": false, "md5_digest": "539d6df59b6a4f925a06a4753d9c57f8", "packagetype": "bdist_wheel", "python_version": "py2.py3", "requires_python": null, "size": 13768, "upload_time": "2018-05-23T20:47:29", "url": "https://files.pythonhosted.org/packages/e8/fb/55cb1796e872d29be208b44413764b33c4d3219a380aada82037fa4ecc39/tap_redshift-1.0.0b5-py2.py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "16e1b7f8fb4dda9ba4e5cda48ac895fb", "sha256": "27c97c3b96a8e11d5e4bf6cca5d445ddf2db48b6a414cdd6a56b45f36da05f55" }, "downloads": -1, "filename": "tap-redshift-1.0.0b5.tar.gz", "has_sig": false, "md5_digest": "16e1b7f8fb4dda9ba4e5cda48ac895fb", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 15522, "upload_time": "2018-05-23T20:47:30", "url": "https://files.pythonhosted.org/packages/b7/39/4d41a9807c567e374af0d160cffbe127c4ea9dc326d591813c16b5b46cea/tap-redshift-1.0.0b5.tar.gz" } ], "1.0.0b6": [ { "comment_text": "", "digests": { "md5": "4a226de5b1a6094d7fcd886d7ca99280", "sha256": "175aca6470d3d83dfedb77a99a917da59fbad56eb230ee09c492acbbcaea0492" }, "downloads": -1, "filename": "tap_redshift-1.0.0b6-py2.py3-none-any.whl", "has_sig": false, "md5_digest": "4a226de5b1a6094d7fcd886d7ca99280", "packagetype": "bdist_wheel", "python_version": "py2.py3", "requires_python": null, "size": 13882, "upload_time": "2018-05-29T14:57:33", "url": "https://files.pythonhosted.org/packages/da/ef/a96b94bdc2ed3d4fbedfaa3fd730d68db8a613506c72d60a919f75650438/tap_redshift-1.0.0b6-py2.py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "dd58a90176cd36347598652f55d40fca", "sha256": "e327f86ad95b83e68abc9e48186634c8fee1afb9fdfd9f934106312d544d4d30" }, "downloads": -1, "filename": "tap-redshift-1.0.0b6.tar.gz", "has_sig": false, "md5_digest": "dd58a90176cd36347598652f55d40fca", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 14205, "upload_time": "2018-05-29T14:57:34", "url": "https://files.pythonhosted.org/packages/4a/b7/e97a6505d2383f5501a926c754a365ef8f86d76b572f52ea900deaa9a6d0/tap-redshift-1.0.0b6.tar.gz" } ], "1.0.0b8": [ { "comment_text": "", "digests": { "md5": "8d8fdd3978850d099b8143a3de490daf", "sha256": "bef2250acd76172979b033b8fbdeca9aad0934539c1d55ad65b50b7c1af57c49" }, "downloads": -1, "filename": "tap_redshift-1.0.0b8-py2.py3-none-any.whl", "has_sig": false, "md5_digest": "8d8fdd3978850d099b8143a3de490daf", "packagetype": "bdist_wheel", "python_version": "py2.py3", "requires_python": null, "size": 10652, "upload_time": "2018-06-13T14:15:17", "url": "https://files.pythonhosted.org/packages/cb/5b/752ba18855489e0d24f6d3861592dadfcc8ffef689405fed5f84020a68bb/tap_redshift-1.0.0b8-py2.py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "cf74f80c17e91656c05ac4dbb36cff18", "sha256": "4d40c00d54bae6595f362fa73b313fdbe9a6248d5a662c15ca83ea01a1a5871c" }, "downloads": -1, "filename": "tap-redshift-1.0.0b8.tar.gz", "has_sig": false, "md5_digest": "cf74f80c17e91656c05ac4dbb36cff18", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 16465, "upload_time": "2018-06-13T14:15:18", "url": "https://files.pythonhosted.org/packages/03/e6/b4db24a1b7588d9f9dea5054234de2ce9086fef97f8547cea7e0cc4ee520/tap-redshift-1.0.0b8.tar.gz" } ], "1.0.0b9": [ { "comment_text": "", "digests": { "md5": "cb68a40b75b5ef5087778ed52f1d0dea", "sha256": "771cc8176d841ef18bfe467da605fc6b6c2db1b9a1788678672ffd41e56dd693" }, "downloads": -1, "filename": "tap_redshift-1.0.0b9-py2.py3-none-any.whl", "has_sig": false, "md5_digest": "cb68a40b75b5ef5087778ed52f1d0dea", "packagetype": "bdist_wheel", "python_version": "py2.py3", "requires_python": null, "size": 10681, "upload_time": "2018-09-27T19:43:52", "url": "https://files.pythonhosted.org/packages/98/ea/31c57b85cc1c104f9ecbc46abb43346c86a117e0116db88a6dbc8dbc69b3/tap_redshift-1.0.0b9-py2.py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "96167e7903397620b54272fe4b24d2cd", "sha256": "9b3975271ac6600c1ea4e266773697b8cc83cac2a5efc190f21fa21b73af3e07" }, "downloads": -1, "filename": "tap-redshift-1.0.0b9.tar.gz", "has_sig": false, "md5_digest": "96167e7903397620b54272fe4b24d2cd", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 14194, "upload_time": "2018-09-27T19:43:53", "url": "https://files.pythonhosted.org/packages/a9/17/3417583b6b62f69d3458d0a4697cca8342d7db6e646d7b2f4049270fb648/tap-redshift-1.0.0b9.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "cb68a40b75b5ef5087778ed52f1d0dea", "sha256": "771cc8176d841ef18bfe467da605fc6b6c2db1b9a1788678672ffd41e56dd693" }, "downloads": -1, "filename": "tap_redshift-1.0.0b9-py2.py3-none-any.whl", "has_sig": false, "md5_digest": "cb68a40b75b5ef5087778ed52f1d0dea", "packagetype": "bdist_wheel", "python_version": "py2.py3", "requires_python": null, "size": 10681, "upload_time": "2018-09-27T19:43:52", "url": "https://files.pythonhosted.org/packages/98/ea/31c57b85cc1c104f9ecbc46abb43346c86a117e0116db88a6dbc8dbc69b3/tap_redshift-1.0.0b9-py2.py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "96167e7903397620b54272fe4b24d2cd", "sha256": "9b3975271ac6600c1ea4e266773697b8cc83cac2a5efc190f21fa21b73af3e07" }, "downloads": -1, "filename": "tap-redshift-1.0.0b9.tar.gz", "has_sig": false, "md5_digest": "96167e7903397620b54272fe4b24d2cd", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 14194, "upload_time": "2018-09-27T19:43:53", "url": "https://files.pythonhosted.org/packages/a9/17/3417583b6b62f69d3458d0a4697cca8342d7db6e646d7b2f4049270fb648/tap-redshift-1.0.0b9.tar.gz" } ] }