{ "info": { "author": "Evan Borgstrom", "author_email": "eborgstrom@nerdwallet.com", "bugtrack_url": null, "classifiers": [ "Development Status :: 4 - Beta", "Intended Audience :: Developers", "License :: OSI Approved :: Apache Software License", "Natural Language :: English", "Programming Language :: Python", "Topic :: Internet", "Topic :: Software Development :: Libraries" ], "description": "Kinesis Python\n==============\n\n.. image:: https://img.shields.io/travis/NerdWalletOSS/kinesis-python.svg\n :target: https://travis-ci.org/NerdWalletOSS/kinesis-python\n\n.. image:: https://img.shields.io/codecov/c/github/NerdWalletOSS/kinesis-python.svg\n :target: https://codecov.io/github/NerdWalletOSS/kinesis-python\n\n.. image:: https://img.shields.io/pypi/v/kinesis-python.svg\n :target: https://pypi.python.org/pypi/kinesis-python\n :alt: Latest PyPI version\n\n\nThe `official Kinesis python library`_ requires the use of Amazon's \"MultiLangDaemon\", which is a Java executable that\noperates by piping messages over STDIN/STDOUT.\n\n.. code-block::\n\n \u0ca0_\u0ca0\n\nWhile the desire to have a single implementation of the client library from a maintenance standpoint makes sense for\nthe team responsible for the KPL, requiring the JRE to be installed and having to account for the overhead of the\nstream being consumed by Java and Python is not desireable for teams working in environments without Java.\n\nThis is a pure-Python implementation of Kinesis producer and consumer classes that leverages Python's multiprocessing\nmodule to spawn a process per shard and then sends the messages back to the main process via a Queue. It only depends\non `boto3`_ (AWS SDK), `offspring`_ (Subprocess implementation) and `six`_ (py2/py3 compatibility).\n\nIt also includes a DynamoDB state back-end that allows for multi-instance consumption of multiple shards, and stores the\ncheckpoint data so that you can resume where you left off in a stream following restarts or crashes.\n\n.. _boto3: https://pypi.python.org/pypi/boto3\n.. _offspring: https://pypi.python.org/pypi/offspring\n.. _six: https://pypi.python.org/pypi/six\n.. _official Kinesis python library: https://github.com/awslabs/amazon-kinesis-client-python\n\n\nOverview\n--------\n\nAll of the functionality is wrapped in two classes: ``KinesisConsumer`` and ``KinesisProducer``\n\nConsumer\n~~~~~~~~\n\nThe consumer works by launching a process per shard in the stream and then implementing the Python iterator protocol.\n\n.. code-block:: python\n\n from kinesis.consumer import KinesisConsumer\n\n consumer = KinesisConsumer(stream_name='my-stream')\n for message in consumer:\n print \"Received message: {0}\".format(message)\n\nMessages received from each of the shard processes are passed back to the main process through a Python Queue where they\nare yielded for processing. Messages are not strictly ordered, but this is a property of Kinesis and not this\nimplementation.\n\n\nLocking, Checkpointing & Multi-instance consumption\n^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nWhen deploying an application with multiple instances DynamoDB can be leveraged as a way to coordinate which instance\nis responsible for which shard, as it is not desirable to have each instance process all records.\n\nWith or without multiple nodes it is also desirable to checkpoint the stream as you process records so that you can\npickup from where you left off if you restart the consumer.\n\nA \"state\" backend that leverages DynamoDB allows consumers to coordinate which node is responsible which shards and\nwhere in the stream we are currently reading from.\n\n.. code-block:: python\n\n from kinesis.consumer import KinesisConsumer\n from kinesis.state import DynamoDB\n\n consumer = KinesisConsumer(stream_name='my-stream', state=DynamoDB(table_name='my-kinesis-state'))\n for message in consumer:\n print \"Received message: {0}\".format(message)\n\n\nThe DynamoDB table must already exist and must have a ``HASH`` key of ``shard``, with type ``S`` (string).\n\n\nProducer\n~~~~~~~~\n\nThe producer works by launching a single process for accumulation and publishing to the stream.\n\n.. code-block:: python\n\n from kinesis.producer import KinesisProducer\n\n producer = KinesisProducer(stream_name='my-stream')\n producer.put('Hello World from Python')\n\n\nBy default the accumulation buffer time is 500ms, or the max record size of 1Mb, whichever occurs first. You can\nchange the buffer time when you instantiate the producer via the ``buffer_time`` kwarg, specified in seconds. For\nexample, if your primary concern is budget and not performance you could accumulate over a 60 second duration.\n\n.. code-block:: python\n\n producer = KinesisProducer(stream_name='my-stream', buffer_time=60)\n\n\nThe background process takes precaution to ensure that any accumulated messages are flushed to the stream at\nshutdown time through signal handlers and the python atexit module, but it is not fully durable and if you were to\nsend a ``kill -9`` to the producer process any accumulated messages would be lost.\n\n\n\nAWS Permissions\n---------------\n\nBy default the producer, consumer & state classes all use the default `boto3 credentials chain`_. If you wish to alter\nthis you can instantiate your own ``boto3.Session`` object and pass it into the constructor via the ``boto3_session``\nkeyword argument of ``KinesisProducer``, ``KinesisConsumer`` or ``DynamoDB``.\n\n.. _boto3 credentials chain: http://boto3.readthedocs.io/en/latest/guide/configuration.html#configuring-credentials\n\n\n", "description_content_type": "", "docs_url": null, "download_url": "", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "https://github.com/NerdWalletOSS/kinesis-python", "keywords": "", "license": "Apache License Version 2.0", "maintainer": "", "maintainer_email": "", "name": "kinesis-python", "package_url": "https://pypi.org/project/kinesis-python/", "platform": "", "project_url": "https://pypi.org/project/kinesis-python/", "project_urls": { "Homepage": "https://github.com/NerdWalletOSS/kinesis-python" }, "release_url": "https://pypi.org/project/kinesis-python/0.2.1/", "requires_dist": [ "boto3 (<2.0,>=1.4.4)", "offspring (<1.0,>=0.0.3)", "six (<2.0,>=1.11.0)" ], "requires_python": "", "summary": "Low level, multiprocessing based AWS Kinesis producer & consumer library", "version": "0.2.1" }, "last_serial": 5805145, "releases": { "0.0.4": [ { "comment_text": "", "digests": { "md5": "e1362a12e8259fd48a47cc03ca055d74", "sha256": "380778ffc8bfb1913151473d6496d70f6ef1dfc528a083935526f7e5c561bbed" }, "downloads": -1, "filename": "kinesis_python-0.0.4-py2-none-any.whl", "has_sig": false, "md5_digest": "e1362a12e8259fd48a47cc03ca055d74", "packagetype": "bdist_wheel", "python_version": "2.7", "requires_python": null, "size": 5535, "upload_time": "2017-04-17T08:55:30", "url": "https://files.pythonhosted.org/packages/2a/71/0f61e34e54ff0a46bdb873fd6abca9bd3bc4c31256df9f54cadeabde7fc1/kinesis_python-0.0.4-py2-none-any.whl" }, { "comment_text": "", "digests": { "md5": "d4fe562364ad4f835883422c1b9ba61d", "sha256": "9b2fc5a91ed973548e59e49675d22dd98f07a1f92eaaa583fd393c711818f9b8" }, "downloads": -1, "filename": "kinesis-python-0.0.4.tar.gz", "has_sig": false, "md5_digest": "d4fe562364ad4f835883422c1b9ba61d", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 4027, "upload_time": "2017-04-17T08:55:27", "url": "https://files.pythonhosted.org/packages/7e/f9/b6a046e454b44e16985cff7ff621876afc6c3ff6079754032ee8a53e04cc/kinesis-python-0.0.4.tar.gz" } ], "0.1.4": [ { "comment_text": "", "digests": { "md5": "786c34618c162fa5edfbc69e9fd3b354", "sha256": "ae7389da7924e7707df294eed3284f47ba63456dab64b4f5ad03f1e2246bea62" }, "downloads": -1, "filename": "kinesis_python-0.1.4-py2-none-any.whl", "has_sig": false, "md5_digest": "786c34618c162fa5edfbc69e9fd3b354", "packagetype": "bdist_wheel", "python_version": "2.7", "requires_python": null, "size": 8416, "upload_time": "2017-12-29T22:27:07", "url": "https://files.pythonhosted.org/packages/cd/7c/b030e8c16fa2c0e31fc3646ba5be810e27c627199ba7dafe5442cb61c55a/kinesis_python-0.1.4-py2-none-any.whl" }, { "comment_text": "", "digests": { "md5": "03d5e6235b7af45270bf547baed4cb13", "sha256": "0f2aac4e1cc7bc63c7267120588f7430e928bf586168dafef91c6b0cb195abb4" }, "downloads": -1, "filename": "kinesis-python-0.1.4.tar.gz", "has_sig": false, "md5_digest": "03d5e6235b7af45270bf547baed4cb13", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 6250, "upload_time": "2017-12-29T22:27:05", "url": "https://files.pythonhosted.org/packages/80/c8/f1951a2ca50f436c059cdc915191e9a4d9358174f9910f6a55a0b37b1107/kinesis-python-0.1.4.tar.gz" } ], "0.1.5": [ { "comment_text": "", "digests": { "md5": "35323cb36a8478f2708c38e870f74bf5", "sha256": "87edabb4568f2c0c9c156e8281ebebab173f645081e790b7c4cb854eba2a3830" }, "downloads": -1, "filename": "kinesis_python-0.1.5-py2-none-any.whl", "has_sig": false, "md5_digest": "35323cb36a8478f2708c38e870f74bf5", "packagetype": "bdist_wheel", "python_version": "py2", "requires_python": null, "size": 12071, "upload_time": "2017-12-31T21:29:40", "url": "https://files.pythonhosted.org/packages/75/c9/bf9a3c05801b2c12476a83dac39657978a6cc03a333abbcc6ea5a499b0c1/kinesis_python-0.1.5-py2-none-any.whl" }, { "comment_text": "", "digests": { "md5": "1d73932a0c5c848ca07c32d7bf4dbaf1", "sha256": "654a6be32911635f67a3299862e84a399eb7dc7249be82b288176eef22c57425" }, "downloads": -1, "filename": "kinesis_python-0.1.5-py3-none-any.whl", "has_sig": false, "md5_digest": "1d73932a0c5c848ca07c32d7bf4dbaf1", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": null, "size": 12070, "upload_time": "2017-12-31T21:30:52", "url": "https://files.pythonhosted.org/packages/a5/f2/1d0a17cf6351e0703cef03edbce0c6d18d835be7196e2670bd6ab775f3b9/kinesis_python-0.1.5-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "c14efadb64aab08b3fbf775fb554c97d", "sha256": "6de1bcfdb439033e3bf9b7d1e6398e576a62964cb12049cd585732c4efcb63f1" }, "downloads": -1, "filename": "kinesis-python-0.1.5.tar.gz", "has_sig": false, "md5_digest": "c14efadb64aab08b3fbf775fb554c97d", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 8751, "upload_time": "2017-12-31T21:29:42", "url": "https://files.pythonhosted.org/packages/cb/15/68817ac1f497ecc3e6711d848d325237881270e9cc4832df762fcde8fc00/kinesis-python-0.1.5.tar.gz" } ], "0.1.6": [ { "comment_text": "", "digests": { "md5": "ec61753fb01099f5597a90dcd5db4347", "sha256": "e3c1581bcd434261dae0390138102c10fb6b33e7372c8a1c0ec603f6dfe0ae67" }, "downloads": -1, "filename": "kinesis_python-0.1.6-py2-none-any.whl", "has_sig": false, "md5_digest": "ec61753fb01099f5597a90dcd5db4347", "packagetype": "bdist_wheel", "python_version": "py2", "requires_python": null, "size": 13179, "upload_time": "2018-01-02T01:19:26", "url": "https://files.pythonhosted.org/packages/69/2c/a6cd5e012151fd5cae8f932d64d5ce2a4b2f3ea7bea6c3b1d13cf1c53f51/kinesis_python-0.1.6-py2-none-any.whl" }, { "comment_text": "", "digests": { "md5": "d83cc70ada4750b7ec91e0735937e8ac", "sha256": "c741b642eff75f41e1ee11edd47664b7f6f4b9e52532a751f3de4a49f7b2d90a" }, "downloads": -1, "filename": "kinesis_python-0.1.6-py3-none-any.whl", "has_sig": false, "md5_digest": "d83cc70ada4750b7ec91e0735937e8ac", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": null, "size": 13179, "upload_time": "2018-01-02T01:17:30", "url": "https://files.pythonhosted.org/packages/8f/e6/34aaf989bc1b59b581153a973ba1999fabf9104d9248424b0bc1121b4cd6/kinesis_python-0.1.6-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "99dbb693a53914a59dbf790367407e91", "sha256": "b78f3c01599ab3730cc2c86f635dac95296aa708dce7707905b22f6ee7d6ad0b" }, "downloads": -1, "filename": "kinesis-python-0.1.6.tar.gz", "has_sig": false, "md5_digest": "99dbb693a53914a59dbf790367407e91", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 9384, "upload_time": "2018-01-02T01:17:31", "url": "https://files.pythonhosted.org/packages/59/9f/2afe56e91cc8c82c07c027fec3d733498db4a9fd03baa9840d8973baacd1/kinesis-python-0.1.6.tar.gz" } ], "0.1.7": [ { "comment_text": "", "digests": { "md5": "c73879c8cc4972244d36259d23c71753", "sha256": "5fc6f00a0b797c12d6eafd160581682de5abe9659d48ca41bbe70635f2674d79" }, "downloads": -1, "filename": "kinesis_python-0.1.7-py2-none-any.whl", "has_sig": false, "md5_digest": "c73879c8cc4972244d36259d23c71753", "packagetype": "bdist_wheel", "python_version": "py2", "requires_python": null, "size": 13180, "upload_time": "2018-01-06T06:58:24", "url": "https://files.pythonhosted.org/packages/b5/88/08fdb3d7aec907e9368ee43bd38368b16551a7430142c8f476ad3aa7bbc8/kinesis_python-0.1.7-py2-none-any.whl" }, { "comment_text": "", "digests": { "md5": "32c914aa9436e8e2cf66a569e127f655", "sha256": "8d6af4c79849ff23ab4747a2aa218bca2931f90769bec534ac0e21157d0e0ccb" }, "downloads": -1, "filename": "kinesis_python-0.1.7-py3-none-any.whl", "has_sig": false, "md5_digest": "32c914aa9436e8e2cf66a569e127f655", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": null, "size": 13181, "upload_time": "2018-01-06T06:58:05", "url": "https://files.pythonhosted.org/packages/d3/8f/006e86b11e9da4da60300f8ed1042124656a85e7ddc4849846054b7e1586/kinesis_python-0.1.7-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "a2910d7ba83f6838b5a2e858fdd47270", "sha256": "222bd852f428b9dd6c87dfb89abaab23394dc950d0dbcf9f51be6fdff5bbf5b0" }, "downloads": -1, "filename": "kinesis-python-0.1.7.tar.gz", "has_sig": false, "md5_digest": "a2910d7ba83f6838b5a2e858fdd47270", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 9497, "upload_time": "2018-01-06T06:58:08", "url": "https://files.pythonhosted.org/packages/21/ce/4b4e213f2dfaa8307d2e60155630a8c4684869c4e1f94ccabc41f375a17b/kinesis-python-0.1.7.tar.gz" } ], "0.1.8": [ { "comment_text": "", "digests": { "md5": "2fd60123b834d81b130cde40e64fabc8", "sha256": "30b5da18f593dd2c7b0fce9f94dfb737b35dee9a2b67c4448d8fffad7cb9fd76" }, "downloads": -1, "filename": "kinesis_python-0.1.8-py2-none-any.whl", "has_sig": false, "md5_digest": "2fd60123b834d81b130cde40e64fabc8", "packagetype": "bdist_wheel", "python_version": "py2", "requires_python": null, "size": 10731, "upload_time": "2018-11-26T22:38:36", "url": "https://files.pythonhosted.org/packages/cb/a7/f3a1a8ceab131c9de9f5224b50f4f4bc1441c2f24fbd196e9626d08af027/kinesis_python-0.1.8-py2-none-any.whl" }, { "comment_text": "", "digests": { "md5": "3960314ad91aa01edfef2dd4dc838cee", "sha256": "c1412ae8562b328c04bfd09cda0f3f131914267348b814369e4a3fc081797ac3" }, "downloads": -1, "filename": "kinesis_python-0.1.8-py3-none-any.whl", "has_sig": false, "md5_digest": "3960314ad91aa01edfef2dd4dc838cee", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": null, "size": 10731, "upload_time": "2018-11-26T22:38:45", "url": "https://files.pythonhosted.org/packages/d6/31/168d9b0c23686f3f207667dc38239ceddce6c2ebd68bf3e791db2d3cf4ce/kinesis_python-0.1.8-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "a3bb70439b7fee363dbc02a53bd31fa7", "sha256": "00650e396edfbb604a506b36acdb033405f0869bc23014d1802d657e66ea42bc" }, "downloads": -1, "filename": "kinesis-python-0.1.8.tar.gz", "has_sig": false, "md5_digest": "a3bb70439b7fee363dbc02a53bd31fa7", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 9570, "upload_time": "2018-11-26T22:38:45", "url": "https://files.pythonhosted.org/packages/5b/da/385c893a883cc7ccad1b56e9f585b43d19f0473ecd9c59585d2256801039/kinesis-python-0.1.8.tar.gz" } ], "0.2.0": [ { "comment_text": "", "digests": { "md5": "8406d1ea7eb59f3eb56a05155394185e", "sha256": "9b9ac4c612724b157ce06bcf6327d8fc5cabdbe184f8d359ef57d94d02787479" }, "downloads": -1, "filename": "kinesis_python-0.2.0-py3-none-any.whl", "has_sig": false, "md5_digest": "8406d1ea7eb59f3eb56a05155394185e", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": null, "size": 10749, "upload_time": "2019-09-06T20:29:17", "url": "https://files.pythonhosted.org/packages/78/fa/50456909d78d434afd516518322c7ddc624ed940a085c46035746b7d0c32/kinesis_python-0.2.0-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "7f645958ffd4b212dca0f3eb6218e843", "sha256": "b7cb190daa9d65795ec817ee8b66e0a70fff5f2d305094602385e0785586a990" }, "downloads": -1, "filename": "kinesis-python-0.2.0.tar.gz", "has_sig": false, "md5_digest": "7f645958ffd4b212dca0f3eb6218e843", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 11346, "upload_time": "2019-09-06T20:29:19", "url": "https://files.pythonhosted.org/packages/22/86/375dde8c4a594dc6304ba3cab247990936e0f912e3b4467ad632e2ef778c/kinesis-python-0.2.0.tar.gz" } ], "0.2.1": [ { "comment_text": "", "digests": { "md5": "0ef46ad131bb754c5bcef11173ac8f09", "sha256": "73aeeffc1f48bdba7a3390939f3b96465145772edbcf96daf2f8c41071f6c48d" }, "downloads": -1, "filename": "kinesis_python-0.2.1-py3-none-any.whl", "has_sig": false, "md5_digest": "0ef46ad131bb754c5bcef11173ac8f09", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": null, "size": 10749, "upload_time": "2019-09-09T19:29:58", "url": "https://files.pythonhosted.org/packages/9d/56/0a82a502f8bd438ee1d24a5561dac323e10e0c503918e3245e241cd720c9/kinesis_python-0.2.1-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "1394f9c63f360569da6e4d67e14e5368", "sha256": "4f3e98f337b0d5e5c77b26a9138e627da32104ade3785b3a136c70389c2b79d3" }, "downloads": -1, "filename": "kinesis-python-0.2.1.tar.gz", "has_sig": false, "md5_digest": "1394f9c63f360569da6e4d67e14e5368", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 11677, "upload_time": "2019-09-09T19:29:59", "url": "https://files.pythonhosted.org/packages/be/bb/7e54b23b64fd49ad9afedd5f3af992980a744023ab01b963e99ea2572a61/kinesis-python-0.2.1.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "0ef46ad131bb754c5bcef11173ac8f09", "sha256": "73aeeffc1f48bdba7a3390939f3b96465145772edbcf96daf2f8c41071f6c48d" }, "downloads": -1, "filename": "kinesis_python-0.2.1-py3-none-any.whl", "has_sig": false, "md5_digest": "0ef46ad131bb754c5bcef11173ac8f09", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": null, "size": 10749, "upload_time": "2019-09-09T19:29:58", "url": "https://files.pythonhosted.org/packages/9d/56/0a82a502f8bd438ee1d24a5561dac323e10e0c503918e3245e241cd720c9/kinesis_python-0.2.1-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "1394f9c63f360569da6e4d67e14e5368", "sha256": "4f3e98f337b0d5e5c77b26a9138e627da32104ade3785b3a136c70389c2b79d3" }, "downloads": -1, "filename": "kinesis-python-0.2.1.tar.gz", "has_sig": false, "md5_digest": "1394f9c63f360569da6e4d67e14e5368", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 11677, "upload_time": "2019-09-09T19:29:59", "url": "https://files.pythonhosted.org/packages/be/bb/7e54b23b64fd49ad9afedd5f3af992980a744023ab01b963e99ea2572a61/kinesis-python-0.2.1.tar.gz" } ] }