{
    "info": {
        "author": "Hengfeng Li",
        "author_email": "hengf.li@gmail.com",
        "bugtrack_url": null,
        "classifiers": [
            "Development Status :: 3 - Alpha",
            "Environment :: No Input/Output (Daemon)",
            "Intended Audience :: Developers",
            "License :: OSI Approved :: MIT License",
            "Operating System :: MacOS :: MacOS X",
            "Operating System :: Microsoft :: Windows",
            "Operating System :: POSIX",
            "Operating System :: Unix",
            "Programming Language :: Python :: 3.6",
            "Programming Language :: Python :: 3.7",
            "Programming Language :: Python :: 3 :: Only",
            "Programming Language :: Python :: Implementation :: CPython"
        ],
        "description": "# Kinesis Consumer in Python\n\n[![alt text][build_status]][build_status_url]\n[![alt text][mit_license]][mit_license_url]\n[![alt text][wheel]][wheel_url] \n[![alt text][pyversion]][pyversion_url] \n[![alt text][pyimp]][pyimp_url]\n\nA kinesis consumer is purely written in python. This is a lightweight wrapper \non top of AWS python library [boto3](https://github.com/boto/boto3). You also can \nconsume records from Kinesis Data Stream (KDS) via: \n\n* Lambda function: I have a demo [kinesis-lambda-sqs-demo](https://github.com/HengfengLi/kinesis-lambda-sqs-demo)\nshowing how to consume records in a serverless and real-time way. \n* [Kinesis Firehose](https://aws.amazon.com/kinesis/firehose/): This is a AWS managed service and easily save records\ninto different sinks, like S3, ElasticSearch, Redshift. \n\n## Installation\n\nInstall the package via `pip`: \n```bash\npip install kcpy\n```\n\n## Getting started\n\n```python\nfrom kcpy import StreamConsumer\nconsumer = StreamConsumer('my_stream_name')\nfor record in consumer:\n    print(record)\n```\n\nThe output would look like:\n\n```bash\n{\n    'ApproximateArrivalTimestamp': datetime.datetime(2018, 11, 13, 11, 57, 55, 117807), \n    'Data': b'Jessica Walter', \n    'PartitionKey': 'Jessica Walter', \n    'SequenceNumber': '1'\n}\n```\n\nOr, you can consume stream data with checkpointing: \n\n```python\nfrom kcpy import StreamConsumer\nconsumer = StreamConsumer('my_stream_name', consumer_name='my_consumer', checkpoint=True)\nfor record in consumer:\n    print(record)\n```\n\n## Checkpointing\n\nBelow shows the schema of checkpointing: \n\n```\n                                                                   producer\n[stream_1]                                                            |\n+---------------+---+---+---+---+---+---+---+---+                     |\n| shard_1       | 1 | 2 | 3 | 4 | 5 | 6 | 7 |...| <-------------------+\n+---------------+---+---+---+---+---+---+---+---+                     |\n| shard_2       | 1 | 2 | 3 | 4 | 5 |...| <---------------------------+\n+---------------+---+---+---+---+---+---+---+---+---+                 |\n| shard_3       | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 |...| <---------------+\n+---------------+---+---+---+---+---+---+---+---+---+\n                          ^                   ^\n                          |                   |\n                      consumer_1          consumer_2\n                          |                   |\n                          |                   +---------+\n                          |                             |\n                          +------------------+          |\n                                             |          |\n                                             v          |\n+---------------+-------------+----------+--------+     |\n| consumer_name | stream_name | shard_id | seq_no |     |\n+---------------+-------------+----------+--------+     |\n| consumer_1    | stream_1    | shard_1  |   5    |     |\n| consumer_1    | stream_1    | shard_2  |   15   |     |\n| consumer_1    | stream_1    | ...      |   15   |     |\n| consumer_1    | stream_1    | shard_N  |   XX   |     |\n| consumer_2    | stream_1    | shard_1  |   6    | <---+\n+---------------+-------------+----------+--------+\n```\n\n## Features\n\n* Read records from a stream with multiple shards\n* Save checkpoint for each shard consumer for a stream\n\n## Todo\n\n* ~~Add type checking with mypy~~\n* ~~Add tox for automating multiple testing environments~~ \n* ~~Add the config for travis CI~~\n* Support other storage solutions (mysql, dynamodb, redis, etc.) for checkpointing  \n* Rebalance when the number of shards changes\n* Allow kcpy to run on multiple machines\n\n## Changelog\n\n### 0.1.7\n\n* Add travis CI config and remove python3.5.  \n\n### 0.1.6\n\n* Fix some issues in setup.py. \n\n### 0.1.5\n\n* Add consumer checkpointing with a simple sqlite storage solution. \n\n### 0.1.4\n\n* Pass aws configurations into boto3 client directly. \n\n### 0.1.3\n\n* Update the README. \n\n### 0.1.2\n\n* Add markdown support for long description. \n\n### 0.1.1\n\n* Add a long description.\n\n### 0.1.0\n\n* First version of kcpy.  \n\n## License\n\nCopyright (c) 2018 Hengfeng Li. It is free software, and may\nbe redistributed under the terms specified in the [LICENSE] file.\n\n[LICENSE]: /LICENSE\n\n[build_status]: https://secure.travis-ci.org/HengfengLi/kcpy.png?branch=master \"Build status\"\n[build_status_url]: https://travis-ci.org/HengfengLi/kcpy\n\n[mit_license]: https://img.shields.io/pypi/l/kcpy.svg \"MIT License\"\n[mit_license_url]: https://opensource.org/licenses/MIT\n\n[wheel]: https://img.shields.io/pypi/wheel/kcpy.svg \"kcpy can be installed via wheel\" \n[wheel_url]: http://pypi.org/project/kcpy/\n\n[pyversion]: https://img.shields.io/pypi/pyversions/kcpy.svg \"Supported Python versions.\"\n[pyversion_url]: http://pypi.org/project/kcpy/\n\n[pyimp]: https://img.shields.io/pypi/implementation/kcpy.svg \"Support Python implementations.\"\n[pyimp_url]: http://pypi.org/project/kcpy/\n\n\n",
        "description_content_type": "text/markdown",
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "",
        "keywords": "kinesis,consumer,stream,processing,queue",
        "license": "MIT License",
        "maintainer": "",
        "maintainer_email": "",
        "name": "kcpy",
        "package_url": "https://pypi.org/project/kcpy/",
        "platform": "",
        "project_url": "https://pypi.org/project/kcpy/",
        "project_urls": null,
        "release_url": "https://pypi.org/project/kcpy/0.1.7/",
        "requires_dist": [
            "boto3"
        ],
        "requires_python": ">=3.6.0",
        "summary": "A kinesis consumer is purely written in python.",
        "version": "0.1.7"
    },
    "last_serial": 4567564,
    "releases": {
        "0.1.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "43ed5e669957edc225210f36cb9705eb",
                    "sha256": "8610dff80b080c9174742babfa01f1fe96be1ab83e40a7ca6fe684e3d1445906"
                },
                "downloads": -1,
                "filename": "kcpy-0.1.0-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "43ed5e669957edc225210f36cb9705eb",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": ">=3.5.0",
                "size": 3063,
                "upload_time": "2018-09-20T12:17:49",
                "url": "https://files.pythonhosted.org/packages/36/19/ad9a9a3df5c6766e1f85c0542f51973e42c14db75260be8487c4673c6675/kcpy-0.1.0-py2.py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "1cdfb8776f8c00e0a567ef065e93a73e",
                    "sha256": "42baae343a214e3866c0c1483fab6a0849e51de07fad3a8d5234e0003aeec4e5"
                },
                "downloads": -1,
                "filename": "kcpy-0.1.0.tar.gz",
                "has_sig": false,
                "md5_digest": "1cdfb8776f8c00e0a567ef065e93a73e",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=3.5.0",
                "size": 2632,
                "upload_time": "2018-09-20T12:17:51",
                "url": "https://files.pythonhosted.org/packages/1d/bd/67b9853d073ae79a98716a61b4c1aefab70cbf0945639494ba9414aa79a2/kcpy-0.1.0.tar.gz"
            }
        ],
        "0.1.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "0521eff466dda516072cddb959534023",
                    "sha256": "df52fec97e200648f841ba4ea68720651891a256b719ad318c7ea7c9a81da77f"
                },
                "downloads": -1,
                "filename": "kcpy-0.1.1-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "0521eff466dda516072cddb959534023",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": ">=3.5.0",
                "size": 3251,
                "upload_time": "2018-09-20T12:33:51",
                "url": "https://files.pythonhosted.org/packages/f9/42/1fc708b17ee733191ab06b76c843a774fbbcf8aecbebe1a45a183a0dcc6c/kcpy-0.1.1-py2.py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "8764b0b72e957c7d4d9afe8b9eb7c035",
                    "sha256": "59fcfe19458bf1b7de84fe68bf9e124a0daa58efab7ad5a16c58e104e6a9dded"
                },
                "downloads": -1,
                "filename": "kcpy-0.1.1.tar.gz",
                "has_sig": false,
                "md5_digest": "8764b0b72e957c7d4d9afe8b9eb7c035",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=3.5.0",
                "size": 2827,
                "upload_time": "2018-09-20T12:33:52",
                "url": "https://files.pythonhosted.org/packages/99/8c/fbe03afb1b4428017a235b43b4b1ef156ea2ec6d8ed940dd4015c4114932/kcpy-0.1.1.tar.gz"
            }
        ],
        "0.1.2": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "f90e3cf451a71d8e475983db06fe57fb",
                    "sha256": "721cf37ad59652ea0935df3ffaf8635330c9ce70a7284becc3413651ae510154"
                },
                "downloads": -1,
                "filename": "kcpy-0.1.2-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "f90e3cf451a71d8e475983db06fe57fb",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": ">=3.5.0",
                "size": 3291,
                "upload_time": "2018-09-20T13:22:18",
                "url": "https://files.pythonhosted.org/packages/1f/a7/9f3003f264e610e27c2638058b32aa34dbe4232f5cf5f4f13b1cb7dbf42a/kcpy-0.1.2-py2.py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "ad84cfee40eecb2645e787e1f0a135d4",
                    "sha256": "521b726660fe486cfb57ef097a48d354ec6be2d87878a5647deabe2796a9553c"
                },
                "downloads": -1,
                "filename": "kcpy-0.1.2.tar.gz",
                "has_sig": false,
                "md5_digest": "ad84cfee40eecb2645e787e1f0a135d4",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=3.5.0",
                "size": 2890,
                "upload_time": "2018-09-20T13:22:20",
                "url": "https://files.pythonhosted.org/packages/aa/6f/8a6acc2afe58c31e2f764462cf71bd8d697f49a5eaed15f73300f5a6a4d3/kcpy-0.1.2.tar.gz"
            }
        ],
        "0.1.3": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "7be07c06e114a9c4622847a30c237207",
                    "sha256": "7922d7dc9eb8fad8147059797363a61b137db7e94ff0892e664e2aab334ce92f"
                },
                "downloads": -1,
                "filename": "kcpy-0.1.3-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "7be07c06e114a9c4622847a30c237207",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": ">=3.5.0",
                "size": 3870,
                "upload_time": "2018-11-13T12:15:55",
                "url": "https://files.pythonhosted.org/packages/33/5e/ab5948f21a3df75eb0cf3dc48c50ea1f0c101beedb7419617a7d4593390d/kcpy-0.1.3-py2.py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "f8d9a3cf023d35f81b12b0e91a4ec965",
                    "sha256": "fc02a7a8a3be4e22dc27778d326e323d32f55c570af465d3f0c0c93b6dcb8068"
                },
                "downloads": -1,
                "filename": "kcpy-0.1.3.tar.gz",
                "has_sig": false,
                "md5_digest": "f8d9a3cf023d35f81b12b0e91a4ec965",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=3.5.0",
                "size": 3563,
                "upload_time": "2018-11-13T12:15:57",
                "url": "https://files.pythonhosted.org/packages/ce/09/cc26648fb23c37aada3dfef575a953ca7c1c294891d4dc66cb7eaa23ccc7/kcpy-0.1.3.tar.gz"
            }
        ],
        "0.1.4": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "a1c370db30296e2cedbae8336196f6ab",
                    "sha256": "3995061b839f2d075db81c1b2d46aa27ce1e60b69a416fb231fddb3b46777644"
                },
                "downloads": -1,
                "filename": "kcpy-0.1.4-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "a1c370db30296e2cedbae8336196f6ab",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": ">=3.5.0",
                "size": 3934,
                "upload_time": "2018-11-17T22:00:32",
                "url": "https://files.pythonhosted.org/packages/e0/a6/2aa2b2bb7068aa82e6e71f0449122f7418c21c0629c97e363e76d837bdec/kcpy-0.1.4-py2.py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "cb43f3d8abab7bac879532c0c50ed3aa",
                    "sha256": "9a2f6e489222b883b3b2678b6c9b09ff49092d06dbeff3cb8a7a681336b0ffca"
                },
                "downloads": -1,
                "filename": "kcpy-0.1.4.tar.gz",
                "has_sig": false,
                "md5_digest": "cb43f3d8abab7bac879532c0c50ed3aa",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=3.5.0",
                "size": 3636,
                "upload_time": "2018-11-17T22:00:34",
                "url": "https://files.pythonhosted.org/packages/80/7b/91ecb90b3c8a4913227ebd1defcf1ed3ea9b8b5df3f3f818c01d01a472d0/kcpy-0.1.4.tar.gz"
            }
        ],
        "0.1.5": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "6ead5148711b8c14c1400eb94fbf8105",
                    "sha256": "ac737084399a0eb4c99d3a07c040fcff88682fd428db2009a3c1bdb12f5ce473"
                },
                "downloads": -1,
                "filename": "kcpy-0.1.5-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "6ead5148711b8c14c1400eb94fbf8105",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": ">=3.5.0",
                "size": 5379,
                "upload_time": "2018-12-01T03:48:41",
                "url": "https://files.pythonhosted.org/packages/23/b0/b8a6877c2a088f8f94ee04ce52edd41c25bf5cba641ea8cb930af2932747/kcpy-0.1.5-py2.py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "605e7c81681aa80a7e0b366e14001854",
                    "sha256": "d0a0728a36c002298e2a570a21007e3c6f1409a44aeac596dcfdb5a3b11ff4fe"
                },
                "downloads": -1,
                "filename": "kcpy-0.1.5.tar.gz",
                "has_sig": false,
                "md5_digest": "605e7c81681aa80a7e0b366e14001854",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=3.5.0",
                "size": 4851,
                "upload_time": "2018-12-01T03:48:43",
                "url": "https://files.pythonhosted.org/packages/0b/ab/5f8c7d2162ff244a6ddb3da61777136715573dc938cc34ea3482bb75f1a4/kcpy-0.1.5.tar.gz"
            }
        ],
        "0.1.6": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "d240e1e7965df69937fb1285c4a84cee",
                    "sha256": "421d659e786cbca73168eafb426592fc6602114c400e2cd7566cf8fc017c2992"
                },
                "downloads": -1,
                "filename": "kcpy-0.1.6-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "d240e1e7965df69937fb1285c4a84cee",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": ">=3.5.0",
                "size": 7030,
                "upload_time": "2018-12-05T10:41:45",
                "url": "https://files.pythonhosted.org/packages/4d/4d/a9bc848e9001192f9fece1c422a96df3e5bd55cc30a561a300554e37fe57/kcpy-0.1.6-py2.py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "0b7a0b471e69c3eb5aa8ddf1c30d2c9d",
                    "sha256": "36142d5fc927d1d09f653b64b5f0f6b1b74ac26b27265121094eff332dccb5c3"
                },
                "downloads": -1,
                "filename": "kcpy-0.1.6.tar.gz",
                "has_sig": false,
                "md5_digest": "0b7a0b471e69c3eb5aa8ddf1c30d2c9d",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=3.5.0",
                "size": 6011,
                "upload_time": "2018-12-05T10:41:46",
                "url": "https://files.pythonhosted.org/packages/65/47/657c492ef1eed9762aa6537a897f0488136dc6332742bc3984c729a5df2e/kcpy-0.1.6.tar.gz"
            }
        ],
        "0.1.7": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "a7d097b1270a6732514c14ba2834dab2",
                    "sha256": "b72e8bcb85c6fb999c3287a01b2509096db416a51b30bfeee07d385f3054e8ec"
                },
                "downloads": -1,
                "filename": "kcpy-0.1.7-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "a7d097b1270a6732514c14ba2834dab2",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": ">=3.6.0",
                "size": 7338,
                "upload_time": "2018-12-06T11:35:18",
                "url": "https://files.pythonhosted.org/packages/97/08/db9b07d1c94880ee36c02cec65f1f4a5fd987ea6faa1da717150834319dc/kcpy-0.1.7-py2.py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "81e23521fafdbc7ffda212d4be3b9292",
                    "sha256": "5f1f46d119bcd0cfc8945a10e6148f905c4b4c357df9590e84b7d643efb19cad"
                },
                "downloads": -1,
                "filename": "kcpy-0.1.7.tar.gz",
                "has_sig": false,
                "md5_digest": "81e23521fafdbc7ffda212d4be3b9292",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=3.6.0",
                "size": 6241,
                "upload_time": "2018-12-06T11:35:19",
                "url": "https://files.pythonhosted.org/packages/c7/8e/d819d04492b2e9c21d3033551d7395bfb423a1e7a2ef32d5471a602d0749/kcpy-0.1.7.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "a7d097b1270a6732514c14ba2834dab2",
                "sha256": "b72e8bcb85c6fb999c3287a01b2509096db416a51b30bfeee07d385f3054e8ec"
            },
            "downloads": -1,
            "filename": "kcpy-0.1.7-py2.py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "a7d097b1270a6732514c14ba2834dab2",
            "packagetype": "bdist_wheel",
            "python_version": "py2.py3",
            "requires_python": ">=3.6.0",
            "size": 7338,
            "upload_time": "2018-12-06T11:35:18",
            "url": "https://files.pythonhosted.org/packages/97/08/db9b07d1c94880ee36c02cec65f1f4a5fd987ea6faa1da717150834319dc/kcpy-0.1.7-py2.py3-none-any.whl"
        },
        {
            "comment_text": "",
            "digests": {
                "md5": "81e23521fafdbc7ffda212d4be3b9292",
                "sha256": "5f1f46d119bcd0cfc8945a10e6148f905c4b4c357df9590e84b7d643efb19cad"
            },
            "downloads": -1,
            "filename": "kcpy-0.1.7.tar.gz",
            "has_sig": false,
            "md5_digest": "81e23521fafdbc7ffda212d4be3b9292",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": ">=3.6.0",
            "size": 6241,
            "upload_time": "2018-12-06T11:35:19",
            "url": "https://files.pythonhosted.org/packages/c7/8e/d819d04492b2e9c21d3033551d7395bfb423a1e7a2ef32d5471a602d0749/kcpy-0.1.7.tar.gz"
        }
    ]
}