{
    "info": {
        "author": "Christopher H. Todd",
        "author_email": "Christopher.Hayden.Todd@gmail.com",
        "bugtrack_url": null,
        "classifiers": [
            "Development Status :: 5 - Production/Stable",
            "Intended Audience :: Developers",
            "Natural Language :: English",
            "Programming Language :: Python",
            "Programming Language :: Python :: 3",
            "Programming Language :: Python :: 3.6",
            "Programming Language :: Python :: 3.7",
            "Programming Language :: Python :: 3.8"
        ],
        "description": "# Christopher H. Todd's Python Library For Interacting With Kafka\n\nThe ctodd-python-lib-kafka project is responsible for interacting with Apache Kafka. This includes producing and consuming records from topics, utilizing .avro format, and other tasks in creating event driven applications with Python.\n\n## Table of Contents\n\n- [Dependencies](#dependencies)\n- [Libraries](#libraries)\n- [Example Scripts](#example-scripts)\n- [Notes](#notes)\n- [TODO](#todo)\n\n## Dependencies\n\n### Python Packages\n\n- confluent-kafka==0.11.6\n- simplejson==3.16.0\n\n## Libraries\n\n### [kafka_admin_helpers.py](https://github.com/ChristopherHaydenTodd/ctodd-python-lib-kafka/blob/master/kafka_helpers/kafka_admin_helpers.py)\n\nThis library is used to interacting with Kafka Admin functionality. This\nincludes getting the admin object that will return details about kafka\nstate.\n\nFunctions:\n\n```\ndef get_kafka_admin_client(kafka_brokers):\n    \"\"\"\n    Purpose:\n        Get a Kafka Admin Client Object. Allows for polling information about Kafka\n        configuration and creating objects in Kafka\n    Args:\n        kafka_brokers (List of Strings): List of host:port combinations for kakfa\n            brokers\n    Return:\n        kafka_admin_client (Kafka Admin Client Obj): Kafka Admin Client Obj for the\n            brokers\n    \"\"\"\n```\n\n### [kafka_consumer_helpers.py](https://github.com/ChristopherHaydenTodd/ctodd-python-lib-kafka/blob/master/kafka_helpers/kafka_consumer_helpers.py)\n\nThis library is used to aid in creating kafka consumers.\n\nFunctions:\n\n```\ndef get_kafka_consumer(\n    kafka_brokers,\n    consumer_group=\"default\",\n    timeout=6000,\n    offset_start=\"latest\",\n    get_stats=True\n):\n    \"\"\"\n    Purpose:\n        Get a Kafka Consumer Object (not yet connected to a topic)\n    Args:\n        kafka_brokers (List of Strings): List of host:port combinations for kakfa brokers\n        consumer_group (String): Consumer group to consume as. default is \"default\"\n        timeout (String): Timeout in ms if no messages are found (during poll). Default\n            is 6000\n        offset_start (String): Where to start consuming with respect to the consumer\n            group/topic offset. Default is \"latest\", which ignores any messages in the\n            topic before the consumer begins consuming\n        get_stats (Bool): Whether or not to print statistics. Default is True\n    Return:\n        kafka_consumer (Kafka Consumer Obj): Kafka Consumer Object\n    \"\"\"\n```\n\n```\ndef consume_topic(kafka_consumer, kafka_topics):\n    \"\"\"\n    Purpose:\n        Consume Kafka Topics\n    Args:\n        kafka_consumer (Kafka Consumer Obj): Kafka Consumer Object\n        kafka_topics (List of Strings): List of Kafka Topics to Consume.\n    Yields:\n        msg (Kafka Message Obj): Message Obj returned from the topic\n    \"\"\"\n```\n\n### [kafka_exceptions.py](https://github.com/ChristopherHaydenTodd/ctodd-python-lib-kafka/blob/master/kafka_helpers/kafka_exceptions.py)\n\nFile for holding custom exception types that will be generated by the kafka_helpers libraries\n\nClasses:\n\n```\nclass TopicNotFound(Exception):\n    \"\"\"\n    Purpose:\n        The TopicNotFound will be raised when attempting to consume a topic that\n        does not exist\n    \"\"\"\n```\n\n\n### [kafka_general_helpers.py](https://github.com/ChristopherHaydenTodd/ctodd-python-lib-kafka/blob/master/kafka_helpers/kafka_general_helpers.py)\n\nThis library is used to interact with kafka not specificlly related to consuming or producing messages\n\nFunctions:\n\n#### N/A\n\n\n### [kafka_producer_helpers.py](https://github.com/ChristopherHaydenTodd/ctodd-python-lib-kafka/blob/master/kafka_helpers/kafka_producer_helpers.py)\n\nThis library is used to aid in creating kafka producers.\n\nFunctions:\n\n```\ndef get_kafka_producer(kafka_brokers, get_stats=True):\n    \"\"\"\n    Purpose:\n        Get a Kafka Producer Object (not yet connected to a topic)\n    Args:\n        kafka_brokers (List of Strings): List of host:port combinations for kakfa brokers\n        get_stats (Bool): Whether or not to print statistics. Default is True\n    Return:\n        kafka_producer (Kafka Producer Obj): Kafka Producer Object\n    \"\"\"\n```\n\n```\ndef produce_message(kafka_producer, kafka_topic, msg):\n    \"\"\"\n    Purpose:\n        Consume Kafka Topics\n    Args:\n        kafka_producer (Kafka Producer Obj): Kafka Producer Object\n        kafka_topic (String): Kafka Topic to Produce message to.\n        msg (String): Message to produce to Kafka\n    Returns:\n        N/A\n    \"\"\"\n```\n\n```\ndef produce_results_callback(err, msg):\n    \"\"\"\n    Purpose:\n        Optional per-message delivery callback (triggered by poll() or\n        flush()) when a message has been successfully delivered or\n        permanently failed delivery (after retries).\n    Args:\n        err (String): Error Message\n        msg (Object): Kafka Callback Message Object\n    Return:\n        N/A\n    \"\"\"\n```\n\n\n### [kafka_topic_helpers.py](https://github.com/ChristopherHaydenTodd/ctodd-python-lib-kafka/blob/master/kafka_helpers/kafka_topic_helpers.py)\n\nThis library is used to interact with kafka topics. This includes getting\na list of the topics, finding details about a topic, creating topics, and\nmore.\n\nFunctions:\n\n```\ndef get_topics(kafka_admin_client, return_system_topics=False):\n    \"\"\"\n    Purpose:\n        Get a List of Kafka Topics.\n    Args:\n        kafka_admin_client (Kafka Admin Client Obj): Kafka Admin Client Obj for the\n            brokers\n    Return:\n        kafka_topics (Dict of Kafka Topics): Key is the topic name and value is a\n            Kafka metadata object that has basic topic information\n    \"\"\"\n```\n\n```\ndef create_kafka_topic(\n    kafka_admin_client, topic_name, topic_replication=1, topic_partitions=1\n):\n    \"\"\"\n    Purpose:\n        Create a Kafka Topic\n    Args:\n        kafka_admin_client (Kafka Admin Client Obj): Kafka Admin Client Obj for the\n            brokers\n        topic_name (String): Name of the topic to create\n        topic_replication (Int): Replication factor for the new topic\n        topic_partitions (Int): Number of partitions to devide the topic into\n    Return:\n        N/A\n    \"\"\"\n```\n\n## Example Scripts\n\nExample executable Python scripts/modules for testing and interacting with the library. These show example use-cases for the libraries and can be used as templates for developing with the libraries or to use as one-off development efforts.\n\n### [consume_from_kafka_topic.py](https://github.com/ChristopherHaydenTodd/ctodd-python-lib-kafka/blob/master/example_usage/consume_from_kafka_topic.py)\n\n```\n    Purpose:\n        Consume from a Kafka Topic\n\n    Steps:\n        - Connect to Kafka\n        - Create Consumer Object\n        - Poll Topic\n        - Parse Message\n        - Print Message\n\n    example script call:\n        python3 consume_from_kafka_topic.py --topic=\"test-env-topic\" \\\n            --broker=\"0.0.0.0:9092\" --consumer-group=\"test-env-consumer\"\n```\n\n### [produce_to_kafka_topic.py](https://github.com/ChristopherHaydenTodd/ctodd-python-lib-kafka/blob/master/example_usage/kafka_producer_helpers.py)\n\n```\n    Purpose:\n        Produce to a Kafka Topic\n\n    Steps:\n        - Connect to Kafka\n        - Create Producer Object\n        - Prompt for Input\n        - Parse Input\n        - Produce Input to Kafka\n\n    example script call:\n        python3 produce_to_kafka_topic.py --topic=\"test-env-topic\" \\\n            --broker=\"localhost:9092\"\n```\n\n### [create_kakfa_topic.py](https://github.com/ChristopherHaydenTodd/ctodd-python-lib-kafka/blob/master/example_usage/create_kakfa_topic.py)\n\n```\n    Purpose:\n        Create a Kafka Topic. Takes in replication and parition information\n\n    Steps:\n        - Connect to Kafka\n        - Create Kafka Admin Client\n        - Create Topic In Kafka\n\n    function call:\n        ---\n    example script call:\n        python3 create_kafka_topic.py --topic-name=\"test-env-topic\" \\\n            --topic-replication=3 --topic-partitions=4 \\\n            --broker=\"localhost:9092\"\n```\n\n## Notes\n\n - Relies on f-string notation, which is limited to Python3.6.  A refactor to remove these could allow for development with Python3.0.x through 3.5.x\n\n## TODO\n\n - Unittest framework in place, but lacking tests\n\n\n",
        "description_content_type": "text/markdown",
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://github.com/ChristopherHaydenTodd/ctodd-python-lib-kafka",
        "keywords": "python,libraries,Kafka,consumer,producer",
        "license": "MIT",
        "maintainer": "",
        "maintainer_email": "",
        "name": "ctodd-python-lib-kafka",
        "package_url": "https://pypi.org/project/ctodd-python-lib-kafka/",
        "platform": "",
        "project_url": "https://pypi.org/project/ctodd-python-lib-kafka/",
        "project_urls": {
            "Homepage": "https://github.com/ChristopherHaydenTodd/ctodd-python-lib-kafka"
        },
        "release_url": "https://pypi.org/project/ctodd-python-lib-kafka/1.0.2/",
        "requires_dist": [
            "simplejson",
            "confluent-kafka"
        ],
        "requires_python": ">3.6",
        "summary": "Python utilities used for interacting with Apache Kafka",
        "version": "1.0.2"
    },
    "last_serial": 5453293,
    "releases": {
        "1.0.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "efe65b1699ae954ea20bf40e66d299c0",
                    "sha256": "9e96f2116c92d8bd6645bb4addb5ab0de563bcf123ccc4dc0119d9f249510dc4"
                },
                "downloads": -1,
                "filename": "ctodd_python_lib_kafka-1.0.1-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "efe65b1699ae954ea20bf40e66d299c0",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": ">3.6",
                "size": 9701,
                "upload_time": "2019-04-21T15:54:25",
                "url": "https://files.pythonhosted.org/packages/27/41/b0bd116828f48f00cce6298fef76e75b29aadbd58976aa2ebb7d2da242c2/ctodd_python_lib_kafka-1.0.1-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "84873082a078706ab599344b93e913ed",
                    "sha256": "39e032f10fe8f16ad350242dd4e9d6bfc7541b65b0478c1c420b3ade7433367c"
                },
                "downloads": -1,
                "filename": "ctodd-python-lib-kafka-1.0.1.tar.gz",
                "has_sig": false,
                "md5_digest": "84873082a078706ab599344b93e913ed",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">3.6",
                "size": 7708,
                "upload_time": "2019-04-21T15:54:27",
                "url": "https://files.pythonhosted.org/packages/16/f1/1de53b6076c62c0edc775d519827acf9ad1e28023f77805370feb1c10733/ctodd-python-lib-kafka-1.0.1.tar.gz"
            }
        ],
        "1.0.2": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "138b71ffb1513a9331e7aa1724634cc4",
                    "sha256": "bf7b5b400d0177f69a189f4d1c50d0f62bbf29e6ffa6587256ab6261b16409c5"
                },
                "downloads": -1,
                "filename": "ctodd_python_lib_kafka-1.0.2-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "138b71ffb1513a9331e7aa1724634cc4",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": ">3.6",
                "size": 11715,
                "upload_time": "2019-06-26T19:06:32",
                "url": "https://files.pythonhosted.org/packages/3e/8f/2fd2e8678d0e8ebfa96fd5a6a95167f7085b01b68f4fbeb0779df86eee59/ctodd_python_lib_kafka-1.0.2-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "0fe49def6e15b66e9717690da88baf00",
                    "sha256": "1ec6dac71d07a6a29728975c260e7a5b4313bae186386bbf7b7fedf1e9b44e77"
                },
                "downloads": -1,
                "filename": "ctodd-python-lib-kafka-1.0.2.tar.gz",
                "has_sig": false,
                "md5_digest": "0fe49def6e15b66e9717690da88baf00",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">3.6",
                "size": 9541,
                "upload_time": "2019-06-26T19:06:34",
                "url": "https://files.pythonhosted.org/packages/e2/c9/cf4ed2616b045e21cae69810e6609707b88410e12b39e6d0a0fd96e5ca5c/ctodd-python-lib-kafka-1.0.2.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "138b71ffb1513a9331e7aa1724634cc4",
                "sha256": "bf7b5b400d0177f69a189f4d1c50d0f62bbf29e6ffa6587256ab6261b16409c5"
            },
            "downloads": -1,
            "filename": "ctodd_python_lib_kafka-1.0.2-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "138b71ffb1513a9331e7aa1724634cc4",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": ">3.6",
            "size": 11715,
            "upload_time": "2019-06-26T19:06:32",
            "url": "https://files.pythonhosted.org/packages/3e/8f/2fd2e8678d0e8ebfa96fd5a6a95167f7085b01b68f4fbeb0779df86eee59/ctodd_python_lib_kafka-1.0.2-py3-none-any.whl"
        },
        {
            "comment_text": "",
            "digests": {
                "md5": "0fe49def6e15b66e9717690da88baf00",
                "sha256": "1ec6dac71d07a6a29728975c260e7a5b4313bae186386bbf7b7fedf1e9b44e77"
            },
            "downloads": -1,
            "filename": "ctodd-python-lib-kafka-1.0.2.tar.gz",
            "has_sig": false,
            "md5_digest": "0fe49def6e15b66e9717690da88baf00",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": ">3.6",
            "size": 9541,
            "upload_time": "2019-06-26T19:06:34",
            "url": "https://files.pythonhosted.org/packages/e2/c9/cf4ed2616b045e21cae69810e6609707b88410e12b39e6d0a0fd96e5ca5c/ctodd-python-lib-kafka-1.0.2.tar.gz"
        }
    ]
}