{ "info": { "author": "Tarasenko Stepan", "author_email": "st.a.tarasenko@gmail.com", "bugtrack_url": null, "classifiers": [ "License :: OSI Approved :: MIT License", "Operating System :: OS Independent", "Programming Language :: Python :: 3" ], "description": "First of all, we need to start confluent kafka server, use this article, to know how to do it:\n https://docs.confluent.io/3.0.0/control-center/docs/quickstart.html\n\nThe purpose of this project is:\n\t- consume binary avro, split it into different topics\n\nTo install this project, you need:\n\t- install all of packages, what are in packages folder\n\t- set up postgres server and execute create_config_tables.sql and insert_to_config_tables.sql files\n\t- put binary_avro_consumer.py and conf.cnf on server and execute python file with command\n\t\tpython3.6 binary_avro_consumer.py (params)\n\nMore about console execution parameters:\n\n\nCreate tables statements are stored in create_config_tables.sql file.\nInsert into config tables statements are stored in insert_to_config_tables.sql file.\nYou should execute the create statements, and insert into those tables your settings.\nYou got such structure for tables:\n\n\n config_key | config_value\n --------------------+----------------\n\n\n topic_name | field_name\n --------------+-------------\n\nEnd here is an example of their filling:\n\n --The config key, means the key of some setting, there an explanation of their meaning--\n\n --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- \n\n bootstrap_server_from - The bootstrap server from what we have messages, can be multiple times in db, because of multiple bootstrap server, kafka is cluster.\n bootstrap_server_from_port - The port of those bootstrap servers, usually bootstrap servers have the same port.\\\n schema_registry - The schema registry url, should starts with http:// or https://\n schema_registry_port - The schema registry port\n topic_read - The topic, from what we need to read messages, so this topic is from `bootstrap_server_from` server.\n group_id - Usually uses default name `example_avro`, this parameter required for consuming\n bootstrap_server_to - The server to what we writes messages, what we read and modified in `bootstrap_server_from`\n bootstrap_server_to_port - The port of `bootstrap_server_to`\n from_beginning - start consuming from beginning 1 - true, 0 - false \n count_messages_consume - count of messages, what consumes per one iteration\n\n config_key | config_value\n ----------------------------+----------------\n bootstrap_server_from | localhost\n bootstrap_server_from_port | 9092\n schema_registry | http://0.0.0.0\n schema_registry_port | 8081\n topic_read | avro-test\n group_id | example_avro\n bootstrap_server_to | localhost\n bootstrap_server_to_port | 9092\n from_beginning | 1\n count_messages_consume | 100\n\n\n --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- \n\n\n topic_name | field_name\n --------------+-------------\n first_topic | uid\n first_topic | somefield\n second_topic | options hel\n second_topic | options mel\n\n\n For example, you have such avro schema:\n\n \"namespace\" : \"my.com.ns\",\n \"name\": \"myrecord\",\n \"type\" : \"record\",\n \"fields\" : [\n {\"name\": \"uid\", \"type\": \"int\"},\n {\"name\": \"somefield\", \"type\": \"string\"},\n {\"name\": \"options\", \"type\": {\n \"type\": \"array\",\n \"items\": {\n \"type\": \"record\",\n \"name\": \"lvl2_record\",\n \"fields\": [\n {\"name\": \"hel\", \"type\": \"string\"},\n {\"name\": \"mel\", \"type\": \"string\"}\n }\n ]\n }\n }\n }\n\n ]\n }\n\n You need to extract such values from this schema: \n\n uid, somefield, options->hel, options->mel, and you need to store this values in first_topic and second_topic, so for example, we store uid and somefield in first_topic, \n and options->hel, options->mel in second_topic. options->hel, options->mel means that field hel is a child of options, the same for mel.\n\n So we write to db: first_topic uid,somefield , what means, plz store uid and somefield in first_topic, the same for second_topic.\n\n --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- \n\n\nHow to understand where actually your avro schema stored in schema registry?\nHere you an answer:\n\nImagine, that you created a topic named test, and registered a schema to schema registry, to know what schema is, and to track if that schema changed, you need:\n To execute such command in terminal, schema registry server should work (insted of http://localhost:8081/subjects you should put you schema registry url):\n\tcurl -X GET http://localhost:8081/subjects\n\n\tOutput of curl: [\"Kafka-value\",\"Kafka-key\",\"test-value\"]\n\n You see, that your test topic also created 'test-value' subject, so the schema what you need is\n http://localhost:8081/subjects/test-value/versions/latest\",\n in some cases your schema can be situated by `Kafka-value` path, so your url will be: http://localhost:8081/subjects/Kafka-value/versions/latest\"\n\n\tChange this url conf.cnf file, more about conf.cnf:\n\n conf.cnf, is config for python script:\n\n On FIRST line is schema registry url.\n On SECOND line is DB name.\n On THIRD line is username.\n On fourth line is password.\n On sixth line is host.\n\tOn the seventh line is the number of avro messages to produce by AvroProducer.\n\n\n\nAfter all of this, you need to start the script called pushpop_complex_avro.py, \nby such command python3.6 pushpop_complex_avro.py (optional params -d -i -e, for debug,info,error respectively).\nAfter script is running, it wait for messages, so you need to produce messages to topic_read:\n python3.6 avro_producer.py\nAnd see, that messages were splited and directed to first_topic and second_topic.\n\n\n\n", "description_content_type": "text/markdown", "docs_url": null, "download_url": "", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "https://example.com", "keywords": "", "license": "", "maintainer": "", "maintainer_email": "", "name": "kafka-avro-binary-consumer", "package_url": "https://pypi.org/project/kafka-avro-binary-consumer/", "platform": "", "project_url": "https://pypi.org/project/kafka-avro-binary-consumer/", "project_urls": { "Homepage": "https://example.com" }, "release_url": "https://pypi.org/project/kafka-avro-binary-consumer/0.0.0.2/", "requires_dist": null, "requires_python": "", "summary": "Kafka avro binary consumer with postgres config", "version": "0.0.0.2" }, "last_serial": 5416126, "releases": { "0.0.0.1": [ { "comment_text": "", "digests": { "md5": "a862d426052e85379289e7179a97313c", "sha256": "e541f08e5966e8d6dad8512466fc0f2eda1fd8e819c4e04760d5e06f890151ba" }, "downloads": -1, "filename": "kafka_avro_binary_consumer-0.0.0.1-py3-none-any.whl", "has_sig": false, "md5_digest": "a862d426052e85379289e7179a97313c", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": null, "size": 6993, "upload_time": "2019-06-18T16:09:39", "url": "https://files.pythonhosted.org/packages/ae/d7/c6823b4cbb10ae6fb46f4b5a01607ddcd8aba0749e9f9d90c389fc737772/kafka_avro_binary_consumer-0.0.0.1-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "a4320323bff9bf1890cf9e1739d4b8c0", "sha256": "d827263f65b422c77b452017f57715f2da5a16c343ebc5caa82555ed98ad639c" }, "downloads": -1, "filename": "kafka_avro_binary_consumer-0.0.0.1.tar.gz", "has_sig": false, "md5_digest": "a4320323bff9bf1890cf9e1739d4b8c0", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 5699, "upload_time": "2019-06-18T16:09:41", "url": "https://files.pythonhosted.org/packages/18/f0/77af15131fddc9b3e07e46bfd9df4fc30972f8f6090d1f296c1b9117b2c1/kafka_avro_binary_consumer-0.0.0.1.tar.gz" } ], "0.0.0.2": [ { "comment_text": "", "digests": { "md5": "1ad3ec615cb7cc8f8453f4946619c283", "sha256": "bf4bcad8931b381c1b29308685832b9aef85021fc14f500e2ffbde8efab86c94" }, "downloads": -1, "filename": "kafka_avro_binary_consumer-0.0.0.2-py3-none-any.whl", "has_sig": false, "md5_digest": "1ad3ec615cb7cc8f8453f4946619c283", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": null, "size": 7006, "upload_time": "2019-06-18T16:18:21", "url": "https://files.pythonhosted.org/packages/83/64/4c8daa86487e4b62e86c4b1fc6f782eae526aeb112c04f7a00bc72150366/kafka_avro_binary_consumer-0.0.0.2-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "e6be1d8328d7bb5da44a1809fe7e858b", "sha256": "95143731211d388f10df1da3b010c105b54f7a044efc80a969ceda665520f7de" }, "downloads": -1, "filename": "kafka_avro_binary_consumer-0.0.0.2.tar.gz", "has_sig": false, "md5_digest": "e6be1d8328d7bb5da44a1809fe7e858b", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 5698, "upload_time": "2019-06-18T16:18:22", "url": "https://files.pythonhosted.org/packages/ff/e2/43f9957ee9c2a1146d9065e37aad96a6fd4ad449e92d74e925e6c90897b8/kafka_avro_binary_consumer-0.0.0.2.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "1ad3ec615cb7cc8f8453f4946619c283", "sha256": "bf4bcad8931b381c1b29308685832b9aef85021fc14f500e2ffbde8efab86c94" }, "downloads": -1, "filename": "kafka_avro_binary_consumer-0.0.0.2-py3-none-any.whl", "has_sig": false, "md5_digest": "1ad3ec615cb7cc8f8453f4946619c283", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": null, "size": 7006, "upload_time": "2019-06-18T16:18:21", "url": "https://files.pythonhosted.org/packages/83/64/4c8daa86487e4b62e86c4b1fc6f782eae526aeb112c04f7a00bc72150366/kafka_avro_binary_consumer-0.0.0.2-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "e6be1d8328d7bb5da44a1809fe7e858b", "sha256": "95143731211d388f10df1da3b010c105b54f7a044efc80a969ceda665520f7de" }, "downloads": -1, "filename": "kafka_avro_binary_consumer-0.0.0.2.tar.gz", "has_sig": false, "md5_digest": "e6be1d8328d7bb5da44a1809fe7e858b", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 5698, "upload_time": "2019-06-18T16:18:22", "url": "https://files.pythonhosted.org/packages/ff/e2/43f9957ee9c2a1146d9065e37aad96a6fd4ad449e92d74e925e6c90897b8/kafka_avro_binary_consumer-0.0.0.2.tar.gz" } ] }