{ "info": { "author": "Joohwan Oh", "author_email": "joohwan.oh@outlook.com", "bugtrack_url": null, "classifiers": [ "Intended Audience :: Developers", "Intended Audience :: End Users/Desktop", "Intended Audience :: Information Technology", "Intended Audience :: Science/Research", "Intended Audience :: System Administrators", "License :: OSI Approved :: MIT License", "Operating System :: MacOS", "Operating System :: POSIX", "Operating System :: Unix", "Programming Language :: Python", "Programming Language :: Python :: 3", "Topic :: Internet", "Topic :: Scientific/Engineering", "Topic :: System :: Distributed Computing", "Topic :: System :: Monitoring", "Topic :: System :: Systems Administration" ], "description": "KQ: Kafka-based Job Queue for Python\n------------------------------------\n\n.. image:: https://travis-ci.org/joowani/kq.svg?branch=master\n :target: https://travis-ci.org/joowani/kq\n :alt: Build Status\n\n.. image:: https://readthedocs.org/projects/kq/badge/?version=latest\n :target: http://kq.readthedocs.io/en/latest/?badge=latest\n :alt: Documentation Status\n\n.. image:: https://badge.fury.io/py/kq.svg\n :target: https://badge.fury.io/py/kq\n :alt: Package Version\n\n.. image:: https://img.shields.io/badge/python-3.5%2C%203.6-blue.svg\n :target: https://github.com/joowani/kq\n :alt: Python Versions\n\n.. image:: https://coveralls.io/repos/github/joowani/kq/badge.svg?branch=master\n :target: https://coveralls.io/github/joowani/kq?branch=master\n :alt: Test Coverage\n\n.. image:: https://img.shields.io/github/issues/joowani/kq.svg\n :target: https://github.com/joowani/kq/issues\n :alt: Issues Open\n\n.. image:: https://img.shields.io/badge/license-MIT-blue.svg\n :target: https://raw.githubusercontent.com/joowani/kq/master/LICENSE\n :alt: MIT License\n\n|\n\n**KQ (Kafka Queue)** is a lightweight Python library which lets you queue and\nexecute jobs asynchronously using `Apache Kafka`_. It uses kafka-python_ under\nthe hood.\n\nAnnouncements\n=============\n\n* KQ version `2.0.0`_ is now out!\n* Please see the releases_ page for latest updates.\n\nRequirements\n============\n\n* `Apache Kafka`_ 0.9+\n* Python 3.5+\n\nInstallation\n============\n\nTo install a stable version from PyPI_ (recommended):\n\n.. code-block:: bash\n\n ~$ pip install kq\n\nTo install the latest version directly from GitHub_:\n\n.. code-block:: bash\n\n ~$ pip install -e git+git@github.com:joowani/kq.git@master#egg=kq\n\nYou may need to use ``sudo`` depending on your environment.\n\nGetting Started\n===============\n\nFirst, ensure that your Kafka instance is up and running:\n\n.. code-block:: bash\n\n ~$ ./kafka-server-start.sh -daemon server.properties\n\nDefine your KQ worker module:\n\n.. code-block:: python\n\n # my_worker.py\n\n import logging\n\n from kafka import KafkaConsumer\n from kq import Worker\n\n # Set up logging.\n formatter = logging.Formatter('[%(levelname)s] %(message)s')\n stream_handler = logging.StreamHandler()\n stream_handler.setFormatter(formatter)\n logger = logging.getLogger('kq.worker')\n logger.setLevel(logging.DEBUG)\n logger.addHandler(stream_handler)\n\n # Set up a Kafka consumer.\n consumer = KafkaConsumer(\n bootstrap_servers='127.0.0.1:9092',\n group_id='group',\n auto_offset_reset='latest'\n )\n\n # Set up a worker.\n worker = Worker(topic='topic', consumer=consumer)\n worker.start()\n\nStart the worker:\n\n.. code-block:: bash\n\n ~$ python my_worker.py\n [INFO] Starting Worker(hosts=127.0.0.1:9092 topic=topic, group=group) ...\n\nEnqueue a function call:\n\n.. code-block:: python\n\n import requests\n\n from kafka import KafkaProducer\n from kq import Queue\n\n # Set up a Kafka producer.\n producer = KafkaProducer(bootstrap_servers='127.0.0.1:9092')\n\n # Set up a queue.\n queue = Queue(topic='topic', producer=producer)\n\n # Enqueue a function call.\n job = queue.enqueue(requests.get, 'https://www.google.com')\n\nSit back and watch the worker process it in the background:\n\n.. code-block:: bash\n\n ~$ python my_worker.py\n [INFO] Starting Worker(hosts=127.0.0.1:9092, topic=topic, group=group) ...\n [INFO] Processing Message(topic=topic, partition=0, offset=0) ...\n [INFO] Executing job c7bf2359: requests.api.get('https://www.google.com')\n [INFO] Job c7bf2359 returned: \n\n**NEW in 2.0.0**: You can now specify the job timeout, message key and partition:\n\n.. code-block:: python\n\n job = queue.using(timeout=5, key=b'foo', partition=0).enqueue(requests.get, 'https://www.google.com')\n\nCheck out the full documentation_ for more information.\n\nContributing\n============\n\nPlease have a look at this page_ before submitting a pull request. Thanks!\n\n\nCredits\n=======\n\nThis project was inspired by RQ_.\n\n.. _Apache Kafka: https://kafka.apache.org\n.. _kafka-python: https://github.com/dpkp/kafka-python\n.. _2.0.0: https://github.com/joowani/kq/releases/tag/2.0.0\n.. _releases: https://github.com/joowani/kq/releases\n.. _PyPI: https://pypi.python.org/pypi/kq\n.. _GitHub: https://github.com/joowani/kq\n.. _documentation: http://kq.readthedocs.io\n.. _page: http://kq.readthedocs.io/en/master/contributing.html\n.. _RQ: https://github.com/rq/rq\n", "description_content_type": "", "docs_url": null, "download_url": "", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "https://github.com/joowani/kq", "keywords": "", "license": "MIT", "maintainer": "", "maintainer_email": "", "name": "kq", "package_url": "https://pypi.org/project/kq/", "platform": "", "project_url": "https://pypi.org/project/kq/", "project_urls": { "Homepage": "https://github.com/joowani/kq" }, "release_url": "https://pypi.org/project/kq/2.0.0/", "requires_dist": null, "requires_python": "", "summary": "Kafka Job Queue for Python", "version": "2.0.0" }, "last_serial": 3926786, "releases": { "0.0.1": [ { "comment_text": "", "digests": { "md5": "4df640129d0c6cc81edd38930bb26d82", "sha256": "43f506345598b53eddaee029695d9af633a6752c9e1b5fbeb9d9a45b3a9aa519" }, "downloads": -1, "filename": "kq-0.0.1.tar.gz", "has_sig": false, "md5_digest": "4df640129d0c6cc81edd38930bb26d82", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 5489, "upload_time": "2016-10-21T04:29:27", "url": "https://files.pythonhosted.org/packages/69/60/ef34511807714446ba0823933e385a2ef931d6d90f1bcc02bc6dc091508a/kq-0.0.1.tar.gz" } ], "1.0": [ { "comment_text": "", "digests": { "md5": "188a08fa96396031b52cdf4e0c4595bc", "sha256": "c34fe3aa20fa2dbb8a31cab271dc8df962dbe5cf353f50ac1e579d2db62b4d7d" }, "downloads": -1, "filename": "kq-1.0.tar.gz", "has_sig": false, "md5_digest": "188a08fa96396031b52cdf4e0c4595bc", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 12458, "upload_time": "2016-10-31T04:26:59", "url": "https://files.pythonhosted.org/packages/d8/74/afffff22d2ae44a05aee10304a9efe1c78a95770b4d486a92a2a47273f87/kq-1.0.tar.gz" } ], "1.0.1": [ { "comment_text": "", "digests": { "md5": "4ba52fd024c142a76618d12d5648caf9", "sha256": "678a20c7f5f5f948591da54b2c609d4e820bf31445cbf8fbde1fad815d5fb6ea" }, "downloads": -1, "filename": "kq-1.0.1.tar.gz", "has_sig": false, "md5_digest": "4ba52fd024c142a76618d12d5648caf9", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 12493, "upload_time": "2016-12-04T07:19:50", "url": "https://files.pythonhosted.org/packages/1f/17/7c84bffafb44247d78aea5670e6f9380cb4efba6105098f677568255c5f4/kq-1.0.1.tar.gz" } ], "1.1.0": [ { "comment_text": "", "digests": { "md5": "4004082d458af994b1620d7ffb1ef286", "sha256": "fecf0558e5901fa1fd1a50f0ea3123afc381317e5fe095cba1c094ade00fc2ad" }, "downloads": -1, "filename": "kq-1.1.0.tar.gz", "has_sig": false, "md5_digest": "4004082d458af994b1620d7ffb1ef286", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 12839, "upload_time": "2016-12-13T14:24:32", "url": "https://files.pythonhosted.org/packages/98/23/4e0a62cd79bb463301fe7e6e1a365a6dda2f422933a58aa6f9a6a9a3cbe1/kq-1.1.0.tar.gz" } ], "1.2.0": [ { "comment_text": "", "digests": { "md5": "89230f5726bba7c4071120010bf6d4de", "sha256": "8c567082475632be046776cc76b4311c721360d727b21090638d1c95fdafd5a4" }, "downloads": -1, "filename": "kq-1.2.0.tar.gz", "has_sig": false, "md5_digest": "89230f5726bba7c4071120010bf6d4de", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 13495, "upload_time": "2017-05-09T09:04:50", "url": "https://files.pythonhosted.org/packages/b0/cd/1a337d566a3577b0aa470085a3bbbece358146fc32f289a915d8472faf7e/kq-1.2.0.tar.gz" } ], "1.3.0": [ { "comment_text": "", "digests": { "md5": "e98d47bb05bb0e67298027bcae9890f2", "sha256": "19602daee01a93857c38adc767f95a387fbae6b8e217ea25c8be3084ab0674a6" }, "downloads": -1, "filename": "kq-1.3.0.tar.gz", "has_sig": false, "md5_digest": "e98d47bb05bb0e67298027bcae9890f2", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 13656, "upload_time": "2017-05-10T08:23:00", "url": "https://files.pythonhosted.org/packages/a8/33/0a32d37a678cb353654e363991199feb341c75c1a59908e7bea413ce3bed/kq-1.3.0.tar.gz" } ], "1.3.1": [ { "comment_text": "", "digests": { "md5": "98c7226b7f1d15b1096cc0dfd83a0727", "sha256": "8170e2ca868aa711461265f8316719aedabf1e6e8d9cff8fa9762d15cc36b651" }, "downloads": -1, "filename": "kq-1.3.1.tar.gz", "has_sig": false, "md5_digest": "98c7226b7f1d15b1096cc0dfd83a0727", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 13656, "upload_time": "2017-11-02T09:56:36", "url": "https://files.pythonhosted.org/packages/79/dd/fb9e0a21f54f050f16e5a2382d5a4a5876fc41f5338e654e2a6fb5a2936e/kq-1.3.1.tar.gz" } ], "1.3.2": [ { "comment_text": "", "digests": { "md5": "bfe3aee5956c7153c28f2241eb1f2748", "sha256": "9f5ad2a7c0c211af8118eeec36cad544d42a0239557a034493ae130749fe4d95" }, "downloads": -1, "filename": "kq-1.3.2.tar.gz", "has_sig": false, "md5_digest": "bfe3aee5956c7153c28f2241eb1f2748", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 13762, "upload_time": "2017-11-09T10:44:10", "url": "https://files.pythonhosted.org/packages/ec/be/6dc72304a84e1a190905472822f44244b68b07f13dc20ea982b86ab613e6/kq-1.3.2.tar.gz" } ], "2.0.0": [ { "comment_text": "", "digests": { "md5": "2653a9f6bfd089291436cbed057df203", "sha256": "5ef36c969bfb5dc535fbb9b956f3700585d3b7185bc68bda39c53dc59444a439" }, "downloads": -1, "filename": "kq-2.0.0.tar.gz", "has_sig": false, "md5_digest": "2653a9f6bfd089291436cbed057df203", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 9669, "upload_time": "2018-06-04T02:50:54", "url": "https://files.pythonhosted.org/packages/1c/3e/eb9053c63c23cf885cf54ad95b5641f029f40f2bc0985af2e3e62d2b97bb/kq-2.0.0.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "2653a9f6bfd089291436cbed057df203", "sha256": "5ef36c969bfb5dc535fbb9b956f3700585d3b7185bc68bda39c53dc59444a439" }, "downloads": -1, "filename": "kq-2.0.0.tar.gz", "has_sig": false, "md5_digest": "2653a9f6bfd089291436cbed057df203", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 9669, "upload_time": "2018-06-04T02:50:54", "url": "https://files.pythonhosted.org/packages/1c/3e/eb9053c63c23cf885cf54ad95b5641f029f40f2bc0985af2e3e62d2b97bb/kq-2.0.0.tar.gz" } ] }