{
"info": {
"author": "Andrei Roskach",
"author_email": "code.impactor@gmail.com",
"bugtrack_url": null,
"classifiers": [
"Intended Audience :: Developers",
"License :: OSI Approved :: MIT License",
"Natural Language :: English",
"Operating System :: OS Independent",
"Programming Language :: Python :: 3.7",
"Topic :: Database",
"Topic :: Software Development :: Libraries"
],
"description": "# arque\nAsyncio Reliable Queue (based on redis)\n\nInspired by Tom DeWire's article \"Reliable Queueing in Redis (Part 1)\" [[1]](#ref1) [[2]](#ref2) and the \"torrelque\" python module [[3]](#ref3).\n\n#### Features:\n - Queue with repeats, delays and dead letters\n - Based on asyncio, aioredis\n - Tested on Python 3.7 and redis server '>=3.0.6', '<=5.0.5'\n\n#### Install:\n```bash\npip install arque\n```\n\n#### Usage:\n\n```python\nimport signal\nimport random\nimport logging\nimport asyncio\nimport aioredis\nimport time\nfrom functools import wraps\nfrom arque import Arque\n\nlogger = logging.getLogger(__name__)\n\n\nasync def shutdown(signal, loop):\n \"\"\"Cleanup tasks tied to the service's shutdown.\"\"\"\n logging.info(f\"Received exit signal {signal.name}...\")\n tasks = [t for t in asyncio.all_tasks() if t is not asyncio.current_task()]\n [task.cancel() for task in tasks]\n logging.info(f\"Cancelling {len(tasks)}outstanding tasks\")\n await asyncio.gather(*tasks)\n logging.info(f\"Flushing metrics\")\n loop.stop()\n\n\ndef aioredis_pool(host='redis://localhost', encoding='utf8'):\n def wrapper(func):\n @wraps(func)\n async def wrapped(loop, redis=None):\n redis = await aioredis.create_redis_pool(host, loop=loop, encoding=encoding)\n try:\n return await func(loop=loop, redis=redis)\n finally:\n redis.close()\n await redis.wait_closed()\n\n return wrapped\n\n return wrapper\n\n\n@aioredis_pool(host='redis://localhost', encoding='utf8')\nasync def produce_task(loop, redis=None):\n logger.info('Starting producing...')\n queue = Arque(redis=redis, loop=loop)\n while True:\n for _ in range(1):\n task = {'value': random.randint(0, 99)}\n task_id = f\"custom_{task['value']}_{time.time()}\"\n logger.debug('Produced task %s', task)\n await queue.enqueue(task, task_id=task_id, task_timeout=10, delay=1)\n await asyncio.sleep(1)\n\n\nasync def process(task_data):\n logger.debug('Consumed task %s', task_data)\n await asyncio.sleep(1)\n\n\n@aioredis_pool(host='redis://localhost', encoding='utf8')\nasync def consume_task(loop, redis=None):\n logger.info('Starting consuming...')\n queue = Arque(redis=redis, loop=loop, working_limit=3)\n while True:\n task_id, task_data = await queue.dequeue()\n if task_id == '__not_found__':\n continue\n\n if task_id == '__overloaded__':\n print(f'TASK ID: {task_id}')\n await asyncio.sleep(1)\n continue\n\n if task_id == '__marked_as_failed___':\n print(f'FAILED ID: {task_id}')\n await asyncio.sleep(1)\n continue\n\n try:\n await process(task_data)\n await queue.release(task_id)\n except Exception:\n logger.exception('Job processing has failed')\n await queue.requeue(task_id, delay=5)\n stats = await queue.get_stats()\n logger.info(stats)\n\n\n@aioredis_pool(host='redis://localhost', encoding='utf8')\nasync def sweep_task(loop, redis=None):\n logger.info('Starting sweeping...')\n queue = Arque(redis=redis, loop=loop, sweep_interval=5)\n await queue.schedule_sweep()\n\n\n@aioredis_pool(host='redis://localhost', encoding='utf8')\nasync def stats_task(loop, redis=None):\n logger.info('Starting stats...')\n queue = Arque(redis=redis, loop=loop)\n while True:\n stats = await queue.get_stats()\n logger.info(stats)\n await asyncio.sleep(5)\n\n\ndef create_tasks(loop):\n tasks = []\n for _ in range(5):\n tasks.append(consume_task(loop))\n tasks.append(produce_task(loop))\n tasks.append(sweep_task(loop))\n tasks.append(stats_task(loop))\n return tasks\n\n\nif __name__ == '__main__':\n logging.basicConfig(level=logging.DEBUG, format='%(asctime)s %(message)s')\n loop = asyncio.get_event_loop()\n signals = (signal.SIGHUP, signal.SIGTERM, signal.SIGINT, signal.SIGUSR1)\n for s in signals:\n loop.add_signal_handler(s, lambda s=s: asyncio.create_task(shutdown(s, loop)))\n try:\n loop.run_until_complete(asyncio.gather(*create_tasks(loop)))\n finally:\n loop.close()\n logging.info(\"Successfully shutdown...\")\n\n``` \n\n#### Reference\n[1] [Reliable Queueing in Redis (Part 1)](http://blog.bronto.com/engineering/reliable-queueing-in-redis-part-1/) \n[2] [DEWIRE Redis as a Reliable Work Queue.pdf](https://www.percona.com/sites/default/files/DEWIRE%20Redis%20as%20a%20Reliable%20Work%20Queue.pdf) \n[3] [torrelque](https://bitbucket.org/saaj/torrelque) \n\n\n",
"description_content_type": "text/markdown",
"docs_url": null,
"download_url": "https://github.com/code-impactor/arque/releases/tag/1.0.7",
"downloads": {
"last_day": -1,
"last_month": -1,
"last_week": -1
},
"home_page": "https://github.com/code-impactor/arque",
"keywords": "asyncio,redis,reliable,queue,asynchronous,python,reliable-queue,work-queue,delay,delayed,jobs,delayed-queue,repeated,tasks,dead letter,failed",
"license": "MIT",
"maintainer": "",
"maintainer_email": "",
"name": "arque",
"package_url": "https://pypi.org/project/arque/",
"platform": "Any",
"project_url": "https://pypi.org/project/arque/",
"project_urls": {
"Download": "https://github.com/code-impactor/arque/releases/tag/1.0.7",
"Homepage": "https://github.com/code-impactor/arque"
},
"release_url": "https://pypi.org/project/arque/1.0.7/",
"requires_dist": [
"aioredis (>=1.2.0)"
],
"requires_python": ">=3.7",
"summary": "Asyncio Reliable Queue (based on redis)",
"version": "1.0.7"
},
"last_serial": 6004303,
"releases": {
"1.0.2": [
{
"comment_text": "",
"digests": {
"md5": "6cb27487a162ad86f357bc1fa65f5db0",
"sha256": "92da458073c4ca04e5c5f2332fd0957490973719b5779f2a843b4b3d7157e281"
},
"downloads": -1,
"filename": "arque-1.0.2.tar.gz",
"has_sig": false,
"md5_digest": "6cb27487a162ad86f357bc1fa65f5db0",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 6312,
"upload_time": "2019-09-20T20:57:09",
"url": "https://files.pythonhosted.org/packages/db/15/e1259b32e17aa623582f91b58fc6b573fd5606adf9e66acf1dad0885f152/arque-1.0.2.tar.gz"
}
],
"1.0.3": [
{
"comment_text": "",
"digests": {
"md5": "f0ba677d1cc17d2f5f3d0dadef61dc80",
"sha256": "9aa6d4418b50eabd69dfaec5b41909c7c4d9a7cf7e5e993f278a8ae7d211f110"
},
"downloads": -1,
"filename": "arque-1.0.3-py3-none-any.whl",
"has_sig": false,
"md5_digest": "f0ba677d1cc17d2f5f3d0dadef61dc80",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7",
"size": 6826,
"upload_time": "2019-09-20T21:09:36",
"url": "https://files.pythonhosted.org/packages/26/4b/a6c08346d1df24fd1eea9c57ca4247a7fb3739887532df4d98c5c3fa439f/arque-1.0.3-py3-none-any.whl"
}
],
"1.0.4": [
{
"comment_text": "",
"digests": {
"md5": "2b21b34c26b08a2fa57e9754bf2b392c",
"sha256": "44101a163ecfef616876dd1befa99b06a43429cadd3a6919aace464683fc4ad2"
},
"downloads": -1,
"filename": "arque-1.0.4-py2-none-any.whl",
"has_sig": false,
"md5_digest": "2b21b34c26b08a2fa57e9754bf2b392c",
"packagetype": "bdist_wheel",
"python_version": "py2",
"requires_python": ">=3.7",
"size": 6870,
"upload_time": "2019-09-20T21:29:02",
"url": "https://files.pythonhosted.org/packages/80/f6/a39e8a0abeea91cbb6bd2f94b06a689bed8ba142a5cd7eb751f4c4ea2df6/arque-1.0.4-py2-none-any.whl"
},
{
"comment_text": "",
"digests": {
"md5": "c8385db23e3730adecdd43ab88502940",
"sha256": "35aad10aabe3fe4c4394a42e2c3260183f1e9a285c0d4e491712e97e664bfcbe"
},
"downloads": -1,
"filename": "arque-1.0.4-py3-none-any.whl",
"has_sig": false,
"md5_digest": "c8385db23e3730adecdd43ab88502940",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7",
"size": 6868,
"upload_time": "2019-09-20T22:12:38",
"url": "https://files.pythonhosted.org/packages/55/62/4381af738d925154103611b26c9c5e92baff89a772978856ec3d72fdcae2/arque-1.0.4-py3-none-any.whl"
},
{
"comment_text": "",
"digests": {
"md5": "0ed3331ef536eef4f758fbfc0b9864c8",
"sha256": "cbdb551b6048acd9cd6e71afc354d0e2423243f632b1cd5e57b062b26e3ab022"
},
"downloads": -1,
"filename": "arque-1.0.4.tar.gz",
"has_sig": false,
"md5_digest": "0ed3331ef536eef4f758fbfc0b9864c8",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 6348,
"upload_time": "2019-09-20T22:12:40",
"url": "https://files.pythonhosted.org/packages/9f/b4/9a039014ad44363d6a6967d6893de66be9c32ffe6398bb8c83fd0b21f03e/arque-1.0.4.tar.gz"
}
],
"1.0.5": [
{
"comment_text": "",
"digests": {
"md5": "9a5ba32f4302804bece839a261f8258d",
"sha256": "fed752523d4615b94e0cb6aa5d3bffa3a8fd63afe6130dbcf2cd1aad315673b7"
},
"downloads": -1,
"filename": "arque-1.0.5-py3-none-any.whl",
"has_sig": false,
"md5_digest": "9a5ba32f4302804bece839a261f8258d",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7",
"size": 6933,
"upload_time": "2019-09-21T17:13:46",
"url": "https://files.pythonhosted.org/packages/05/eb/31d08bf91cac7d59b9e377f73389f3ac807aa7833573833cc4afc97dd78c/arque-1.0.5-py3-none-any.whl"
},
{
"comment_text": "",
"digests": {
"md5": "1cd85f82ebe0010fcc9aa84e7cc66a95",
"sha256": "5c3bc1cd826e243ea940e95d55d02906d60b9a28992df96a300882b9ae6e43a2"
},
"downloads": -1,
"filename": "arque-1.0.5.tar.gz",
"has_sig": false,
"md5_digest": "1cd85f82ebe0010fcc9aa84e7cc66a95",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 6371,
"upload_time": "2019-09-21T17:13:48",
"url": "https://files.pythonhosted.org/packages/fc/2c/953a787229afa3f0425e61c5b2970c243910b2c6b185b77eab3aecf0dfa4/arque-1.0.5.tar.gz"
}
],
"1.0.6": [
{
"comment_text": "",
"digests": {
"md5": "188e3aa9ddc0c4cbb468216fb1e65fa1",
"sha256": "233cc563dde86adbf2de3bc62d85975b8490d6c7c0c9fcb7a9e77880dfa47453"
},
"downloads": -1,
"filename": "arque-1.0.6-py3-none-any.whl",
"has_sig": false,
"md5_digest": "188e3aa9ddc0c4cbb468216fb1e65fa1",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7",
"size": 6950,
"upload_time": "2019-09-23T18:09:47",
"url": "https://files.pythonhosted.org/packages/30/f9/cf122d0683b03e3c4d6f2618145453e52898b5b62ef71c79fdbf9146df9d/arque-1.0.6-py3-none-any.whl"
},
{
"comment_text": "",
"digests": {
"md5": "a8303f43022387222e59b6bbc2a5faa9",
"sha256": "617916617ac25c917e46ede82cdd5a7298ab40a3e78759e9932074660b3dd634"
},
"downloads": -1,
"filename": "arque-1.0.6.tar.gz",
"has_sig": false,
"md5_digest": "a8303f43022387222e59b6bbc2a5faa9",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 6407,
"upload_time": "2019-09-23T18:09:50",
"url": "https://files.pythonhosted.org/packages/fa/b3/a37fc84e1dfcf22f08dd9c692c8429a218c36b09e745c4d06d09dd15a98f/arque-1.0.6.tar.gz"
}
],
"1.0.7": [
{
"comment_text": "",
"digests": {
"md5": "b83c5b58eba6c7e96ff2b104e9d43fd5",
"sha256": "094ae7c46d77cb710893cefb940489b5ebef00980a8af04d708cf2e551e45d9b"
},
"downloads": -1,
"filename": "arque-1.0.7-py3-none-any.whl",
"has_sig": false,
"md5_digest": "b83c5b58eba6c7e96ff2b104e9d43fd5",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7",
"size": 7033,
"upload_time": "2019-10-20T18:48:33",
"url": "https://files.pythonhosted.org/packages/63/56/eb40b377f65fb345857535f9b533070613be8f11080e684afd9e429fa738/arque-1.0.7-py3-none-any.whl"
},
{
"comment_text": "",
"digests": {
"md5": "4f51ad3c22174db73c400dbd6700159e",
"sha256": "bede2ff7ac78a433b44446c56a329a0fae52acc498c04644c3152ff492953bd7"
},
"downloads": -1,
"filename": "arque-1.0.7.tar.gz",
"has_sig": false,
"md5_digest": "4f51ad3c22174db73c400dbd6700159e",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 6490,
"upload_time": "2019-10-20T18:48:36",
"url": "https://files.pythonhosted.org/packages/d6/80/6aaf61b99c96fcdc9d90020f664727c9cb8cf3a55a76a2c4780bd5cab25d/arque-1.0.7.tar.gz"
}
]
},
"urls": [
{
"comment_text": "",
"digests": {
"md5": "b83c5b58eba6c7e96ff2b104e9d43fd5",
"sha256": "094ae7c46d77cb710893cefb940489b5ebef00980a8af04d708cf2e551e45d9b"
},
"downloads": -1,
"filename": "arque-1.0.7-py3-none-any.whl",
"has_sig": false,
"md5_digest": "b83c5b58eba6c7e96ff2b104e9d43fd5",
"packagetype": "bdist_wheel",
"python_version": "py3",
"requires_python": ">=3.7",
"size": 7033,
"upload_time": "2019-10-20T18:48:33",
"url": "https://files.pythonhosted.org/packages/63/56/eb40b377f65fb345857535f9b533070613be8f11080e684afd9e429fa738/arque-1.0.7-py3-none-any.whl"
},
{
"comment_text": "",
"digests": {
"md5": "4f51ad3c22174db73c400dbd6700159e",
"sha256": "bede2ff7ac78a433b44446c56a329a0fae52acc498c04644c3152ff492953bd7"
},
"downloads": -1,
"filename": "arque-1.0.7.tar.gz",
"has_sig": false,
"md5_digest": "4f51ad3c22174db73c400dbd6700159e",
"packagetype": "sdist",
"python_version": "source",
"requires_python": ">=3.7",
"size": 6490,
"upload_time": "2019-10-20T18:48:36",
"url": "https://files.pythonhosted.org/packages/d6/80/6aaf61b99c96fcdc9d90020f664727c9cb8cf3a55a76a2c4780bd5cab25d/arque-1.0.7.tar.gz"
}
]
}