{ "info": { "author": "Anthony Grimes", "author_email": "anthony@scopely.com", "bugtrack_url": null, "classifiers": [ "Development Status :: 3 - Alpha", "Natural Language :: English", "Programming Language :: Python :: 3.5" ], "description": "Asyncio Throttler\n=================\n\n``asyncio`` and ``async``/``await`` are pretty awesome tools!\nAsynchronous stuff! In Python! Without insanity!\n\nWell, unfortunately given the nature of this style of programming it's\nnot bloody easy sometimes.\n\nWell, this project provides at least one tool to help ya out in a\nreasonably simple way.\n\n*asyncio\\_throttler* is a throttling system for Python 3.5+, designed\naiohttp throttling in mind but designed to be flexible enough to handle\nmost throttling and rate limiting needs.\n\nUsage\n-----\n\nWell, get it:\n\n::\n\n $ pyvenv env/\n $ . env/bin/activate\n $ pip install asyncio_throttler # Pin the damned version in setup.py\n # you bloody savage.\n\nIf you don't have Python 3.5, check out\n`pyenv `__ (not to be confused with\npy\\ **v**\\ env), a Python version manager similar to rbenv.\n\nIf you really don't want pyenv, ``brew install python3`` will\nnon-destructively install Python 3.5+.\n\nAnyways, here's the terrible usage example I wrote while developing the\nthing. The code is well documented, concise, and hopefully easily\nunderstandable by humans, but this should getcha started. I'll make\nbetter docs, I promise. I gotta sleep at some point.\n\n**WARNING: THIS WILL NEVER COMPLETE, THE THROTTLE ERROR WILL BOUNCE\nAROUND FOREVER. ON PURPOSE. TO DEMONSTRATE THINGS DON'T GET LOST.\nSERIOUSLY.**\n\n.. code:: python\n\n \"\"\"Dump test module I built while writing this thing. Need to make real tests,\n but whatcha gonna do ya got schedules and stuff amirite?\n\n \"\"\"\n import logging\n import asyncio\n from asyncio_throttler import Throttler, ThrottleException\n\n # Demonstrates that windowing, throttling, and every other known feature\n # works, I think.\n if __name__ == '__main__':\n logger = logging.getLogger('testthrottler')\n logger.setLevel(logging.DEBUG)\n handler = logging.StreamHandler()\n format_template = '%(asctime)s:%(name)s:%(levelname)s \u2013 %(message)s'\n handler.setFormatter(logging.Formatter(fmt=format_template,\n datefmt='%Y-%m-%d %H:%M:%S'))\n logger.addHandler(handler)\n\n async def dummy_consumer(item):\n print(\"Item received:\", item)\n await asyncio.sleep(2)\n\n import random\n async def dummy_task():\n logger.info(\"Executed\")\n return await asyncio.sleep(1, random.randrange(1, 1000))\n\n async def bad_dummy_task():\n logger.info(\"Executed and gonna throw a throttle\")\n raise ThrottleException(bad_dummy_task())\n\n loop = asyncio.get_event_loop()\n\n # roflcoptr\n todo_list = [dummy_task() for _ in range(1, 31)]\n todo_list.append(bad_dummy_task())\n todo_list = todo_list + [dummy_task() for _ in range(1, 31)]\n\n throttler = Throttler(\n todo_list,\n dummy_consumer,\n time_window=10,\n per_time_window=20,\n concurrency=5,\n log_handler=logging.StreamHandler(),\n log_level=logging.DEBUG,\n loop=loop\n )\n\n loop.run_until_complete(throttler.run())\n loop.close()\n\nHow's It Works\n--------------\n\nA Throttler is instantiated with a list of awaitables, an async\nfunction, and numerous keyword arg knobs you adjust to suit your\npurposes.\n\nInside are two ``asyncio.Queue`` objects, and one ``asyncio.LifoQueue``.\n\n- ``exceptions`` is a ``Queue`` for non-throttle exceptions we catch.\n- ``processed`` is a ``Queue`` for processed output. This is what your\n consumer will consume from.\n- ``todo`` is a ``LifoQueue`` that holds your unprocessed task list.\n It's initially fed from a ``reverse`` of the list you pass to\n ``Throttler``, which is fast and an iterator. It's ``LIFO`` just so\n we can pop throttled items back into it at the front.\n\nSeveral internal functions are composed to create an async producer and\nconsumer loop where items are processed as fast as possible given the\nrestrictions imposed at ``Throttler`` instantiation. It'll backoff\n``time_window`` when throttled, only execute ``concurrency`` of your\ntasks at a time, and will wait ``time_window`` after triggering the\nprocessing of ``per_time_window`` items.\n\nThat oughta cover a few cases...\n\nAnyways, the async ``consumer_fn`` you pass in will be executed as\nresults become available, immediately, for writing to disk or somethin'.\n\nNotes\n-----\n\nThis was painful.", "description_content_type": null, "docs_url": null, "download_url": null, "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "https://github.com/scopely/asyncio_throttler", "keywords": null, "license": "MIT", "maintainer": null, "maintainer_email": null, "name": "asyncio-throttler", "package_url": "https://pypi.org/project/asyncio-throttler/", "platform": "UNKNOWN", "project_url": "https://pypi.org/project/asyncio-throttler/", "project_urls": { "Homepage": "https://github.com/scopely/asyncio_throttler" }, "release_url": "https://pypi.org/project/asyncio-throttler/0.1.0/", "requires_dist": null, "requires_python": null, "summary": "Throttling tools for py3.5+ asyncio.", "version": "0.1.0" }, "last_serial": 1926248, "releases": { "0.1.0": [ { "comment_text": "", "digests": { "md5": "68ff96e896425a422fa0035c2d3dfe33", "sha256": "76431861e2fcb17a14007009cfaa4f26db3fbe6b53874b5802dfc16f3a943ca7" }, "downloads": -1, "filename": "asyncio_throttler-0.1.0-py3-none-any.whl", "has_sig": false, "md5_digest": "68ff96e896425a422fa0035c2d3dfe33", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": null, "size": 8912, "upload_time": "2016-01-27T21:27:34", "url": "https://files.pythonhosted.org/packages/aa/f3/4a82944b6bb7c83af4b700a2c33cbf5dbf0c09b96f56d3e939f3e69ffb84/asyncio_throttler-0.1.0-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "62bb5db485c86d519db992f5f08070a8", "sha256": "64cb52715fdd91eab32ad5db7cb32dae62b1a20ad4af8d4abdf402cba42a4325" }, "downloads": -1, "filename": "asyncio_throttler-0.1.0.tar.gz", "has_sig": false, "md5_digest": "62bb5db485c86d519db992f5f08070a8", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 5896, "upload_time": "2016-01-27T21:28:00", "url": "https://files.pythonhosted.org/packages/4f/a1/be96073700512b4eda0cd9faf2047a36d9f0e859c97bc55209ac60db5d57/asyncio_throttler-0.1.0.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "68ff96e896425a422fa0035c2d3dfe33", "sha256": "76431861e2fcb17a14007009cfaa4f26db3fbe6b53874b5802dfc16f3a943ca7" }, "downloads": -1, "filename": "asyncio_throttler-0.1.0-py3-none-any.whl", "has_sig": false, "md5_digest": "68ff96e896425a422fa0035c2d3dfe33", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": null, "size": 8912, "upload_time": "2016-01-27T21:27:34", "url": "https://files.pythonhosted.org/packages/aa/f3/4a82944b6bb7c83af4b700a2c33cbf5dbf0c09b96f56d3e939f3e69ffb84/asyncio_throttler-0.1.0-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "62bb5db485c86d519db992f5f08070a8", "sha256": "64cb52715fdd91eab32ad5db7cb32dae62b1a20ad4af8d4abdf402cba42a4325" }, "downloads": -1, "filename": "asyncio_throttler-0.1.0.tar.gz", "has_sig": false, "md5_digest": "62bb5db485c86d519db992f5f08070a8", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 5896, "upload_time": "2016-01-27T21:28:00", "url": "https://files.pythonhosted.org/packages/4f/a1/be96073700512b4eda0cd9faf2047a36d9f0e859c97bc55209ac60db5d57/asyncio_throttler-0.1.0.tar.gz" } ] }