{ "info": { "author": "Vi Engineering", "author_email": "voiceai-eng@dialpad.com", "bugtrack_url": null, "classifiers": [ "Development Status :: 4 - Beta", "Intended Audience :: Developers", "License :: OSI Approved :: MIT License", "Operating System :: OS Independent", "Programming Language :: Python :: 3", "Programming Language :: Python :: 3.6", "Programming Language :: Python :: 3.7", "Topic :: Internet" ], "description": "Asyncio Python Client for Google Cloud Pub/Sub\n==============================================\n\n|pypi| |pythons-aio|\n\nInstallation\n------------\n\n.. code-block:: console\n\n $ pip install --upgrade gcloud-aio-pubsub\n\nUsage\n-----\n\nThis Pub/Sub implementation is based on ``google-cloud-pubsub >= 0.29.4``\n\nCurrently we have only implemented an asyncio version of ``SubscriberClient``\nas the subscription pattern does not work with asyncio by default. The official\nGoogle publisher returns a future which is mostly useable as-is. We've not yet\nseen a need to build a non-asyncio threadsafe version of the library -- the\nupstream Google libraries have this well-handled.\n\nHere's the rough usage pattern for subscribing:\n\n.. code-block:: python\n\n from gcloud.aio.pubsub import SubscriberClient\n from google.cloud.pubsub_v1.subscriber.message import Message\n\n client = SubscriberClient()\n # create subscription if it doesn't already exist\n client.create_subscription('subscription_name', 'topic_name')\n\n async def message_callback(message: Message) -> None:\n try:\n # just an example: process the message however you need to here...\n result = handle(message)\n await upload_result(result)\n except Exception:\n message.nack()\n else:\n message.ack()\n\n # subscribe to the subscription, receiving a Future that acts as a keepalive\n keep_alive = client.subscribe('subscription_name', message_callback)\n\n # have the client run forever, pulling messages from this subscription,\n # passing them to the specified callback function, and wrapping it in an\n # asyncio task.\n client.run_forever(keep_alive)\n\nConfiguration\n-------------\n\nOur create_subscription method is a thing wrapper and thus supports all keyword\nconfiguration arguments from the official pubsub client which you can find in\nthe `official Google documentation`_.\n\nWhen subscribing to a subscription you can optionally pass in a ``FlowControl``\nand/or ``Scheduler`` instance.\n\n.. code-block:: python\n\n example_flow_control = FlowControl(\n max_messages=1,\n resume_threshold=0.8,\n max_request_batch_size=1,\n max_request_batch_latency=0.1,\n max_lease_duration=10,\n )\n\n keep_alive = client.subscribe(\n 'subscription_name',\n message_callback,\n flow_control=example_flow_control\n )\n\nUnderstanding how modifying ``FlowControl`` affects how your pubsub runtime\nwill operate can be confusing so here's a handy dandy guide!\n\nWelcome to @TheKevJames's guide to configuring Google Pubsub Subscription\npolicies! Settle in, grab a drink, and stay a while.\n\nThe Subscriber is controlled by a FlowControl configuration tuple defined\n`here `_:\nthat configuration object ``f`` gets used by the Subscriber in the following\nways:\n\nMax Concurrency\n~~~~~~~~~~~~~~~\n\nThe subscriber is allowed to lease new tasks whenever its currently leased\ntasks ``x`` satisfy:\n\n.. code-block:: python\n\n (\n (len(x) < f.resume_threshold * f.max_messages)\n and (sum(x.bytes) < f.resume_threshold * f.max_bytes)\n )\n\nIn practice, this means we should set these values with the following\nrestrictions:\n\n- the maximum number of concurrently leased tasks at peak is:\n ``= (f.max_messages * f.resume_threshold) + f.max_request_batch_size``\n- the maximum memory usage of our leased tasks at peak is:\n ``= (f.max_bytes * f.resume_threshold) + (f.max_request_batch_size *\n bytes_per_task)``\n- these values are constrain each other, ie. we limit ourselves to the lesser\n of these values given:\n ``max_tasks * bytes_per_task <> max_memory``\n\nAside: it seems like OCNs on Pubsub are ~1538 bytes each\n\nLeasing Requests\n~~~~~~~~~~~~~~~~\n\nWhen leasing new tasks, the ``Subscriber`` uses the following algorithm:\n\n.. code-block:: python\n\n def lease_more_tasks():\n start = time.now()\n yield queue.Queue.get(block=True) # always returns >=1\n\n for _ in range(f.max_request_batch_size - 1):\n elapsed = time.now() - start\n yield queue.Queue.get(\n block=False,\n timeout=f.max_request_batch_latency-elapsed)\n if elapsed >= f.max_request_batch_latency:\n break\n\nIn practice, this means we should set ``f.max_request_batch_size`` given the\nabove concurrent concerns and set ``f.max_request_batch_latency`` given\nwhatever latency ratio we are willing to accept.\n\nThe expected best-case time for ``Queue.get()`` off a full queue is no worse\nthan 0.3ms. This Queue should be filling up as fast as grpc can make requests\nto Google Pubsub, which should be Fast Enough(tm) to keep it filled, given\n*those* requests are batched.\n\nTherefore, we can expect:\n\n- avg_lease_latency: ``~= f.max_request_batch_size * 0.0003``\n- worst_case_latency: ``~= f.max_request_batch_latency``\n\nNote that leasing occurs based on ``f.resume_threshold``, so some of this\nlatency is concurrent with task execution.\n\nTask Expiry\n~~~~~~~~~~~\n\nAny task which has not been acked or nacked counts against the current leased\ntask count. Our worker thread should ensure all tasks are acked or nacked, but\nthe ``FlowControl`` config allows us to handle any other cases. Note that\nleasing works as follows:\n\n- When a subscriber leases a task, Google Pubsub will not re-lease that\n task until ``subscription.ack_deadline_seconds = 10`` (configurable\n per-subscription) seconds have passed.\n- If a client calls ``ack()`` on a task, it is immediately removed from Google\n Pubsub.\n- If a client calls ``nack()`` on a task, it immediately allows Google Pubsub\n to re-lease that task to a new client. The client drops the task from its\n memory.\n- If ``f.max_lease_duration`` passes between a message being leased and acked,\n the client will send a ``nack`` (see above workflow). It will NOT drop the\n task from its memory -- eg. the ``worker(task)`` process may still be run.\n\nNotes:\n\n- all steps are best-effort, eg. read \"a task will be deleted\" as \"a task will\n probably get deleted, if the distributed-system luck is with you\"\n- in the above workflow \"Google Pubsub\" refers to the server-side system, eg.\n managed by Google where the tasks are actually stored.\n\nIn practice, we should thus set ``f.max_lease_duration`` to no lower than\nour 95% percentile task latency at high load. The lower this value is,\nthe better our throughput will be in extreme cases.\n\nConfusion\n~~~~~~~~~\n\n``f.max_requests`` is defined, but seems to be unused.\n\nContributing\n------------\n\nPlease see our `contributing guide`_.\n\n.. _contributing guide: https://github.com/talkiq/gcloud-aio/blob/master/.github/CONTRIBUTING.rst\n.. _official Google documentation: https://github.com/googleapis/google-cloud-python/blob/11c72ade8b282ae1917fba19e7f4e0fe7176d12b/pubsub/google/cloud/pubsub_v1/gapic/subscriber_client.py#L236\n\n.. |pypi| image:: https://img.shields.io/pypi/v/gcloud-aio-pubsub.svg?style=flat-square\n :alt: Latest PyPI Version\n :target: https://pypi.org/project/gcloud-aio-pubsub/\n\n.. |pythons-aio| image:: https://img.shields.io/pypi/pyversions/gcloud-aio-pubsub.svg?style=flat-square\n :alt: Python Version Support\n :target: https://pypi.org/project/gcloud-aio-pubsub/\n\n\n", "description_content_type": "", "docs_url": null, "download_url": "", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "https://github.com/talkiq/gcloud-aio", "keywords": "", "license": "MIT License", "maintainer": "", "maintainer_email": "", "name": "gcloud-aio-pubsub", "package_url": "https://pypi.org/project/gcloud-aio-pubsub/", "platform": "Posix; MacOS X; Windows", "project_url": "https://pypi.org/project/gcloud-aio-pubsub/", "project_urls": { "Homepage": "https://github.com/talkiq/gcloud-aio" }, "release_url": "https://pypi.org/project/gcloud-aio-pubsub/1.1.0/", "requires_dist": [ "google-cloud-pubsub (>=0.29.4)" ], "requires_python": ">= 3.6", "summary": "Asyncio Python Client for Google Cloud Pub/Sub", "version": "1.1.0" }, "last_serial": 5993316, "releases": { "0.5.0": [ { "comment_text": "", "digests": { "md5": "c54f97eaddc725c8e7bfa31c832b623c", "sha256": "aaae966ea8704f422fb4258ae6436957332c94997f1245efe5d464315f6d11d3" }, "downloads": -1, "filename": "gcloud_aio_pubsub-0.5.0-py2.py3-none-any.whl", "has_sig": false, "md5_digest": "c54f97eaddc725c8e7bfa31c832b623c", "packagetype": "bdist_wheel", "python_version": "py2.py3", "requires_python": null, "size": 8910, "upload_time": "2017-11-14T01:44:46", "url": "https://files.pythonhosted.org/packages/04/3b/0c6a0e03db675474067c3d1013fceeb3d46ae530727acf647f48500e7b3e/gcloud_aio_pubsub-0.5.0-py2.py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "9b1303bb7d45f110e673cde13c91af79", "sha256": "688b34875cd6d35932953ca4bf11cdf94d56bfa294ac7655d99af95aeabe8878" }, "downloads": -1, "filename": "gcloud-aio-pubsub-0.5.0.tar.gz", "has_sig": false, "md5_digest": "9b1303bb7d45f110e673cde13c91af79", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 4378, "upload_time": "2017-11-14T01:44:47", "url": "https://files.pythonhosted.org/packages/07/f9/34bde3aba75b8581583477e1279438580aa0ceb659cd2ffebf381a66f6ef/gcloud-aio-pubsub-0.5.0.tar.gz" } ], "0.5.1": [ { "comment_text": "", "digests": { "md5": "c380bfad2fbc352854f86104df64b55f", "sha256": "fa344c40a13c38e08139299517974e9b5b6b0c8b574461e5eb9a7f461fcc5887" }, "downloads": -1, "filename": "gcloud_aio_pubsub-0.5.1-py2.py3-none-any.whl", "has_sig": false, "md5_digest": "c380bfad2fbc352854f86104df64b55f", "packagetype": "bdist_wheel", "python_version": "py2.py3", "requires_python": null, "size": 8967, "upload_time": "2018-03-22T23:42:52", "url": "https://files.pythonhosted.org/packages/e0/98/d12eaadad5fac2f1871e18d30e7c17f881ab254f9752f17f14e638ee1136/gcloud_aio_pubsub-0.5.1-py2.py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "51cd71d87da12ac1c8c5cebcc60f7e4f", "sha256": "b80776bba1d75a6b34740c756ce8a5e392c4d074fb8af88e58290add43e86e77" }, "downloads": -1, "filename": "gcloud-aio-pubsub-0.5.1.tar.gz", "has_sig": false, "md5_digest": "51cd71d87da12ac1c8c5cebcc60f7e4f", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 5127, "upload_time": "2018-03-22T23:42:53", "url": "https://files.pythonhosted.org/packages/4d/a5/d65b3e3a28ba85895c29e0dc14981dd8513d80f2f91c353a4b13c7f179be/gcloud-aio-pubsub-0.5.1.tar.gz" } ], "1.0.0": [ { "comment_text": "", "digests": { "md5": "07a2741e4eff5097bad7de8b19a4ef92", "sha256": "8e7c0219a56708d0fe90f13a8babd132bfc85f83ed01a47a83fdf2d9ec21c29c" }, "downloads": -1, "filename": "gcloud_aio_pubsub-1.0.0-py2.py3-none-any.whl", "has_sig": false, "md5_digest": "07a2741e4eff5097bad7de8b19a4ef92", "packagetype": "bdist_wheel", "python_version": "py2.py3", "requires_python": null, "size": 10021, "upload_time": "2019-06-21T00:31:58", "url": "https://files.pythonhosted.org/packages/36/71/e2b64a074203376d48eabaa8b63df18c5eb477bd973d9b6a6afa0d0b4dbf/gcloud_aio_pubsub-1.0.0-py2.py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "243f26cfa65a16c33be36a30e930f64a", "sha256": "2255a9e56ba9f1d0c1fe27b080910a58998fe19c77e8834b9dc05102b07fae09" }, "downloads": -1, "filename": "gcloud-aio-pubsub-1.0.0.tar.gz", "has_sig": false, "md5_digest": "243f26cfa65a16c33be36a30e930f64a", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 8270, "upload_time": "2019-06-21T00:32:00", "url": "https://files.pythonhosted.org/packages/b9/c4/951eddc44f597a93c1159a757b462781ca23dc27bda56fe2c908d8e480d8/gcloud-aio-pubsub-1.0.0.tar.gz" } ], "1.0.1": [ { "comment_text": "", "digests": { "md5": "c84735e74c446452d81e97f5947441bb", "sha256": "25b5ee48dbdc7467980edaa311c9ca3385c28ff7a608122ad898e64de8d07e91" }, "downloads": -1, "filename": "gcloud_aio_pubsub-1.0.1-py2.py3-none-any.whl", "has_sig": false, "md5_digest": "c84735e74c446452d81e97f5947441bb", "packagetype": "bdist_wheel", "python_version": "py2.py3", "requires_python": null, "size": 10020, "upload_time": "2019-07-17T20:32:12", "url": "https://files.pythonhosted.org/packages/89/66/fdd780ee972733ae1cc7591301bb5917e0e07e9322e7527b58a8f3c70d5a/gcloud_aio_pubsub-1.0.1-py2.py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "2452a740595aaf66f93e4cb253b47c68", "sha256": "99a83ca5cf987736ef86702640eb83b31dd04af6422e1bbee2639390872ffcfa" }, "downloads": -1, "filename": "gcloud-aio-pubsub-1.0.1.tar.gz", "has_sig": false, "md5_digest": "2452a740595aaf66f93e4cb253b47c68", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 8213, "upload_time": "2019-07-17T20:32:15", "url": "https://files.pythonhosted.org/packages/02/de/af22e1aac8056927c3508500d314f331442c7d8b49e91412a485922d3da0/gcloud-aio-pubsub-1.0.1.tar.gz" } ], "1.0.2": [ { "comment_text": "", "digests": { "md5": "df5894e8f4684512f7fa4d737a2d5079", "sha256": "3dc7993177d58fe894b039ad353dba06d8903242f20ec023c52c9fa199a7fe9f" }, "downloads": -1, "filename": "gcloud_aio_pubsub-1.0.2-py2.py3-none-any.whl", "has_sig": false, "md5_digest": "df5894e8f4684512f7fa4d737a2d5079", "packagetype": "bdist_wheel", "python_version": "py2.py3", "requires_python": ">= 3.6", "size": 10035, "upload_time": "2019-10-15T02:50:41", "url": "https://files.pythonhosted.org/packages/81/f0/efc15e03b5da8d4ac20a55efef7645634ffabb989b607e9ac9f8659f064d/gcloud_aio_pubsub-1.0.2-py2.py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "976ca712923ecbde3a371199894dcc8c", "sha256": "346f69d8bdc8e84af26bd63e995cde1eb00ce87f56aea7cde9175e0b36976d72" }, "downloads": -1, "filename": "gcloud-aio-pubsub-1.0.2.tar.gz", "has_sig": false, "md5_digest": "976ca712923ecbde3a371199894dcc8c", "packagetype": "sdist", "python_version": "source", "requires_python": ">= 3.6", "size": 8234, "upload_time": "2019-10-15T02:50:43", "url": "https://files.pythonhosted.org/packages/61/d8/350591fffef3cbd2ae39b81fcee64212e8a2967130359ad8e99ab0666d21/gcloud-aio-pubsub-1.0.2.tar.gz" } ], "1.1.0": [ { "comment_text": "", "digests": { "md5": "e0236c9d121ae0ba5e95c99c08c4154d", "sha256": "eabb27d5be6807ba711808bb923d9e428d71c119a0e51ee1d01480e9fa533d53" }, "downloads": -1, "filename": "gcloud_aio_pubsub-1.1.0-py2.py3-none-any.whl", "has_sig": false, "md5_digest": "e0236c9d121ae0ba5e95c99c08c4154d", "packagetype": "bdist_wheel", "python_version": "py2.py3", "requires_python": ">= 3.6", "size": 10116, "upload_time": "2019-10-18T02:00:08", "url": "https://files.pythonhosted.org/packages/62/4f/0d4a0b38c8f9d1dee366932dd575c9c2a11020fcf9179f93045270450f25/gcloud_aio_pubsub-1.1.0-py2.py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "f35b4a3c88299de0c00ae4bfa9725694", "sha256": "7a2b13ad4c1f1844f17d9c8adc69e3b2385ab485cf7d94ae810cb417b498fe5b" }, "downloads": -1, "filename": "gcloud-aio-pubsub-1.1.0.tar.gz", "has_sig": false, "md5_digest": "f35b4a3c88299de0c00ae4bfa9725694", "packagetype": "sdist", "python_version": "source", "requires_python": ">= 3.6", "size": 8319, "upload_time": "2019-10-18T02:00:09", "url": "https://files.pythonhosted.org/packages/9e/27/bb33872f640ebcadd032d21ab5ca645c38ac78a5fdbd8206da8a177018db/gcloud-aio-pubsub-1.1.0.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "e0236c9d121ae0ba5e95c99c08c4154d", "sha256": "eabb27d5be6807ba711808bb923d9e428d71c119a0e51ee1d01480e9fa533d53" }, "downloads": -1, "filename": "gcloud_aio_pubsub-1.1.0-py2.py3-none-any.whl", "has_sig": false, "md5_digest": "e0236c9d121ae0ba5e95c99c08c4154d", "packagetype": "bdist_wheel", "python_version": "py2.py3", "requires_python": ">= 3.6", "size": 10116, "upload_time": "2019-10-18T02:00:08", "url": "https://files.pythonhosted.org/packages/62/4f/0d4a0b38c8f9d1dee366932dd575c9c2a11020fcf9179f93045270450f25/gcloud_aio_pubsub-1.1.0-py2.py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "f35b4a3c88299de0c00ae4bfa9725694", "sha256": "7a2b13ad4c1f1844f17d9c8adc69e3b2385ab485cf7d94ae810cb417b498fe5b" }, "downloads": -1, "filename": "gcloud-aio-pubsub-1.1.0.tar.gz", "has_sig": false, "md5_digest": "f35b4a3c88299de0c00ae4bfa9725694", "packagetype": "sdist", "python_version": "source", "requires_python": ">= 3.6", "size": 8319, "upload_time": "2019-10-18T02:00:09", "url": "https://files.pythonhosted.org/packages/9e/27/bb33872f640ebcadd032d21ab5ca645c38ac78a5fdbd8206da8a177018db/gcloud-aio-pubsub-1.1.0.tar.gz" } ] }