{
    "info": {
        "author": "Bradley  Atkins",
        "author_email": "bradley.atkinz@gmail.com",
        "bugtrack_url": null,
        "classifiers": [
            "License :: OSI Approved :: MIT License",
            "Programming Language :: Python :: 3",
            "Programming Language :: Python :: 3.7"
        ],
        "description": "# API Rate Limiter\n> A simple Python object that allows you to implement client side API \nrate limiting.\n\n---\n\n## Introduction\n\nThis project was created to address a specific issue \u2013 API rate limiting\n when scanning an AWS platform.\n\nThe problem with AWS applying rate limiting to the services is that it \nis indiscriminate in nature. So if you are running a multi-threaded \nscanner such as ScoutSuite, while for example an Auto Scaling Group is \ntrying to scale something, the auto Scaling can fail because AWS does \nnot distinguish between the third party scanner and its own services.\n\nThis is a cause for frustration amongst the platform support team and \ncan also cause an outage, particularly in the early stages of \ndevelopment of a new service, where resilience and high availability are\nstill but a twinkle in the architect's eye...\n\nThis is becoming more and more of an issue as consumers realise the \nnecessity of scanning their cloud platforms.\n\n## NOTE!\n\nApologies for the lack of images if you are viewing this on PyPi. For \nsome reason PyPi insist on changing the URLs to \nhttps://warehouse-test-camo.cmh1.psfhosted.org/..... I've no idea why.\nYou can read this README correctly on the home project on GitHub.\n\n## History\n\nI started looking into this issue when scanning my own client's platform\n and hit the rate limiting issue when scanning some 15k of snapshots.\n\nInitially I managed to mitigate this using the Network Link Conditioner \non the MAC I was using to run the scanner. \n\nThe NLC allows you to add latency to all outbound packets on a given \nnetwork like so:\n\n![](images/nlc.png)\n\n![](images/nlc-profile.png)\n\nThis blanket approach to delaying all outbound packets did resolve the \nissue of the rate limiting, but at a cost of causing the scanner to now\nrun for two hours in order to complete a scan of all of our development \nenvironments.\n\nNot only was this slow, but it's also not OS agnostic, so hardly a good \nsolution.\n\nIt occurred to me that if a mechanism could be created within boto3 \nitself to queue outbound API calls at a configurable rate, then this \nmight might prove to be a more general solution to the issue.\n\nSo I forked botocore [here](https://github.com/museadmin/botorate) into \na project that combined forks of botocore, boto3 and ScoutSuite.\n\nThis project has the queue implemented in it and boto3 has been \nrefactored to pass through the value of the API rate in ms to botocore. \nScoutSuite is included and has simply been hardcoded to apply the queue \nto ec2 clients.\n\nThis has enabled me to conduct comparative runs between this solution \nand using the Network Link Conditioner. With the former completing in 30\n minutes and the latter in around 2 hours.\n\nThe queue was only applied to ec2 clients because I was only \nexperiencing rate limiting on the scanning of the snapshots, around 15k \n+\n\nAt the time of writing botocore seem to be unwilling to accept the PR \nfor this as they feel that it is beyond the scope of their project. \n\nc'est la vie.\n\n## The Solution\n\nRather than cry over a missed opportunity I've now taken the rate \nlimiter and packaged it up as a stand alone utility that anyone can \nconsume in their own projects should they need to avoid server-side rate \nlimiting:   \n\n![](images/api-rate-overview.jpg)\n\nIn the diagram above, each thread needing to make an API call using \na/the ec2 client calls a method that first enqueues the call in a FIFO \nqueue and then waits for it to reach the head of the queue. Thereby \ntranslating the asynchronous calls from the multiple threads into a \nsynchronous stream of calls at a configurable frequency.\n\nThis approach allows each thread to continue to leverage parallel \nprocessing of tasks while only waiting on the actual API call. So you \nstill see the increase in efficiency of the multi threaded approach.\n\nBy instantiating individual queues for each AWS service, each can be \nindividually configured with an appropriate rate for the consumer's \nplatform, or not rate limited at all.\n\ne.g. in my case I only had to limit the EC2 client because of the \nexcessive number of snapshots being scanned. \n\n## Installation\n\nIn the usual Python fashion:\n\n```sh\nimport ApiRateLimiter\n```\n\n## Usage example\n\nSee \"The Solution\" above or look at the tests in the GitHub repo:\n\n[Tests on GitHub](https://github.com/museadmin/api-rate-limiter/tree/master/api-rate-limiter/tests)\n\n### Basic Usage\n\nSetup:\n\n* Instantiate the rate limiter\n* Start it running in a background thread\n* Call the enqueue() method to join the queue\n* Poll the waiting state until false\n* Make your API call\n\nOn close:\n\n* Soft stop the rate limiter \u2013 Waits for background thread to exit or timeout\n\n#### Example\n```\n    rate_limiter = ApiRateLimiter(100)\n    rate_limiter.start()\n\n    ...\n\n    def some_method()\n        waiter = self.rate_limiter.enqueue()\n        while waiter.waiting is True:\n            pass\n\n        client.describe_instances()\n\n    ...\n\n    rate_limiter.stop(True) \n\n\n```\n\n## Release History\n\n* 0.1.0\n    * CHANGE: Initial code commit\n    * ADD LICENCE\n    * ADD Detailed README\n    * FIX Error handling in integration test\n\n* 0.1.1\n    * FIX Remove requires for multiprocessing as now in Python3 main lib\n## Meta\n\nBradley Atkins \u2013 bradley.atkinz @ gmail.com\n\nDistributed under the MIT license. See ``LICENSE`` for more information.\n\n[This Project on GitHub](https://github.com/museadmin/api-rate-limiter)\n\n## Contributing\n\n1. Fork it (<https://github.com/museadmin/api-rate-limiter>)\n2. Create your feature branch (`git checkout -b feature/fooBar`)\n3. Commit your changes (`git commit -am 'Add some fooBar'`)\n4. Push to the branch (`git push origin feature/fooBar`)\n5. Create a new Pull Request\n6. Email me if I don't notice!\n\n\n",
        "description_content_type": "text/markdown",
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://github.com/museadmin/api-rate-limiter",
        "keywords": "",
        "license": "MIT",
        "maintainer": "",
        "maintainer_email": "",
        "name": "api-rate-limiter",
        "package_url": "https://pypi.org/project/api-rate-limiter/",
        "platform": "",
        "project_url": "https://pypi.org/project/api-rate-limiter/",
        "project_urls": {
            "Homepage": "https://github.com/museadmin/api-rate-limiter"
        },
        "release_url": "https://pypi.org/project/api-rate-limiter/0.1.1/",
        "requires_dist": [
            "queue",
            "time",
            "threading"
        ],
        "requires_python": "",
        "summary": "Implement client side API rate limiting",
        "version": "0.1.1"
    },
    "last_serial": 5814745,
    "releases": {
        "0.1.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "c2686f40bc7878da3933717d97579ed8",
                    "sha256": "bd3bfdcc62d73823ae4c2c8fe0bbc19bfd79e4c21aaff05783dba9475251e5ba"
                },
                "downloads": -1,
                "filename": "api_rate_limiter-0.1.0-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "c2686f40bc7878da3933717d97579ed8",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 6751,
                "upload_time": "2019-09-11T12:36:24",
                "url": "https://files.pythonhosted.org/packages/39/23/a76ed918b187e938a5ef66f5497ef989ce3d4d25664522dc0ff2e0719e30/api_rate_limiter-0.1.0-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "8d3d9d7ff3a3a6404a61eb479e0c2458",
                    "sha256": "b6801bfa6b6f6b8e8115c7dc37b09c3f9e49531970f4bc031188d9740ed0f426"
                },
                "downloads": -1,
                "filename": "api-rate-limiter-0.1.0.tar.gz",
                "has_sig": false,
                "md5_digest": "8d3d9d7ff3a3a6404a61eb479e0c2458",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 5980,
                "upload_time": "2019-09-11T12:36:27",
                "url": "https://files.pythonhosted.org/packages/8f/24/43cca1103b71cf1c2746852081a8902ce72359498d2ea0905cb9f3b1cd3f/api-rate-limiter-0.1.0.tar.gz"
            }
        ],
        "0.1.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "86e6ec03992fbec17366082c0da6d4bf",
                    "sha256": "f152e455e6e82d2ee81d0f1a5cfcc16fdeb30aba3c057b17e32b708c650ef92f"
                },
                "downloads": -1,
                "filename": "api_rate_limiter-0.1.1-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "86e6ec03992fbec17366082c0da6d4bf",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 6781,
                "upload_time": "2019-09-11T12:43:55",
                "url": "https://files.pythonhosted.org/packages/0a/09/3215a017dd6dc5631a58cc2076fbad0037c8c8ce71a56cab13a45636791e/api_rate_limiter-0.1.1-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "b8eb35f3932f6738f9fb107ae2099ea1",
                    "sha256": "6ba8642031d0d221482b498e409c6f78e0ddd54afacffbd6000e323c354e5d31"
                },
                "downloads": -1,
                "filename": "api-rate-limiter-0.1.1.tar.gz",
                "has_sig": false,
                "md5_digest": "b8eb35f3932f6738f9fb107ae2099ea1",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 6021,
                "upload_time": "2019-09-11T12:43:56",
                "url": "https://files.pythonhosted.org/packages/de/92/22a729a125e83e227fbb69fc6a9a012d9facb66ec0084a9e9b8b5a97124f/api-rate-limiter-0.1.1.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "86e6ec03992fbec17366082c0da6d4bf",
                "sha256": "f152e455e6e82d2ee81d0f1a5cfcc16fdeb30aba3c057b17e32b708c650ef92f"
            },
            "downloads": -1,
            "filename": "api_rate_limiter-0.1.1-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "86e6ec03992fbec17366082c0da6d4bf",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": null,
            "size": 6781,
            "upload_time": "2019-09-11T12:43:55",
            "url": "https://files.pythonhosted.org/packages/0a/09/3215a017dd6dc5631a58cc2076fbad0037c8c8ce71a56cab13a45636791e/api_rate_limiter-0.1.1-py3-none-any.whl"
        },
        {
            "comment_text": "",
            "digests": {
                "md5": "b8eb35f3932f6738f9fb107ae2099ea1",
                "sha256": "6ba8642031d0d221482b498e409c6f78e0ddd54afacffbd6000e323c354e5d31"
            },
            "downloads": -1,
            "filename": "api-rate-limiter-0.1.1.tar.gz",
            "has_sig": false,
            "md5_digest": "b8eb35f3932f6738f9fb107ae2099ea1",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 6021,
            "upload_time": "2019-09-11T12:43:56",
            "url": "https://files.pythonhosted.org/packages/de/92/22a729a125e83e227fbb69fc6a9a012d9facb66ec0084a9e9b8b5a97124f/api-rate-limiter-0.1.1.tar.gz"
        }
    ]
}