{
    "info": {
        "author": "Panagiotis Garefalakis",
        "author_email": "pangaref@gmail.com",
        "bugtrack_url": null,
        "classifiers": [
            "Development Status :: 4 - Beta",
            "Environment :: Web Environment",
            "Intended Audience :: Developers",
            "License :: OSI Approved :: MIT License",
            "Natural Language :: English",
            "Operating System :: OS Independent",
            "Programming Language :: Python",
            "Programming Language :: Python :: 2",
            "Programming Language :: Python :: 3",
            "Topic :: Internet :: WWW/HTTP",
            "Topic :: Software Development :: Libraries :: Python Modules"
        ],
        "description": "HTTP Request Randomizer |Build Status| |Coverage Status| |Dependency Status| |PyPI version|\n===========================================================================================\n\n`Vietnamese version <README-vi.md>`__\n\nA convenient way to implement HTTP requests is using Pythons'\n**requests** library. One of requests\u2019 most popular features is simple\nproxying support. HTTP as a protocol has very well-defined semantics for\ndealing with proxies, and this contributed to the widespread deployment\nof HTTP proxies\n\nProxying is very useful when conducting intensive web crawling/scrapping\nor when you just want to hide your identity (anonymization).\n\nIn this project I am using public proxies to randomise http requests\nover a number of IP addresses and using a variety of known user agent\nheaders these requests look to have been produced by different\napplications and operating systems.\n\nProxies\n-------\n\nProxies provide a way to use server P (the middleman) to contact server\nA and then route the response back to you. In more nefarious circles,\nit's a prime way to make your presence unknown and pose as many clients\nto a website instead of just one client. Often times websites will block\nIPs that make too many requests, and proxies is a way to get around\nthis. But even for simulating an attack, you should know how it's done.\n\nUser Agent\n----------\n\nSurprisingly, the only thing that tells a server the application\ntriggered the request (like browser type or from a script) is a header\ncalled a \"user agent\" which is included in the HTTP request.\n\nThe source code\n---------------\n\nThe project code in this repository is crawling **four** different\npublic proxy websites: \\* http://proxyfor.eu/geo.php \\*\nhttp://free-proxy-list.net \\* http://rebro.weebly.com/proxy-list.html \\*\nhttp://www.samair.ru/proxy/time-01.htm\n\nAfter collecting the proxy data and filtering the slowest ones it is\nrandomly selecting one of them to query the target url. The request\ntimeout is configured at 30 seconds and if the proxy fails to return a\nresponse it is deleted from the application proxy list. I have to\nmention that for each request a different agent header is used. The\ndifferent headers are stored in the **/data/user\\_agents.txt** file\nwhich contains around 900 different agents.\n\nInstallation\n------------\n\nIf you wish to use this module as a `CLI\ntool <#command-line-interface>`__, install it globally via pip:\n\n::\n\n      pip install http-request-randomizer\n\nOtherwise, you can clone the repository and use setup tools:\n\n::\n\n    python setup.py install\n\nHow to use\n----------\n\n-  `Command-line interface <#command-line-interface>`__\n-  `Library API <#api>`__\n\nCommand-line interface\n----------------------\n\nAssuming that you have **http-request-randomizer** installed, you can\nuse the commands below:\n\nshow help message:\n\n::\n\n    proxyList   -h, --help\n\nspecify proxy provider(s) (required):\n\n::\n\n      -s {proxyforeu,rebro,samair,freeproxy,all} \n\nSpecify output stream (default: sys.stdout), could also be a file:\n\n::\n\n      -o, --outfile\n\nspecify provider timeout threshold in seconds:\n\n::\n\n      -t, --timeout\n\nspecify proxy bandwidth threshold in KBs:\n\n::\n\n      -bw, --bandwidth\n\nshow program's version number:\n\n::\n\n      -v, --version\n\nAPI\n---\n\nTo use **http-request-randomizer** as a library, include it in your\nrequirements.txt file. Then you can simply generate a proxied request\nusing a method call:\n\n.. code:: python\n\n    import time\n    from http_request_randomizer.requests.proxy.requestProxy import RequestProxy\n\n    if __name__ == '__main__':\n\n        start = time.time()\n        req_proxy = RequestProxy()\n        print(\"Initialization took: {0} sec\".format((time.time() - start)))\n        print(\"Size: {0}\".format(len(req_proxy.get_proxy_list())))\n        print(\"ALL = {0} \".format(list(map(lambda x: x.get_address(), req_proxy.get_proxy_list()))))\n\n        test_url = 'http://ipv4.icanhazip.com'\n\n        while True:\n            start = time.time()\n            request = req_proxy.generate_proxied_request(test_url)\n            print(\"Proxied Request Took: {0} sec => Status: {1}\".format((time.time() - start), request.__str__()))\n            if request is not None:\n                print(\"\\t Response: ip={0}\".format(u''.join(request.text).encode('utf-8')))\n            print(\"Proxy List Size: {0}\".format(len(req_proxy.get_proxy_list())))\n\n            print(\"-> Going to sleep..\")\n            time.sleep(10)\n\nDocumentation\n-------------\n\n`http-request-randomizer\ndocumentation <http://pythonhosted.org/http-request-randomizer>`__\n\nContributing\n------------\n\nMany thanks to the open-source community for\n`contributing <https://github.com/pgaref/HTTP_Request_Randomizer/blob/master/CONTRIBUTORS.md>`__\nto this project!\n\nFaced an issue?\n---------------\n\nOpen an issue\n`here <https://github.com/pgaref/HTTP_Request_Randomizer/issues>`__, and\nbe as detailed as possible :)\n\nFeels like a feature is missing?\n--------------------------------\n\nFeel free to open a ticket! PRs are always welcome!\n\nLicense\n-------\n\nThis project is licensed under the terms of the MIT license.\n\n.. |Build Status| image:: https://travis-ci.org/pgaref/HTTP_Request_Randomizer.svg?branch=master\n   :target: https://travis-ci.org/pgaref/HTTP_Request_Randomizer\n.. |Coverage Status| image:: https://coveralls.io/repos/github/pgaref/HTTP_Request_Randomizer/badge.svg?branch=master\n   :target: https://coveralls.io/github/pgaref/HTTP_Request_Randomizer?branch=master\n.. |Dependency Status| image:: https://requires.io/github/pgaref/HTTP_Request_Randomizer/requirements.svg?branch=la55u-master\n   :target: https://requires.io/github/pgaref/HTTP_Request_Randomizer/requirements/?branch=la55u-master\n.. |PyPI version| image:: https://badge.fury.io/py/http-request-randomizer.svg\n   :target: https://badge.fury.io/py/http-request-randomizer",
        "description_content_type": "",
        "docs_url": "https://pythonhosted.org/http-request-randomizer/",
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "http://pgaref.com/blog/python-proxy",
        "keywords": "",
        "license": "MIT",
        "maintainer": "",
        "maintainer_email": "",
        "name": "http-request-randomizer",
        "package_url": "https://pypi.org/project/http-request-randomizer/",
        "platform": "any",
        "project_url": "https://pypi.org/project/http-request-randomizer/",
        "project_urls": {
            "Homepage": "http://pgaref.com/blog/python-proxy"
        },
        "release_url": "https://pypi.org/project/http-request-randomizer/1.2.3/",
        "requires_dist": null,
        "requires_python": "",
        "summary": "A package using public proxies to randomise http requests.",
        "version": "1.2.3"
    },
    "last_serial": 4031449,
    "releases": {
        "0.0.3": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "a62ed73e17742182ea1497fe9bd22c81",
                    "sha256": "d855cd1a3520ea5273120d3fa488dd8b66f9233bef8aae0bac896b556d5bbac2"
                },
                "downloads": -1,
                "filename": "http-request-randomizer-0.0.3.tar.gz",
                "has_sig": false,
                "md5_digest": "a62ed73e17742182ea1497fe9bd22c81",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 4822,
                "upload_time": "2016-09-14T19:07:26",
                "url": "https://files.pythonhosted.org/packages/e8/e7/283df13da2d0b8b9d800db6dd6096f0e50df10e4636ceed41b11b3b268e6/http-request-randomizer-0.0.3.tar.gz"
            }
        ],
        "0.0.5": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "973e78bf706502dfcc03502fd5d878a8",
                    "sha256": "e5ec6245b20ee289db682e00305740464cd7fbc851e04257560964c542b74b7f"
                },
                "downloads": -1,
                "filename": "HTTP_Request_Randomizer-0.0.5-py2-none-any.whl",
                "has_sig": false,
                "md5_digest": "973e78bf706502dfcc03502fd5d878a8",
                "packagetype": "bdist_wheel",
                "python_version": "py2",
                "requires_python": null,
                "size": 26509,
                "upload_time": "2016-10-31T23:16:14",
                "url": "https://files.pythonhosted.org/packages/28/aa/d0317af4d141dfc8bee7fde49ee58aab0966279ac12c426b9c7e4f8793fa/HTTP_Request_Randomizer-0.0.5-py2-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "df28d2d0274b2d7d90d9e76ff76f948a",
                    "sha256": "316af87928ac9e0d7965ac7b26a0ff3396205169101444aea3892bd3d76baeec"
                },
                "downloads": -1,
                "filename": "HTTP_Request_Randomizer-0.0.5-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "df28d2d0274b2d7d90d9e76ff76f948a",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 26512,
                "upload_time": "2016-10-31T23:15:44",
                "url": "https://files.pythonhosted.org/packages/03/90/63daa9ca88f3839e9890721a1891c5827cd69413346ed3abdc5bd4b094e5/HTTP_Request_Randomizer-0.0.5-py2.py3-none-any.whl"
            }
        ],
        "1.0.3": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "28a578213380321bc0ce6760ff3d3a5e",
                    "sha256": "bf550a1362d524811192b28be9bfe04212a5753f9eba66cbcd6855e80d39a6f4"
                },
                "downloads": -1,
                "filename": "http_request_randomizer-1.0.3.tar.gz",
                "has_sig": false,
                "md5_digest": "28a578213380321bc0ce6760ff3d3a5e",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 21320,
                "upload_time": "2017-04-12T17:19:33",
                "url": "https://files.pythonhosted.org/packages/f9/92/10f8937df3dc282443f9a53d8e8530a7ea01b8cdd56a61065c89c1f6fcd7/http_request_randomizer-1.0.3.tar.gz"
            }
        ],
        "1.0.4": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "8333f52d1ca1cc65a0f305e1cf27b18a",
                    "sha256": "d5fe749deab9a6f83be0ac4714bbc319392ab49d76fc1abbbfb6bd2ec7fb6e9a"
                },
                "downloads": -1,
                "filename": "http_request_randomizer-1.0.4.tar.gz",
                "has_sig": false,
                "md5_digest": "8333f52d1ca1cc65a0f305e1cf27b18a",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 19890,
                "upload_time": "2017-05-05T14:06:26",
                "url": "https://files.pythonhosted.org/packages/0f/dd/c9cbf17d46db06f922c2d4151399d9eac3042cbc8dc21e0ae88bfedd6b14/http_request_randomizer-1.0.4.tar.gz"
            }
        ],
        "1.0.5": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "4ad9192afc8e85cfc566e547278781c3",
                    "sha256": "4602c66c8872047aa9ca1427cc93241cdd1e1884ff910a302ccc117802470e51"
                },
                "downloads": -1,
                "filename": "http_request_randomizer-1.0.5.tar.gz",
                "has_sig": false,
                "md5_digest": "4ad9192afc8e85cfc566e547278781c3",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 17077,
                "upload_time": "2017-06-11T17:57:54",
                "url": "https://files.pythonhosted.org/packages/2b/14/2ef8e85cb8096c0e4a8fdb3db2d061e43e850f1f786d06abd3953fd7637d/http_request_randomizer-1.0.5.tar.gz"
            }
        ],
        "1.0.7": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "b0e5892be453f972216b089f01fd1910",
                    "sha256": "364274cf627080d4ba849389c434f29ba4c72cca0a3ccc2290144ddf6f42238f"
                },
                "downloads": -1,
                "filename": "http_request_randomizer-1.0.7.tar.gz",
                "has_sig": false,
                "md5_digest": "b0e5892be453f972216b089f01fd1910",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 17052,
                "upload_time": "2017-07-06T18:54:45",
                "url": "https://files.pythonhosted.org/packages/b1/5a/60e69bf2dc635f0885e43b8676f4b40e6b1a3f90c97cd4fcfe188d95b74c/http_request_randomizer-1.0.7.tar.gz"
            }
        ],
        "1.1.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "9b2f2bf7267c1410b5acb80c1aaf842a",
                    "sha256": "1b4b67c95090cba58e23bcd92642bd163eca27c7cedd60997441309f8340148f"
                },
                "downloads": -1,
                "filename": "http_request_randomizer-1.1.0.tar.gz",
                "has_sig": false,
                "md5_digest": "9b2f2bf7267c1410b5acb80c1aaf842a",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 19216,
                "upload_time": "2017-07-30T21:22:10",
                "url": "https://files.pythonhosted.org/packages/b1/37/ee6ea94ca128a97e931385a9d0372f3109bd6a4a24dc2a1ef2d1195270c9/http_request_randomizer-1.1.0.tar.gz"
            }
        ],
        "1.1.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "d1d7e4f7b654acb751ba63fdbe40a297",
                    "sha256": "e1c232f48df3f623894374c776ca6ace5e7a3821d1db59c3394e2200dbcca464"
                },
                "downloads": -1,
                "filename": "http_request_randomizer-1.1.1.tar.gz",
                "has_sig": false,
                "md5_digest": "d1d7e4f7b654acb751ba63fdbe40a297",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 19387,
                "upload_time": "2017-08-08T06:42:45",
                "url": "https://files.pythonhosted.org/packages/8d/f9/99d3f9661569a3c5a6b61068381248f446bc83f64252afab65bf4cc01007/http_request_randomizer-1.1.1.tar.gz"
            }
        ],
        "1.2.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "8f593ebc427cc5f800e22ae02fa57fb0",
                    "sha256": "b7b75b813b4f35b0222ee9813cae9d45beae2913adb66d7454a9d2a4ace1dda7"
                },
                "downloads": -1,
                "filename": "http_request_randomizer-1.2.1.tar.gz",
                "has_sig": false,
                "md5_digest": "8f593ebc427cc5f800e22ae02fa57fb0",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 31123,
                "upload_time": "2018-01-09T11:38:08",
                "url": "https://files.pythonhosted.org/packages/d2/ef/fd777b3ef7b5549a6aca38d159e1e1467841089a31be5e384e18a643ae65/http_request_randomizer-1.2.1.tar.gz"
            }
        ],
        "1.2.2": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "d4c2c476787ed6ba5ed91baf616d22da",
                    "sha256": "23c1785f73b1360ea267544b6df1f0e8871225bcc12d7f607eadec187643f4c5"
                },
                "downloads": -1,
                "filename": "http_request_randomizer-1.2.2.tar.gz",
                "has_sig": false,
                "md5_digest": "d4c2c476787ed6ba5ed91baf616d22da",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 31958,
                "upload_time": "2018-02-21T13:34:57",
                "url": "https://files.pythonhosted.org/packages/f8/cf/989559bf51ddaec7719d0669c613500a73ee88d35c86d536754292dd699d/http_request_randomizer-1.2.2.tar.gz"
            }
        ],
        "1.2.3": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "4b85f08a70c01edaaf581bdf64ff1878",
                    "sha256": "b0e70ce8e49c56d31302c2054dc36843c6c17a48ffe086c557c8a7d177b34d66"
                },
                "downloads": -1,
                "filename": "http_request_randomizer-1.2.3.tar.gz",
                "has_sig": false,
                "md5_digest": "4b85f08a70c01edaaf581bdf64ff1878",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 33258,
                "upload_time": "2018-07-05T01:33:13",
                "url": "https://files.pythonhosted.org/packages/1d/76/17914e8ab62dfc59aaf0c2808aaf825d9e7509c41a9e2fee432e10040494/http_request_randomizer-1.2.3.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "4b85f08a70c01edaaf581bdf64ff1878",
                "sha256": "b0e70ce8e49c56d31302c2054dc36843c6c17a48ffe086c557c8a7d177b34d66"
            },
            "downloads": -1,
            "filename": "http_request_randomizer-1.2.3.tar.gz",
            "has_sig": false,
            "md5_digest": "4b85f08a70c01edaaf581bdf64ff1878",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 33258,
            "upload_time": "2018-07-05T01:33:13",
            "url": "https://files.pythonhosted.org/packages/1d/76/17914e8ab62dfc59aaf0c2808aaf825d9e7509c41a9e2fee432e10040494/http_request_randomizer-1.2.3.tar.gz"
        }
    ]
}