{ "info": { "author": "Jake Cahill", "author_email": "kaelscion@gmail.com", "bugtrack_url": null, "classifiers": [ "Intended Audience :: Developers", "Programming Language :: Python :: 3" ], "description": "# slither\nA simple, easy to use framework for adding randomized, anonymous IP addresses and user-agents to web scrapers, crawlers, and penetration testing solutions\n\nSlither is designed to add randomized User-Agent strings and Anonymous IP addresses from proxy sources around the web for use in your web scraping, penetration testing, or data aggregation projects. Please respect the owners and hardware of the data you are scraping. **I BEAR NO RESPONIBILITY IF YOU USE THIS SOFTWARE WITHOUT PERMISSION OR TO DO HARM TO WEB ASSETS. RESPECT THE DATA AND PLATFORMS. BE ETHICAL WITH YOUR SCRAPING AND UNDERSTAND THAT ALL ASSETS HAVE FINITE RESOURCES. DONT DDOS THINGS. ITS BAD.**\n\n**SLITHER IS ONYL COMPATIBLE WITH PYTHON 3. NO SUPPORT FOR PYTHON 2 IS PLANNED**. That being said, to install on a machine with only Python 3 installed:\n\n`pip install slitherlib`\n\nOn a machine with both Python 2 and 3 installed:\n\n`pip3 install slitherlib`\n\nFrom there, simply from slitherlib.slither import Snake to add the package to the project file that contains your scraping code.\n\nEach instance of the Slither class has two variables, ip and ua for IP Address and User-Agent respectively. The IPs are pulled, in real time, from web proxy sources every time you declare and instance of the Slither class so no need to worry about IPs going stale. The majority of the addresses are less than 20 minutes old when pulled down and many are less than 10. An example of anonymizing your scraper with the Requests library looks like this:\n\n```python\nimport requests\n\nfrom random import choice\nfrom slitherlib.slither import Snake\n\ns = requests.Session()\nsnake = Snake()\nip_addresses = snake.ips\nuser_agents = snake.uas\nheaders = {\n \"User-Agent\" : choice(user_agents)\n }\ntry:\n ip = ip_addresses.pop()\n r = s.get('https://www.google.com', proxies={'https' : ip, 'http' : ip} , headers=headers)\n print(r.status_code)\nexcept requests.exceptions.ProxyError:\n print('Proxy Timed Out. Removing and Retrying')\n ip = ip_addresses.pop()\n r = s.get('https://www.google.com', proxies={'https' : ip, 'http' : ip} , headers=headers)\nexcept IndexError:\n print(\"We've run out of IPs and/or User-Agents! Re-run your script to get more!\")\n```\n\nThis method also supports concurrency and adding an individual IP and/or UA to each thread or process that is spawned by your project! Accomplishing this is done as follows:\n\n```python\nimport requests\n\nfrom slither import Slither\nfrom concurrent.futures import ThreadPoolExecutor, wait, as_completed\n\n# specify the number of threads your scraper will use as the \"thread count\".\n# By default, thread_count is set to \"all\", meaning that you will pull down all available IP addresses and user agents. To specify a specific number, pass the number of desired items as an int to the named `thread_count` argument\n\nnum_of_threads = 7\nfutures = list_of_urls_to_scrape\nnew_slither = Slither(thread_count=num_of_threads)\n#returns a list of dictionaries of IPs and User-Agents\nfor i in new_slither.masks:\n #spawn your threads here assigning i['address'] to your thread's proxy parameter and \n #i['user-agent'] to each thread's 'User-Agent' header parameter\n ...\n```\n\nHave fun and happy Scraping!\n\n\n", "description_content_type": "text/markdown", "docs_url": null, "download_url": "", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "https://github.com/kaelscion/slither", "keywords": "", "license": "MIT", "maintainer": "", "maintainer_email": "", "name": "slitherlib", "package_url": "https://pypi.org/project/slitherlib/", "platform": "", "project_url": "https://pypi.org/project/slitherlib/", "project_urls": { "Homepage": "https://github.com/kaelscion/slither" }, "release_url": "https://pypi.org/project/slitherlib/0.1.4/", "requires_dist": [ "requests", "bs4" ], "requires_python": "", "summary": "A web-proxy IP and user agent anonymizing framework for web scrapers, penetration testers, and ethical hackers", "version": "0.1.4" }, "last_serial": 5525303, "releases": { "0.1.0": [ { "comment_text": "", "digests": { "md5": "3c87177b72233eb37f7c84b18096f643", "sha256": "bb688af1be6c62e8df825c27b716f4b5640633131c7240e274ee242260e153fa" }, "downloads": -1, "filename": "slitherlib-0.1.0-py3-none-any.whl", "has_sig": false, "md5_digest": "3c87177b72233eb37f7c84b18096f643", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": null, "size": 4684, "upload_time": "2019-07-12T21:33:32", "url": "https://files.pythonhosted.org/packages/66/60/49d732530dc2c1b14b2d8cd5cd13f857c5385ceda42d66470548e6cef274/slitherlib-0.1.0-py3-none-any.whl" } ], "0.1.1": [ { "comment_text": "", "digests": { "md5": "3063085f6e3aad44094f34930fa4deb1", "sha256": "79b8ca16804f19889fe005ab31b98cbf752286c0d8a704f96f55a835c6b319a0" }, "downloads": -1, "filename": "slitherlib-0.1.1-py3-none-any.whl", "has_sig": false, "md5_digest": "3063085f6e3aad44094f34930fa4deb1", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": null, "size": 4686, "upload_time": "2019-07-12T21:39:39", "url": "https://files.pythonhosted.org/packages/9f/ed/1bb198deb75eed1c40f625c7ec51679f55b6b1c3d59c4e9337450fea1121/slitherlib-0.1.1-py3-none-any.whl" } ], "0.1.2": [ { "comment_text": "", "digests": { "md5": "f35e70c15b0f907d894be50ce0d60e1b", "sha256": "ea8ed4362f1aba301d471b5331b277e54b58b633ce45b83bad01b96d47ebd584" }, "downloads": -1, "filename": "slitherlib-0.1.2-py3-none-any.whl", "has_sig": false, "md5_digest": "f35e70c15b0f907d894be50ce0d60e1b", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": null, "size": 5496, "upload_time": "2019-07-12T21:47:41", "url": "https://files.pythonhosted.org/packages/a4/d1/72fb378a24b4807b11350802545064ea8d6dde38ac4b6a1942255cdd2e29/slitherlib-0.1.2-py3-none-any.whl" } ], "0.1.3": [ { "comment_text": "", "digests": { "md5": "3305e8b0885c6bee3336e14f753888a3", "sha256": "5e3f192c3e1c9a0124d3fe3bfa464539c319ff7aa0c1221750712813bebf450f" }, "downloads": -1, "filename": "slitherlib-0.1.3-py3-none-any.whl", "has_sig": false, "md5_digest": "3305e8b0885c6bee3336e14f753888a3", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": null, "size": 6974, "upload_time": "2019-07-12T22:21:25", "url": "https://files.pythonhosted.org/packages/ed/2d/6b2c3b5bb4312809e8d3f625bad7f5997dc534b7832e3fac21129a68a69a/slitherlib-0.1.3-py3-none-any.whl" } ], "0.1.4": [ { "comment_text": "", "digests": { "md5": "1b0c47aa91889ce3382827920a655513", "sha256": "c881e1dd81b908ae4bd54c71dd88ebbd603f384847c0add87340ea7f5ba4d46c" }, "downloads": -1, "filename": "slitherlib-0.1.4-py3-none-any.whl", "has_sig": false, "md5_digest": "1b0c47aa91889ce3382827920a655513", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": null, "size": 6976, "upload_time": "2019-07-12T22:23:40", "url": "https://files.pythonhosted.org/packages/2d/8e/ad1e6b8885286d903506a6f4c24a04b59ec7d3b7bd51857b5adabb9abe8d/slitherlib-0.1.4-py3-none-any.whl" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "1b0c47aa91889ce3382827920a655513", "sha256": "c881e1dd81b908ae4bd54c71dd88ebbd603f384847c0add87340ea7f5ba4d46c" }, "downloads": -1, "filename": "slitherlib-0.1.4-py3-none-any.whl", "has_sig": false, "md5_digest": "1b0c47aa91889ce3382827920a655513", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": null, "size": 6976, "upload_time": "2019-07-12T22:23:40", "url": "https://files.pythonhosted.org/packages/2d/8e/ad1e6b8885286d903506a6f4c24a04b59ec7d3b7bd51857b5adabb9abe8d/slitherlib-0.1.4-py3-none-any.whl" } ] }