{ "info": { "author": "Neal Wong", "author_email": "ibprnd@gmail.com", "bugtrack_url": null, "classifiers": [ "Development Status :: 5 - Production/Stable", "Framework :: Scrapy", "Intended Audience :: Developers", "License :: OSI Approved :: BSD License", "Programming Language :: Python", "Topic :: Internet :: WWW/HTTP" ], "description": "scrapy-user-agents\n=====================\n\nRandom User-Agent middleware picks up ``User-Agent`` strings based on `Python User Agents `__ and `MDN `__.\n\nInstallation\n-------------\n\nThe simplest way is to install it via `pip`:\n\n pip install scrapy-user-agents\n\nConfiguration\n-------------\n\nTurn off the built-in ``UserAgentMiddleware`` and add\n``RandomUserAgentMiddleware``.\n\nIn Scrapy >=1.0:\n\n.. code:: python\n\n DOWNLOADER_MIDDLEWARES = {\n 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None,\n 'scrapy_user_agents.middlewares.RandomUserAgentMiddleware': 400,\n }\n\nIn Scrapy <1.0:\n\n.. code:: python\n\n DOWNLOADER_MIDDLEWARES = {\n 'scrapy.contrib.downloadermiddleware.useragent.UserAgentMiddleware': None,\n 'scrapy_user_agents.middlewares.RandomUserAgentMiddleware': 400,\n }\n\nUser-Agent File\n---------------------------\n\nA default User-Agent file is included in this repository, it contains about 2200 user agent strings collected from using . You can supply your own User-Agent file by set ``RANDOM_UA_FILE``.\n\n\nConfiguring User-Agent type\n---------------------------\n\nThere's a configuration parameter ``RANDOM_UA_TYPE`` in format ``.``, default is ``desktop.chrome``. For device_type part, only ``desktop``, ``mobile``, ``tablet`` are supported. For browser_type part, only ``chrome``, ``firefox``, ``safari``, ``ie``, ``safari`` are supported. If you don't want to fix to only one browser type, you can use ``random`` to choose from all browser types.\n\nYou can set ``RANDOM_UA_SAME_OS_FAMILY`` to True to just use user agents that belong to the same os family, such as windows, mac os, linux, or android, ios, etc. Default value is True.\n\nUsage with `scrapy-proxies`\n---------------------------\n\nTo use with middlewares of random proxy such as `scrapy-proxies `_, you need:\n\n1. set ``RANDOM_UA_PER_PROXY`` to True to allow switch per proxy\n\n2. set priority of ``RandomUserAgentMiddleware`` to be greater than ``scrapy-proxies``, so that proxy is set before handle UA\n\n\nConfiguring Fake-UserAgent fallback\n-----------------------------------\n\nThere's a configuration parameter ``FAKEUSERAGENT_FALLBACK`` defaulting to\n``None``. You can set it to a string value, for example ``Mozilla`` or\n``Your favorite browser``, this configuration can completely disable any\nannoying exception.\n\n\n", "description_content_type": "", "docs_url": null, "download_url": "", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "https://github.com/hyan15/crawler-demo/tree/master/crawling-basic/scrapy_user_agents", "keywords": "scrapy proxy user-agent web-scraping", "license": "New BSD License", "maintainer": "", "maintainer_email": "", "name": "scrapy-user-agents", "package_url": "https://pypi.org/project/scrapy-user-agents/", "platform": "", "project_url": "https://pypi.org/project/scrapy-user-agents/", "project_urls": { "Homepage": "https://github.com/hyan15/crawler-demo/tree/master/crawling-basic/scrapy_user_agents" }, "release_url": "https://pypi.org/project/scrapy-user-agents/0.1.1/", "requires_dist": [ "user-agents" ], "requires_python": "", "summary": "Automatically pick an User-Agent for every request", "version": "0.1.1" }, "last_serial": 4408632, "releases": { "0.1.0": [ { "comment_text": "", "digests": { "md5": "ad6f47cceb3bc64c05d8e6dc7ca988a4", "sha256": "0256acb0559a05e3ea5e6f1465e0fe3e5c14974d513d5ec47706d8475b31a6d2" }, "downloads": -1, "filename": "scrapy_user_agents-0.1.0-py2.py3-none-any.whl", "has_sig": false, "md5_digest": "ad6f47cceb3bc64c05d8e6dc7ca988a4", "packagetype": "bdist_wheel", "python_version": "py2.py3", "requires_python": null, "size": 7347, "upload_time": "2018-10-22T21:28:00", "url": "https://files.pythonhosted.org/packages/14/0f/ed73e7f8c991a9dc58947c6e08427964aecabf7cc917f825fef67a8ce2ab/scrapy_user_agents-0.1.0-py2.py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "62572493b9a2fbf48c1c7336d7fd68df", "sha256": "8b8c89ffbcee3a8f42e5a5ef8a90d67ce315a025d1f54f45e98c57d9e8553279" }, "downloads": -1, "filename": "scrapy_user_agents-0.1.0.tar.gz", "has_sig": false, "md5_digest": "62572493b9a2fbf48c1c7336d7fd68df", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 4265, "upload_time": "2018-10-22T21:28:02", "url": "https://files.pythonhosted.org/packages/8b/1b/f5fdd1732007a1383e24b76e6cee0d644bc7292efcb5e03c17bf7ee7e586/scrapy_user_agents-0.1.0.tar.gz" } ], "0.1.1": [ { "comment_text": "", "digests": { "md5": "5c34d14eb5955e76ea21c42d781c8a30", "sha256": "284c9af555f3128697a2953ab3cdb987b160b091a12896562d969cf9e81d1350" }, "downloads": -1, "filename": "scrapy_user_agents-0.1.1-py2.py3-none-any.whl", "has_sig": false, "md5_digest": "5c34d14eb5955e76ea21c42d781c8a30", "packagetype": "bdist_wheel", "python_version": "py2.py3", "requires_python": null, "size": 27913, "upload_time": "2018-10-23T23:33:41", "url": "https://files.pythonhosted.org/packages/50/1f/58a58f465f6d3c75b6cca0e470613349504b8c69f3f3963c898ebabdfa21/scrapy_user_agents-0.1.1-py2.py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "90ceaf139d9d9bad8a082413f5696e6f", "sha256": "aa1f78c8cbae42f1a7159c5ea16c2638ac17e78d7d44111d164ed099ec48705f" }, "downloads": -1, "filename": "scrapy_user_agents-0.1.1.win-amd64.zip", "has_sig": false, "md5_digest": "90ceaf139d9d9bad8a082413f5696e6f", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 30076, "upload_time": "2018-10-23T23:33:42", "url": "https://files.pythonhosted.org/packages/89/18/dcf232312662f4242439691142ef58b90c59eb8bb196b9cc86fcbd8c6c08/scrapy_user_agents-0.1.1.win-amd64.zip" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "5c34d14eb5955e76ea21c42d781c8a30", "sha256": "284c9af555f3128697a2953ab3cdb987b160b091a12896562d969cf9e81d1350" }, "downloads": -1, "filename": "scrapy_user_agents-0.1.1-py2.py3-none-any.whl", "has_sig": false, "md5_digest": "5c34d14eb5955e76ea21c42d781c8a30", "packagetype": "bdist_wheel", "python_version": "py2.py3", "requires_python": null, "size": 27913, "upload_time": "2018-10-23T23:33:41", "url": "https://files.pythonhosted.org/packages/50/1f/58a58f465f6d3c75b6cca0e470613349504b8c69f3f3963c898ebabdfa21/scrapy_user_agents-0.1.1-py2.py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "90ceaf139d9d9bad8a082413f5696e6f", "sha256": "aa1f78c8cbae42f1a7159c5ea16c2638ac17e78d7d44111d164ed099ec48705f" }, "downloads": -1, "filename": "scrapy_user_agents-0.1.1.win-amd64.zip", "has_sig": false, "md5_digest": "90ceaf139d9d9bad8a082413f5696e6f", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 30076, "upload_time": "2018-10-23T23:33:42", "url": "https://files.pythonhosted.org/packages/89/18/dcf232312662f4242439691142ef58b90c59eb8bb196b9cc86fcbd8c6c08/scrapy_user_agents-0.1.1.win-amd64.zip" } ] }