{ "info": { "author": "Tudor Corcimar", "author_email": "", "bugtrack_url": null, "classifiers": [ "Development Status :: 3 - Alpha", "Intended Audience :: Developers", "License :: OSI Approved :: MIT License", "Programming Language :: Python :: 3.7", "Topic :: Utilities" ], "description": "![Scraperex](https://i.ibb.co/NKNQj9m/scraper.png \"Scraperex\")\n\nWeb scraper using dynamic proxy and user agent.\n\n#### Description\n\nScraperex is simple and easy to use web scraper for retreiving data from request and **avoiding HTTP 503 error** (usually emerges when server is watching for bots/crawlers/requests while regular scraping). \n\nPakage is generating random user-agent headers using [fake-useragent](https://pypi.org/project/fake-useragent/), and a list of proxy servers that is used while maiking requests.\n\n\n#### Installation\n```python\npip install scraperex\n```\n\n#### Dependencies\n* [requests](https://pypi.org/project/requests/)\n* [fake-useragent](https://pypi.org/project/fake-useragent/)\n\n#### Usage\n```python\nimport scraperex\n```\n**Scrape textual response**\nBasic usage requires one paramether of type dictionary which contains ```url``` to the web resource, and ```regex``` which will extract the data from the response.\n```python\nconfig = {\n 'my_scraping': {\n 'method': 'GET',\n 'url': 'https://www.resource.for/scraping/1',\n 'params': { perPage: 5 },\n 'regex': r'my_regular_expression'\n }\n}\n\nresult = scraperex.find(config)\n```\n**Scrape json response**\nIf you set configuration ```json``` option **True** then ```regex``` option will be ignored and ```(requests) response.json()``` will be invoked.\n```python\nconfig = {\n 'my_scraping': {\n 'method': 'GET',\n 'url': 'https://www.resource.for/scraping/2',\n 'params': { perPage: 5 },\n 'json': True\n }\n}\n\nresult = scraperex.find(config)\n```\n\n***Note**: If proxy server fails, next one from the list will be used, while proxy list is not exhausted or limit is not touched. You can set limitation by sending ```attempts``` parameter. By default attempts are set to 3.*\n```python\nresult = scraperex.find(config, attempts = 1)\n```\n\n#### Config\nConfig ```config``` must be of type dictionary which must contain at least one item used for scraping (as you can guess the amount of the requests will equal at least to the items amount).\n\n***Note**: You also can build a tree of configurations and the same structure will be in your result.*\n\n```python\nconfig = {\n 'item_A': {...},\n 'item_B': {\n 'item_C': {..},\n 'item_D': {..},\n }\n}\n```\n\n***Structured textual ( regex ) results***\nYou can rescrape response content as many times as you wish by passing to ```regex``` property dictionary instead of string (also it is useful if you want to structure your results, scraping will result in the same structure you defined).\n\n```python\n 'item_B': {\n 'url': 'https://www.resource.for/scraping/2',\n 'regex': {\n 'structure_item': {\n 'child_structure_item_1': r'my_regular_expression'\n }\n 'structure_item_1': r'my_regular_expression_1'\n }\n }\n```\n\n*Note: Constructive criticism is always awaited, please share your thoughts with me: [GitHub](https://github.com/userforce/scraper).*", "description_content_type": "text/markdown", "docs_url": null, "download_url": "", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "https://github.com/userforce/scraper", "keywords": "scraper proxy user-agent", "license": "MIT", "maintainer": "", "maintainer_email": "", "name": "scraperex", "package_url": "https://pypi.org/project/scraperex/", "platform": "", "project_url": "https://pypi.org/project/scraperex/", "project_urls": { "Homepage": "https://github.com/userforce/scraper" }, "release_url": "https://pypi.org/project/scraperex/0.2.1/", "requires_dist": null, "requires_python": ">=3.7", "summary": "Scraperex is making ridiculously easy to scrape web resources, staying hidden, by using dynamic proxy and user agent.", "version": "0.2.1" }, "last_serial": 5283231, "releases": { "0.0.1": [ { "comment_text": "", "digests": { "md5": "2c1de41d1c3cc8f07da81e6df7e35176", "sha256": "b1054d3d78f03e2a5d49a17d42f40ed1ab7ab6577b8b5a714cfb5a8a88bb1002" }, "downloads": -1, "filename": "scraperex-0.0.1.tar.gz", "has_sig": false, "md5_digest": "2c1de41d1c3cc8f07da81e6df7e35176", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 3366, "upload_time": "2019-05-14T10:55:16", "url": "https://files.pythonhosted.org/packages/7f/2f/9314e170f9937473639dcddc112abd3f73bdbca89bea06d6af9e22e6d92e/scraperex-0.0.1.tar.gz" } ], "0.0.10": [ { "comment_text": "", "digests": { "md5": "8284d9eb8f3a9cb4094a435a9f20a298", "sha256": "301ac04f85a73ee4567db288d0171b693c299f2e2e4efb82d9e4176c4942c005" }, "downloads": -1, "filename": "scraperex-0.0.10.tar.gz", "has_sig": false, "md5_digest": "8284d9eb8f3a9cb4094a435a9f20a298", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.7", "size": 3738, "upload_time": "2019-05-14T15:32:12", "url": "https://files.pythonhosted.org/packages/75/21/25fc07cdb8cfab519ad4e701f85ce7d8a0363c320365e9f7f2f13c1db1b5/scraperex-0.0.10.tar.gz" } ], "0.0.11": [ { "comment_text": "", "digests": { "md5": "bb903d009d09853f9e03c8414e231acd", "sha256": "74f74aafdee214fbfe2d304d8ef3df5f82df817454b0b75ea33587837cc1862d" }, "downloads": -1, "filename": "scraperex-0.0.11.tar.gz", "has_sig": false, "md5_digest": "bb903d009d09853f9e03c8414e231acd", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.7", "size": 3764, "upload_time": "2019-05-14T16:02:17", "url": "https://files.pythonhosted.org/packages/67/5d/6aa91bcf773e8879aa33fdc362a3508aea3f66f23c224ef4559926f46b24/scraperex-0.0.11.tar.gz" } ], "0.0.12": [ { "comment_text": "", "digests": { "md5": "28766355b138700b57b23c51a41d7070", "sha256": "b712a66b4e836b96b28c5afc87c216b458f479515df11273a77df52fa1929cad" }, "downloads": -1, "filename": "scraperex-0.0.12.tar.gz", "has_sig": false, "md5_digest": "28766355b138700b57b23c51a41d7070", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.7", "size": 3795, "upload_time": "2019-05-14T16:08:16", "url": "https://files.pythonhosted.org/packages/bc/dc/82d7bdf92abc81f568c84844d86c0c35f500b3fc3076b2ecaf347728ee98/scraperex-0.0.12.tar.gz" } ], "0.0.13": [ { "comment_text": "", "digests": { "md5": "dd9538fc6e7f90494493581dc104bafd", "sha256": "c353a88d9cd77b6d73718465c50afc74aa739b1cf2bf6e3d1dbdbb59fe17bad6" }, "downloads": -1, "filename": "scraperex-0.0.13.tar.gz", "has_sig": false, "md5_digest": "dd9538fc6e7f90494493581dc104bafd", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.7", "size": 3776, "upload_time": "2019-05-14T16:31:06", "url": "https://files.pythonhosted.org/packages/7d/ff/3a55984138f0e0727113b030aed48b7927eca5a6ff4cfed79b76affccbf5/scraperex-0.0.13.tar.gz" } ], "0.0.14": [ { "comment_text": "", "digests": { "md5": "ba3ce4e55029ba784896afac1f6fd472", "sha256": "43856c69e7992dccc0dae0e44c17e3bf7721a4cf42a66fc9085e3b58470cd9fc" }, "downloads": -1, "filename": "scraperex-0.0.14-py3-none-any.whl", "has_sig": false, "md5_digest": "ba3ce4e55029ba784896afac1f6fd472", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": ">=3.7", "size": 4812, "upload_time": "2019-05-14T17:56:58", "url": "https://files.pythonhosted.org/packages/91/8f/d6eeccf78cac9909afeba98ad4030190d38fbe0375648e213a0ee1adda36/scraperex-0.0.14-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "14ea50952b5c8e35bc957d8469bca25a", "sha256": "65375757dd5a7a7d5c15b9b8898a7567176b5a9626ce68b3c6166d9ec13e0b07" }, "downloads": -1, "filename": "scraperex-0.0.14.tar.gz", "has_sig": false, "md5_digest": "14ea50952b5c8e35bc957d8469bca25a", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.7", "size": 3741, "upload_time": "2019-05-14T17:56:59", "url": "https://files.pythonhosted.org/packages/76/7c/5be6bff543c04b17e6d4fb8c19178ac91d1f9047c9eddcbb9b04f6049f4d/scraperex-0.0.14.tar.gz" } ], "0.0.15": [ { "comment_text": "", "digests": { "md5": "992ef145a6a314f5a831f8a0853aa096", "sha256": "5d3a3de253c667c1a8b1acef62366d5e8f4e7024c91832b1b3d2fc6f29056c32" }, "downloads": -1, "filename": "scraperex-0.0.15.tar.gz", "has_sig": false, "md5_digest": "992ef145a6a314f5a831f8a0853aa096", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.7", "size": 3719, "upload_time": "2019-05-14T18:00:16", "url": "https://files.pythonhosted.org/packages/d5/7c/adc94975c6b215a91932feb03c98aef0376e7d1e51fa4bb62abce7175924/scraperex-0.0.15.tar.gz" } ], "0.0.16": [ { "comment_text": "", "digests": { "md5": "0305c0f713ab12634a40b87925748bc0", "sha256": "dc3b48b89fefe854178e9483e6a3db5089b252ae70f9261bd18113534dfcc0bb" }, "downloads": -1, "filename": "scraperex-0.0.16.tar.gz", "has_sig": false, "md5_digest": "0305c0f713ab12634a40b87925748bc0", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.7", "size": 3713, "upload_time": "2019-05-14T18:24:58", "url": "https://files.pythonhosted.org/packages/2c/2b/0bb4cb306ccbcee31b3f3487b35cb7c78e08bccb8a30199b83bd348b1a00/scraperex-0.0.16.tar.gz" } ], "0.0.17": [ { "comment_text": "", "digests": { "md5": "289458155dbc1ef41507684f38162398", "sha256": "518caafdcc088414fe360ea89ecafdcac9ab544548126c0f6a988b0bc9827192" }, "downloads": -1, "filename": "scraperex-0.0.17.tar.gz", "has_sig": false, "md5_digest": "289458155dbc1ef41507684f38162398", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.7", "size": 3714, "upload_time": "2019-05-14T18:27:16", "url": "https://files.pythonhosted.org/packages/c8/1d/935b5a6c3d011fd966e24248df11bb5c9dda33ab98bc2638432d7c371d5a/scraperex-0.0.17.tar.gz" } ], "0.0.18": [ { "comment_text": "", "digests": { "md5": "a9081f309c672fbcb91595d5f7b3592d", "sha256": "fc465b1367fa28ba0765e256d9bfb0202379b20f7b4775086b3c72bd15d35781" }, "downloads": -1, "filename": "scraperex-0.0.18.tar.gz", "has_sig": false, "md5_digest": "a9081f309c672fbcb91595d5f7b3592d", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.7", "size": 3864, "upload_time": "2019-05-14T19:19:12", "url": "https://files.pythonhosted.org/packages/30/41/178c0de058df9ed514b7bd4c5343bf9bbb610b83462d1d326b528d6e09c0/scraperex-0.0.18.tar.gz" } ], "0.0.19": [ { "comment_text": "", "digests": { "md5": "eaa38b98fea5468768f9f6855f43333c", "sha256": "cc616d761ab9c170617e30c04cfc22931e8a4650ff8478ab93018d476d3a3f27" }, "downloads": -1, "filename": "scraperex-0.0.19.tar.gz", "has_sig": false, "md5_digest": "eaa38b98fea5468768f9f6855f43333c", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.7", "size": 3945, "upload_time": "2019-05-14T19:30:20", "url": "https://files.pythonhosted.org/packages/f4/b0/87035925db142ea0dfa8db01a7c84ab681c42d1d3c155e1a53a473ff75b2/scraperex-0.0.19.tar.gz" } ], "0.0.2": [ { "comment_text": "", "digests": { "md5": "24c30644634d6d6e975fa7ca3f954832", "sha256": "0ecf961577105a183d9447f7e746b8055fba1c74a1e868e3229bc367b83e0b2c" }, "downloads": -1, "filename": "scraperex-0.0.2.tar.gz", "has_sig": false, "md5_digest": "24c30644634d6d6e975fa7ca3f954832", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.7", "size": 2587, "upload_time": "2019-05-14T11:19:32", "url": "https://files.pythonhosted.org/packages/a4/51/0673c779e0f36bd9887a9afb771b426819316892c76035767292e65f0eed/scraperex-0.0.2.tar.gz" } ], "0.0.21": [ { "comment_text": "", "digests": { "md5": "9d1d76e767176f4936b1ec3de2344d36", "sha256": "b677c64b69ec474ae7c49e88d273aeccf285a66a37f004256aa1033d92c45e81" }, "downloads": -1, "filename": "scraperex-0.0.21.tar.gz", "has_sig": false, "md5_digest": "9d1d76e767176f4936b1ec3de2344d36", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.7", "size": 3949, "upload_time": "2019-05-14T19:46:39", "url": "https://files.pythonhosted.org/packages/af/ce/cb4cf28ebd566d44e5621daa7aac6daefc15b134bae01e3e531600dff4dd/scraperex-0.0.21.tar.gz" } ], "0.0.3": [ { "comment_text": "", "digests": { "md5": "274eb0b3fa8fa05690dbdf3069d2efa7", "sha256": "8415019cf7b59bd8e947dbe3cd93069bf26cda090a176dc6cdadd58fb34ab264" }, "downloads": -1, "filename": "scraperex-0.0.3.tar.gz", "has_sig": false, "md5_digest": "274eb0b3fa8fa05690dbdf3069d2efa7", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.7", "size": 3690, "upload_time": "2019-05-14T13:15:21", "url": "https://files.pythonhosted.org/packages/c8/92/7f06e8c7368a12a0b105356b15b967c8e5627afe6cead08fb93d29bca300/scraperex-0.0.3.tar.gz" } ], "0.0.4": [ { "comment_text": "", "digests": { "md5": "40957d81551da8ed025913f2f05ffd5e", "sha256": "a7f422a2f85be10eb43dcc441c6b542c31241e73ae20b926320c6adb3b9a902b" }, "downloads": -1, "filename": "scraperex-0.0.4.tar.gz", "has_sig": false, "md5_digest": "40957d81551da8ed025913f2f05ffd5e", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.7", "size": 3671, "upload_time": "2019-05-14T14:19:25", "url": "https://files.pythonhosted.org/packages/93/d6/abe7c1369b77ec261fe0e5caeff6de9fb502147d4870889d29d4496cac47/scraperex-0.0.4.tar.gz" } ], "0.0.5": [ { "comment_text": "", "digests": { "md5": "ba1e131aff9cc2e39c1a6bd076c39556", "sha256": "11e30bf7c232a4c4c35a60c29e8d619f70ac64dad647bf8acc2d3fa4bdd4f191" }, "downloads": -1, "filename": "scraperex-0.0.5.tar.gz", "has_sig": false, "md5_digest": "ba1e131aff9cc2e39c1a6bd076c39556", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.7", "size": 3670, "upload_time": "2019-05-14T14:34:01", "url": "https://files.pythonhosted.org/packages/cc/35/57cbdb8499f44e0c27dbf35f50f24b51da64a9eb90297bbc208c9f7eaa3a/scraperex-0.0.5.tar.gz" } ], "0.0.6": [ { "comment_text": "", "digests": { "md5": "271426781e08b59d45af9257194b174d", "sha256": "30f1b1145b20843d8f0a78d673aaead4568ffce24e63db289486c1e6fbb0140b" }, "downloads": -1, "filename": "scraperex-0.0.6.tar.gz", "has_sig": false, "md5_digest": "271426781e08b59d45af9257194b174d", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.7", "size": 3695, "upload_time": "2019-05-14T15:02:35", "url": "https://files.pythonhosted.org/packages/ce/66/84b1851fa9ce745295c1ba329fd98a69e26d8db1e294aeb7c95825eb80d2/scraperex-0.0.6.tar.gz" } ], "0.0.7": [ { "comment_text": "", "digests": { "md5": "67ff7cbcc956a2429a3a4b8b8dde81ad", "sha256": "bc9d17abb298f4c4f463b8a338ee7c02bf401cd7c636a6fb9c2142ba18c34007" }, "downloads": -1, "filename": "scraperex-0.0.7.tar.gz", "has_sig": false, "md5_digest": "67ff7cbcc956a2429a3a4b8b8dde81ad", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.7", "size": 3698, "upload_time": "2019-05-14T15:08:07", "url": "https://files.pythonhosted.org/packages/4a/bb/3eca73955ee0e8dbcb83e24614fd4f5feb79f1469b50eaadae2e22d584ca/scraperex-0.0.7.tar.gz" } ], "0.0.8": [ { "comment_text": "", "digests": { "md5": "b88061c972f55e73c542e66087a5831b", "sha256": "6741b73a49848d30661d27736b708c957fff695bc17a5ee17492072241ab9b77" }, "downloads": -1, "filename": "scraperex-0.0.8.tar.gz", "has_sig": false, "md5_digest": "b88061c972f55e73c542e66087a5831b", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.7", "size": 3704, "upload_time": "2019-05-14T15:12:48", "url": "https://files.pythonhosted.org/packages/1b/df/582d966b2087bf05ff2e4c06a60b78adb5363fa8263cacf2cbd283cbf15f/scraperex-0.0.8.tar.gz" } ], "0.0.9": [ { "comment_text": "", "digests": { "md5": "dacabcbf06f92f2d2c660b0c58ac0a90", "sha256": "ad49d2bd2c196e59da4ec074d8590acb71905190a912860ce19a641eacf65111" }, "downloads": -1, "filename": "scraperex-0.0.9.tar.gz", "has_sig": false, "md5_digest": "dacabcbf06f92f2d2c660b0c58ac0a90", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.7", "size": 3695, "upload_time": "2019-05-14T15:27:43", "url": "https://files.pythonhosted.org/packages/fb/19/1443d264e265320eb29174780770e843ef7de857163ab2e1416e662afc70/scraperex-0.0.9.tar.gz" } ], "0.1.0": [ { "comment_text": "", "digests": { "md5": "6743d6a1363bfac372897cd892ecc6af", "sha256": "1c42f88f052e268e18fa032ac47dd9284337dcf7925d0408c13b7fc791295e54" }, "downloads": -1, "filename": "scraperex-0.1.0.tar.gz", "has_sig": false, "md5_digest": "6743d6a1363bfac372897cd892ecc6af", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.7", "size": 4217, "upload_time": "2019-05-15T12:47:50", "url": "https://files.pythonhosted.org/packages/ea/2e/d264ce0271ada08dafb5743f498a6b53d80133ef970e9b86c56b196c4b4d/scraperex-0.1.0.tar.gz" } ], "0.1.1": [ { "comment_text": "", "digests": { "md5": "06eb4eacbab33a45e115429674ea6b38", "sha256": "33da5163dfc8ffeb7eeb2b75af5d7cf4c4e09cc26acb7c7804e7cf303125e2b3" }, "downloads": -1, "filename": "scraperex-0.1.1.tar.gz", "has_sig": false, "md5_digest": "06eb4eacbab33a45e115429674ea6b38", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.7", "size": 4326, "upload_time": "2019-05-15T14:26:11", "url": "https://files.pythonhosted.org/packages/d3/57/cc4c9e310e204a6fbbd4862375038801b98c484c0bd68f83646183774da7/scraperex-0.1.1.tar.gz" } ], "0.1.2": [ { "comment_text": "", "digests": { "md5": "28ce8d2e2f68004cdd6ff0de6bb73fda", "sha256": "6f54b13779b7edfdf425733a4ff4579d5c0f9c0c8fcf31ecfd529f095e2ad875" }, "downloads": -1, "filename": "scraperex-0.1.2.tar.gz", "has_sig": false, "md5_digest": "28ce8d2e2f68004cdd6ff0de6bb73fda", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.7", "size": 4313, "upload_time": "2019-05-15T14:30:38", "url": "https://files.pythonhosted.org/packages/f4/9d/9209ebfde2bdf03e4ff5afa7b64fa0a062363dea1c6dfd43a17632097b69/scraperex-0.1.2.tar.gz" } ], "0.2.0": [ { "comment_text": "", "digests": { "md5": "e0b82feaee25fd0080f78eafcfc1edb7", "sha256": "1edba4525d06c1dbf0a22fd726c05033051c2b8e7e3f78981f23cbc32f385d89" }, "downloads": -1, "filename": "scraperex-0.2.0.tar.gz", "has_sig": false, "md5_digest": "e0b82feaee25fd0080f78eafcfc1edb7", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.7", "size": 4902, "upload_time": "2019-05-17T16:22:31", "url": "https://files.pythonhosted.org/packages/50/bd/6a674b4e747590f3c8ec5950060b7fafcceeac152b3ee3216686af1a25ec/scraperex-0.2.0.tar.gz" } ], "0.2.1": [ { "comment_text": "", "digests": { "md5": "c9f9eefee1a203a1c9f5ce95fb02eff1", "sha256": "291e11013eb60548c586f7d6d3a962882a44edf706da4fa1f610ba0b4cd91915" }, "downloads": -1, "filename": "scraperex-0.2.1.tar.gz", "has_sig": false, "md5_digest": "c9f9eefee1a203a1c9f5ce95fb02eff1", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.7", "size": 5155, "upload_time": "2019-05-17T17:11:11", "url": "https://files.pythonhosted.org/packages/f5/71/aa9dec3d844cb9a8dbbcc5f641fedad967a8b5465071bd5252a08e7e2568/scraperex-0.2.1.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "c9f9eefee1a203a1c9f5ce95fb02eff1", "sha256": "291e11013eb60548c586f7d6d3a962882a44edf706da4fa1f610ba0b4cd91915" }, "downloads": -1, "filename": "scraperex-0.2.1.tar.gz", "has_sig": false, "md5_digest": "c9f9eefee1a203a1c9f5ce95fb02eff1", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.7", "size": 5155, "upload_time": "2019-05-17T17:11:11", "url": "https://files.pythonhosted.org/packages/f5/71/aa9dec3d844cb9a8dbbcc5f641fedad967a8b5465071bd5252a08e7e2568/scraperex-0.2.1.tar.gz" } ] }