{ "info": { "author": "Scrapinghub", "author_email": "info@scrapinghub.com", "bugtrack_url": null, "classifiers": [ "Development Status :: 4 - Beta", "License :: OSI Approved :: BSD License", "Operating System :: OS Independent", "Programming Language :: Python", "Programming Language :: Python :: 2", "Programming Language :: Python :: 2.7", "Programming Language :: Python :: 3", "Programming Language :: Python :: 3.5" ], "description": "===================\nscrapy-querycleaner\n===================\n\n.. image:: https://travis-ci.org/scrapy-plugins/scrapy-querycleaner.svg?branch=master\n :target: https://travis-ci.org/scrapy-plugins/scrapy-querycleaner\n\n.. image:: https://codecov.io/gh/scrapy-plugins/scrapy-querycleaner/branch/master/graph/badge.svg\n :target: https://codecov.io/gh/scrapy-plugins/scrapy-querycleaner\n\nThis is a Scrapy spider middleware to clean up the request URL GET query parameters\nat the output of the spider in accordance with the patterns provided by the user.\n\n\nInstallation\n============\n\nInstall scrapy-querycleaner using ``pip``::\n\n $ pip install scrapy-querycleaner\n\n\nConfiguration\n=============\n\n1. Add ``QueryCleanerMiddleware`` by including it in ``SPIDER_MIDDLEWARES``\n in your ``settings.py`` file::\n\n SPIDER_MIDDLEWARES = {\n 'scrapy_querycleaner.QueryCleanerMiddleware': 100,\n }\n\n Here, priority ``100`` is just an example.\n Set its value depending on other middlewares you may have enabled already.\n\n2. Enable the middleware using either ``QUERYCLEANER_REMOVE``\n or ``QUERYCLEANER_KEEP`` (or both) in your ``setting.py``.\n\n\nUsage\n=====\n\nAt least one of the following settings needs to be present for the\nmiddleware to be enabled.\n\n\n.. note::\n You can specify a list of parameter names by using the ``|`` (*OR*) regex operator.\n\n For example, the pattern ``search|login|postid`` will match query parameters *search*,\n *login* and *postid*.\n This is by far the most common usage case.\n\n And by setting ``QUERYCLEANER_REMOVE`` value to ``.*``\n you can completely remove all URL query parameters.\n\n\nSupported settings\n------------------\n\n``QUERYCLEANER_REMOVE``\n a pattern (regular expression) that a query parameter name must match\n in order to be removed from the URL. (All the others will be accepted.)\n\n``QUERYCLEANER_KEEP``\n a pattern that a query parameter name must match in order to be kept in the URL.\n (All the others will be removed.)\n\nYou can combine both if some query parameters patterns should be kept and some should not.\n\nThe **remove** pattern has precedence over the *keep* one.\n\n\nExample\n-------\n\nLet's suppose that the spider extracts URLs like::\n\n http://www.example.com/product.php?pid=135&cid=12&ttda=12\n\nand we want to leave only the parameter ``pid``.\n\nTo achieve this objective we can use either ``QUERYCLEANER_REMOVE``\nor ``QUERYCLEANER_KEEP``:\n\n- In the first case, the pattern would be ``cid|ttda``::\n\n QUERYCLEANER_REMOVE = 'cid|ttda'\n\n- In the second case, ``pid``::\n\n QUERYCLEANER_KEEP = 'pid'\n\n\nThe best solution depends on a particular case, that is,\nhow the query filters will affect any other URL that the spider is expected to extract.\n", "description_content_type": null, "docs_url": null, "download_url": "", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "https://github.com/scrapy-plugins/scrapy-querycleaner", "keywords": "", "license": "BSD", "maintainer": "", "maintainer_email": "", "name": "scrapy-querycleaner", "package_url": "https://pypi.org/project/scrapy-querycleaner/", "platform": "Any", "project_url": "https://pypi.org/project/scrapy-querycleaner/", "project_urls": { "Homepage": "https://github.com/scrapy-plugins/scrapy-querycleaner" }, "release_url": "https://pypi.org/project/scrapy-querycleaner/1.0.0/", "requires_dist": [ "scrapy (>=1.0)", "six" ], "requires_python": "", "summary": "Scrapy spider middleware to clean up query parameters in request URLs", "version": "1.0.0" }, "last_serial": 2195705, "releases": { "0.1.0": [ { "comment_text": "", "digests": { "md5": "d4013e5dc8ea7195fce08ee14e01c40d", "sha256": "84fc57f2e3e37ada3aede6ff41ef3ad2070f9ea227910c535b172ab2f1eff81b" }, "downloads": -1, "filename": "scrapy_querycleaner-0.1.0-py2.py3-none-any.whl", "has_sig": false, "md5_digest": "d4013e5dc8ea7195fce08ee14e01c40d", "packagetype": "bdist_wheel", "python_version": "py2.py3", "requires_python": null, "size": 3777, "upload_time": "2016-06-30T11:39:41", "url": "https://files.pythonhosted.org/packages/78/8c/3b41aabf92dcbdd197ffc8f0a8e21e5ca5fb373a6ee8b4a06fd342140c25/scrapy_querycleaner-0.1.0-py2.py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "ab0f6da43aadf5dab3f565ab7e4f75e6", "sha256": "df8d9a7e3e81ae7f82939890971b66a67a475e1e28bea64248ce509173a34914" }, "downloads": -1, "filename": "scrapy-querycleaner-0.1.0.tar.gz", "has_sig": false, "md5_digest": "ab0f6da43aadf5dab3f565ab7e4f75e6", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 3244, "upload_time": "2016-06-30T11:39:45", "url": "https://files.pythonhosted.org/packages/8a/f0/fe2183b9545cd646114018e33ce9d06361ed7e93f77b28c0dc0c279d6129/scrapy-querycleaner-0.1.0.tar.gz" } ], "1.0.0": [ { "comment_text": "", "digests": { "md5": "156fa3f1a03f8c64a1dad77c345e2778", "sha256": "a40002384a277db89797fcf6c029bec46b05642c1e646f1476385b803745b333" }, "downloads": -1, "filename": "scrapy_querycleaner-1.0.0-py2.py3-none-any.whl", "has_sig": false, "md5_digest": "156fa3f1a03f8c64a1dad77c345e2778", "packagetype": "bdist_wheel", "python_version": "py2.py3", "requires_python": null, "size": 3774, "upload_time": "2016-06-30T12:08:54", "url": "https://files.pythonhosted.org/packages/80/2a/a3d6b7779dff0932017fcbfdad4b67b710160e524ac2914103ee963dee71/scrapy_querycleaner-1.0.0-py2.py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "e38bc7780bb86d577ebdb5f3676a9919", "sha256": "3f3fdc7558076e7a0dfdadb803d42661c372b75f05c436a7a721e28d16ec5d5a" }, "downloads": -1, "filename": "scrapy-querycleaner-1.0.0.tar.gz", "has_sig": false, "md5_digest": "e38bc7780bb86d577ebdb5f3676a9919", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 3231, "upload_time": "2016-06-30T12:09:28", "url": "https://files.pythonhosted.org/packages/19/15/ede0e13684f7eb1d685e3428c78899408ffb72f4a63c613240feddbbb8af/scrapy-querycleaner-1.0.0.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "156fa3f1a03f8c64a1dad77c345e2778", "sha256": "a40002384a277db89797fcf6c029bec46b05642c1e646f1476385b803745b333" }, "downloads": -1, "filename": "scrapy_querycleaner-1.0.0-py2.py3-none-any.whl", "has_sig": false, "md5_digest": "156fa3f1a03f8c64a1dad77c345e2778", "packagetype": "bdist_wheel", "python_version": "py2.py3", "requires_python": null, "size": 3774, "upload_time": "2016-06-30T12:08:54", "url": "https://files.pythonhosted.org/packages/80/2a/a3d6b7779dff0932017fcbfdad4b67b710160e524ac2914103ee963dee71/scrapy_querycleaner-1.0.0-py2.py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "e38bc7780bb86d577ebdb5f3676a9919", "sha256": "3f3fdc7558076e7a0dfdadb803d42661c372b75f05c436a7a721e28d16ec5d5a" }, "downloads": -1, "filename": "scrapy-querycleaner-1.0.0.tar.gz", "has_sig": false, "md5_digest": "e38bc7780bb86d577ebdb5f3676a9919", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 3231, "upload_time": "2016-06-30T12:09:28", "url": "https://files.pythonhosted.org/packages/19/15/ede0e13684f7eb1d685e3428c78899408ffb72f4a63c613240feddbbb8af/scrapy-querycleaner-1.0.0.tar.gz" } ] }