{
    "info": {
        "author": "granitosaurus",
        "author_email": "granitosaurus@pm.me",
        "bugtrack_url": null,
        "classifiers": [],
        "description": "# scrapy-test\n\nScrapy test is a validation/test framework for validating scrapy results. \nThis framework is capable of testing scrapy crawl and stats output.\n\nSee [example](example/readme.md) project for hackernews crawler with full test suite.\n\n# Philosophy and Architecture\n\n![architecture illustration](illustrations/architecture.png)\n\n`scrapy-test` tries to replicate `scrapy.Item` definition but instead of defining fields it defines test for every field.  \nTests are callables that either returns a failure message if some condition is met. \nExample item specification:\n\n```\nclass MyItem(Item):\n    name = Field()\n    url = Field()\n\nclass TestMyItem(ItemSpec):\n    item_cls = MyItem\n\n    # define tests\n    name_test = Match('some-regex-pattern')\n    url_test = lamda v: 'bad url' if 'cat' in v else ''\n\n    # define coverage\n    url_cov = 100  # 100% - every item should have url field\n```\n\n`scrapy-test` also supports stats output validation. When scrapy finished crawling it outputs various stats like error count etc. `StatSpec` can be defined to validate these stats:\n\n```\nclass MyStats(StatsSpec):\n    spider_cls = MySpder1, MySpider2\n    # or multiple spiders\n    validation = {  #stat_name_pattern : tests\n        'item_scraped_count': MoreThan(1),\n        'downloader/response_status_count/50\\d': LessThan(1),\n    }\n    # required stat keys\n    required = ['stat_pattern.+']\n```\n\nFinally `scrapy-test` determines failure by asserting if there are any messages generated by either stat ir item specifications (exit code 1 and 0 respectively).\n\n\n# Usage\n\n## Setup\n\n1. `test.py` module should be created in spider directory.  \nFor example creating `test.py` \n\n    ```\n    scrapy-test-example/\n    \u251c\u2500\u2500 example\n    \u2502\u00a0\u00a0 \u251c\u2500\u2500 __init__.py\n    \u2502\u00a0\u00a0 \u2514\u2500\u2500 test.py\n    \u2514\u2500\u2500 scrapy.cfg\n    ```\n\n2. Add test file config to `scrapy.cfg`:\n\n    ```ini\n    [settings]\n    default = example.settings\n    [test]\n    root = example.test \n    ```\n\n3. Define `ItemSpec` for item field validation:\n\n    ```python\n    from scrapytest.tests import Match, Equal, Type, MoreThan, Map, Len, Required\n\n    class TestPost(ItemSpec):\n        # defining item that is being covered\n        item_cls = PostItem\n\n        # defining field tests\n        title_test = Match('.{5,}')\n        points_test = Type(int), MoreThan(0)\n        author_test = Type(str), Match('.{3}')\n        comments_test = Type(list), Required()\n\n        # also supports methods!\n        def url_test(self, value: str):\n            if not value.startswith('http'):\n                return f'Invalid url: {value}'\n            return ''\n    ```\n\n    `ItemSpec` class should contain attributes that end in `_test` these attributes have be callables (functions, methods etc.) that return message(s) if failure is encountered. See the `url_test` example above.\n\n4. Define `StatSpec` for crawl stats validation:\n\n    ```python\n    class TestStats(StatsSpec):\n        # stat pattern: test functions\n        validate = {  # this is default\n            'log_count/ERROR$': LessThan(1),\n            'item_scraped_count': MoreThan(1),\n            'finish_reason': Match('finished'),\n        }\n        # these stats shoudl be required\n        required = ['some_cool_stat']  \n    ```\n\n    `StatsSpec` should contain `validate` attribute with `pattern: tests` dictionary. \n\n5. Define `Spider` classes:\n\n    ```python\n    from project.spiders import HackernewsSpider \n\n    class TestHackernewsSpider(HackernewsSpider):\n        test_urls = [\n            \"https://news.ycombinator.com/item?id=19187417\",\n        ]\n\n        def start_requests(self):\n            for url in self.test_urls:\n                yield Request(url, self.parse_submission)\n    ```\n    This spider should extend your production spider that simply crawls the urls without doing discovery. Alternatively you can also not extend anything for live testing.\n\n## Running\n\n```shell\n$ scrapy-test --help                                                                                                 \nUsage: scrapy-test [OPTIONS] [SPIDER_NAME]\n\n  run scrapy-test tests and output messages and appropriate exit code (1 for\n  failed, 0 for passed)\n\nOptions:\n  --cache  enable HTTPCACHE_ENABLED setting for this run\n  --help       Show this message and exit.\n```\n\nTo run the tests use cli command:\n```\n$ scrapy-test <spider_name>\n```\n\nSpider name can be skipped for running all spiders\n\n## Notifications\n\n`scrapy-test` supports notification hooks on either test failure or success:\n\n\n      --notify-on-error TEXT    send notification on failure, choice from:\n                                ['slack']\n      --notify-on-all TEXT      send notification on failure or success, choice\n                                from: ['slack']\n      --notify-on-success TEXT  send notification on success, choice from:\n                                ['slack']\n\nRight `scrapy-test` offers these notifiers:\n\n    * Slack - to configure slack notification follow slack [incoming webhooks](https://slack.com/apps/A0F7XDUAZ-incoming-webhooks) app and supply these settings in `scrapy.cfg`:\n\n        slack_url = https://hooks.slack.com/services/AAA/BBB/CCC\n        # where the message goes to\n        slack_channel = #cats\n        # bot's name\n        slack_username = bender\n        # bot's avatar\n        slack_icon_emoji = :bender:\n        # maintainer will be mentioned on error\n        slack_maintainer = @bernard\n\n\n\n\n",
        "description_content_type": "text/markdown",
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://gitlab.com/granitosaurus/scrapy-test",
        "keywords": "",
        "license": "GPLv3",
        "maintainer": "",
        "maintainer_email": "",
        "name": "scrapy-test",
        "package_url": "https://pypi.org/project/scrapy-test/",
        "platform": "",
        "project_url": "https://pypi.org/project/scrapy-test/",
        "project_urls": {
            "Homepage": "https://gitlab.com/granitosaurus/scrapy-test"
        },
        "release_url": "https://pypi.org/project/scrapy-test/0.6.1/",
        "requires_dist": [
            "click",
            "scrapy"
        ],
        "requires_python": "",
        "summary": "scrapy output testing framework",
        "version": "0.6.1"
    },
    "last_serial": 5456005,
    "releases": {
        "0.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "17732492f22b2f25665fb345e4f74120",
                    "sha256": "dd7b846ba2b0b7879522015dcc411f72189335221f5c04516a57649b904273ec"
                },
                "downloads": -1,
                "filename": "scrapy_test-0.1-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "17732492f22b2f25665fb345e4f74120",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 9156,
                "upload_time": "2019-02-25T17:02:32",
                "url": "https://files.pythonhosted.org/packages/c3/71/cbc75cc86f70c1368bbe498c2a0f81db620fa711a91ba339fe3a4ede746e/scrapy_test-0.1-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "5a3f85712a2c48a5fed3760dd9262dbe",
                    "sha256": "ea064bb0b8988f53a28fa047406e70adc2685a611d94bc554c1cba447650263a"
                },
                "downloads": -1,
                "filename": "scrapy-test-0.1.tar.gz",
                "has_sig": false,
                "md5_digest": "5a3f85712a2c48a5fed3760dd9262dbe",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 6740,
                "upload_time": "2019-02-25T17:02:35",
                "url": "https://files.pythonhosted.org/packages/6c/ac/1000dbe76eacf8b7c8effe17e1e80af533032c46ff3797dd4e890b48ed5a/scrapy-test-0.1.tar.gz"
            }
        ],
        "0.2": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "5a0d882c82bee98a5249a9788141ac94",
                    "sha256": "82ec9cccefcbd0ffecd63c935cebf46612094616e0e260574eb9872b968a37d2"
                },
                "downloads": -1,
                "filename": "scrapy_test-0.2-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "5a0d882c82bee98a5249a9788141ac94",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 23433,
                "upload_time": "2019-03-05T10:35:06",
                "url": "https://files.pythonhosted.org/packages/31/89/c9b007b204efa151ddf3e45a2f940ae44b0bbdc59a59bf030d15e53159c5/scrapy_test-0.2-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "56af90277e429eafe7867cfee1556abf",
                    "sha256": "eb000a764df80ca7359d55076d35920e23fb8222b7b90c989ae6ef127119d69c"
                },
                "downloads": -1,
                "filename": "scrapy-test-0.2.tar.gz",
                "has_sig": false,
                "md5_digest": "56af90277e429eafe7867cfee1556abf",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 8436,
                "upload_time": "2019-03-05T10:35:09",
                "url": "https://files.pythonhosted.org/packages/98/8c/b5e9732c33982019497779dda14a17d3919e54898f1235c387fac955531f/scrapy-test-0.2.tar.gz"
            }
        ],
        "0.4": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "003ee63fa819e199c6a2ba9728012cb8",
                    "sha256": "5b5bdb8be64f042ef319f6884ab1261307ac116e984b3e408ebbb8b986d1c693"
                },
                "downloads": -1,
                "filename": "scrapy_test-0.4-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "003ee63fa819e199c6a2ba9728012cb8",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 25708,
                "upload_time": "2019-04-09T11:38:55",
                "url": "https://files.pythonhosted.org/packages/ba/66/caab7e86963e70573e70d2b9d6c5d2a3b64f0210336c60ad60d6eaa8e925/scrapy_test-0.4-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "5c4d0506f1ab7468e37e395a9630b4c0",
                    "sha256": "3c9528d545d035f5a9baae37c437bb06e33107ee4442d4e2af3f146adffaf235"
                },
                "downloads": -1,
                "filename": "scrapy-test-0.4.tar.gz",
                "has_sig": false,
                "md5_digest": "5c4d0506f1ab7468e37e395a9630b4c0",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 10646,
                "upload_time": "2019-04-09T11:39:00",
                "url": "https://files.pythonhosted.org/packages/59/1a/76098f9b9be4558716fb6beada444af0d99b21b1687f904f8ab9931ed9c7/scrapy-test-0.4.tar.gz"
            }
        ],
        "0.5": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "8e22c3b10f3d9092f4808efbc9335688",
                    "sha256": "6cf5970f4ae7c797cc922c6a7c0557f1e5a00b060b180a2104b5e9d1bebc6615"
                },
                "downloads": -1,
                "filename": "scrapy_test-0.5-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "8e22c3b10f3d9092f4808efbc9335688",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 26275,
                "upload_time": "2019-04-12T11:18:45",
                "url": "https://files.pythonhosted.org/packages/34/cc/774c9b27c579ffb9a6243203362d201e46910811caf9e9ff05dad3084b22/scrapy_test-0.5-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "3d8e3254ed87ef0df8662b3d1a871e01",
                    "sha256": "104ae90d43bf7ecd598a9bb3f9a63e7a1754c976437f69521ca7c650fa24edef"
                },
                "downloads": -1,
                "filename": "scrapy-test-0.5.tar.gz",
                "has_sig": false,
                "md5_digest": "3d8e3254ed87ef0df8662b3d1a871e01",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 11163,
                "upload_time": "2019-04-12T11:18:49",
                "url": "https://files.pythonhosted.org/packages/e7/1f/498abaeaafdb1d3012fff05a7ba94bf647718a8835317b8af8567cac87e9/scrapy-test-0.5.tar.gz"
            }
        ],
        "0.5.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "d76f74475d6c310d87cd84dbe4d720e4",
                    "sha256": "c2c4ff36992cc0fc81e89701182e2973e935ad334778d047a31f4779337ad337"
                },
                "downloads": -1,
                "filename": "scrapy_test-0.5.1-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "d76f74475d6c310d87cd84dbe4d720e4",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 26218,
                "upload_time": "2019-04-18T12:42:41",
                "url": "https://files.pythonhosted.org/packages/0e/c7/4afd118c8f4315812e991a7a09fe6c6667e39bb3f6427ff3f507fa0ccc5d/scrapy_test-0.5.1-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "8bbb70625a601e3ca1e9bf0728bd1457",
                    "sha256": "efabb87689a0fb6d2ec000291f0a681353235996ed83e74ca9b5e1d6b27d37d8"
                },
                "downloads": -1,
                "filename": "scrapy-test-0.5.1.tar.gz",
                "has_sig": false,
                "md5_digest": "8bbb70625a601e3ca1e9bf0728bd1457",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 11089,
                "upload_time": "2019-04-18T12:42:46",
                "url": "https://files.pythonhosted.org/packages/85/ff/3585c9a7f5eea1aa4a449675d746ac43474a1ff7abeaba38174f5a2abd25/scrapy-test-0.5.1.tar.gz"
            }
        ],
        "0.6": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "7d0f7277e01b1968b78f553350428035",
                    "sha256": "3e37fd9a2095cefa1b406ae4254fe205ca60e5312348837189bea1a8f91d23a9"
                },
                "downloads": -1,
                "filename": "scrapy_test-0.6-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "7d0f7277e01b1968b78f553350428035",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 26617,
                "upload_time": "2019-04-19T07:21:03",
                "url": "https://files.pythonhosted.org/packages/8f/46/0f6c833a9eb539b16bdf4b5e5f62c7dcc7ba463d29412d3b1c7377380637/scrapy_test-0.6-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "f423d6da2b010881bbdbb05b1d813b24",
                    "sha256": "3ffeaaa1b92d5b5659ac763624a1132bd87ab54487132b51f87a2074e8d94f8b"
                },
                "downloads": -1,
                "filename": "scrapy-test-0.6.tar.gz",
                "has_sig": false,
                "md5_digest": "f423d6da2b010881bbdbb05b1d813b24",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 11484,
                "upload_time": "2019-04-19T07:21:10",
                "url": "https://files.pythonhosted.org/packages/b2/17/46ec2666b8fe92398bad60ec090e818bb1556198556da21813cefb94aa0b/scrapy-test-0.6.tar.gz"
            }
        ],
        "0.6.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "5ba6bca5a30e3080031d2180947850f7",
                    "sha256": "cc48795387a9ecb778b97884b0d42ca3b56a54dbe801af826fe3a0771eecfe88"
                },
                "downloads": -1,
                "filename": "scrapy_test-0.6.1-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "5ba6bca5a30e3080031d2180947850f7",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 26736,
                "upload_time": "2019-06-27T09:14:05",
                "url": "https://files.pythonhosted.org/packages/bd/f4/552a33a4e3f1dd3513dac94df4c39bcdcc2c787b1dd1970db5581f5ff7c0/scrapy_test-0.6.1-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "d28a69e2870fdeffe92f5dd27c210455",
                    "sha256": "aa98ba7b4ef2cfb94de68cbb0e2ceec9e74a9071878135345ae932261fbaf496"
                },
                "downloads": -1,
                "filename": "scrapy-test-0.6.1.tar.gz",
                "has_sig": false,
                "md5_digest": "d28a69e2870fdeffe92f5dd27c210455",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 11574,
                "upload_time": "2019-06-27T09:14:10",
                "url": "https://files.pythonhosted.org/packages/a6/fa/c43b4286e8c4aa7f8cecb37d7d5d665de4b0908e405b9ef71591e944dd49/scrapy-test-0.6.1.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "5ba6bca5a30e3080031d2180947850f7",
                "sha256": "cc48795387a9ecb778b97884b0d42ca3b56a54dbe801af826fe3a0771eecfe88"
            },
            "downloads": -1,
            "filename": "scrapy_test-0.6.1-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "5ba6bca5a30e3080031d2180947850f7",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": null,
            "size": 26736,
            "upload_time": "2019-06-27T09:14:05",
            "url": "https://files.pythonhosted.org/packages/bd/f4/552a33a4e3f1dd3513dac94df4c39bcdcc2c787b1dd1970db5581f5ff7c0/scrapy_test-0.6.1-py3-none-any.whl"
        },
        {
            "comment_text": "",
            "digests": {
                "md5": "d28a69e2870fdeffe92f5dd27c210455",
                "sha256": "aa98ba7b4ef2cfb94de68cbb0e2ceec9e74a9071878135345ae932261fbaf496"
            },
            "downloads": -1,
            "filename": "scrapy-test-0.6.1.tar.gz",
            "has_sig": false,
            "md5_digest": "d28a69e2870fdeffe92f5dd27c210455",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 11574,
            "upload_time": "2019-06-27T09:14:10",
            "url": "https://files.pythonhosted.org/packages/a6/fa/c43b4286e8c4aa7f8cecb37d7d5d665de4b0908e405b9ef71591e944dd49/scrapy-test-0.6.1.tar.gz"
        }
    ]
}