{
    "info": {
        "author": "my8100",
        "author_email": "my8100@gmail.com",
        "bugtrack_url": null,
        "classifiers": [
            "License :: OSI Approved :: GNU General Public License v3 (GPLv3)",
            "Programming Language :: Python :: 2.7",
            "Programming Language :: Python :: 3.6",
            "Programming Language :: Python :: 3.7"
        ],
        "description": "# LogParser: A tool for parsing Scrapy log files periodically and incrementally, designed for [*ScrapydWeb*](https://github.com/my8100/scrapydweb).\n\n[![PyPI - logparser Version](https://img.shields.io/pypi/v/logparser.svg)](https://pypi.org/project/logparser/)\n[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/logparser.svg)](https://pypi.org/project/logparser/)\n[![CircleCI](https://circleci.com/gh/my8100/logparser/tree/master.svg?style=shield)](https://circleci.com/gh/my8100/logparser/tree/master)\n[![codecov](https://codecov.io/gh/my8100/logparser/branch/master/graph/badge.svg)](https://codecov.io/gh/my8100/logparser)\n[![Coverage Status](https://coveralls.io/repos/github/my8100/logparser/badge.svg?branch=master)](https://coveralls.io/github/my8100/logparser?branch=master)\n[![Downloads - total](https://pepy.tech/badge/logparser)](https://pepy.tech/project/logparser)\n[![GitHub license](https://img.shields.io/github/license/my8100/logparser.svg)](https://github.com/my8100/logparser/blob/master/LICENSE)\n\n\n## Installation\n- Use pip:\n```bash\npip install logparser\n```\nNote that you may need to execute `python -m pip install --upgrade pip` first in order to get the latest version of logparser, or download the tar.gz file from https://pypi.org/project/logparser/#files and get it installed via `pip install logparser-x.x.x.tar.gz`\n\n- Use git:\n```bash\npip install --upgrade git+https://github.com/my8100/logparser.git\n```\nOr:\n```bash\ngit clone https://github.com/my8100/logparser.git\ncd logparser\npython setup.py install\n```\n\n## Usage\n### To use in Python\n<details>\n<summary>View codes</summary>\n\n```python\nIn [1]: from logparser import parse\n\nIn [2]: log = \"\"\"2018-10-23 18:28:34 [scrapy.utils.log] INFO: Scrapy 1.5.0 started (bot: demo)\n   ...: 2018-10-23 18:29:41 [scrapy.statscollectors] INFO: Dumping Scrapy stats:\n   ...: {'downloader/exception_count': 3,\n   ...:  'downloader/exception_type_count/twisted.internet.error.TCPTimedOutError': 3,\n   ...:  'downloader/request_bytes': 1336,\n   ...:  'downloader/request_count': 7,\n   ...:  'downloader/request_method_count/GET': 7,\n   ...:  'downloader/response_bytes': 1669,\n   ...:  'downloader/response_count': 4,\n   ...:  'downloader/response_status_count/200': 2,\n   ...:  'downloader/response_status_count/302': 1,\n   ...:  'downloader/response_status_count/404': 1,\n   ...:  'dupefilter/filtered': 1,\n   ...:  'finish_reason': 'finished',\n   ...:  'finish_time': datetime.datetime(2018, 10, 23, 10, 29, 41, 174719),\n   ...:  'httperror/response_ignored_count': 1,\n   ...:  'httperror/response_ignored_status_count/404': 1,\n   ...:  'item_scraped_count': 2,\n   ...:  'log_count/CRITICAL': 5,\n   ...:  'log_count/DEBUG': 14,\n   ...:  'log_count/ERROR': 5,\n   ...:  'log_count/INFO': 75,\n   ...:  'log_count/WARNING': 3,\n   ...:  'offsite/domains': 1,\n   ...:  'offsite/filtered': 1,\n   ...:  'request_depth_max': 1,\n   ...:  'response_received_count': 3,\n   ...:  'retry/count': 2,\n   ...:  'retry/max_reached': 1,\n   ...:  'retry/reason_count/twisted.internet.error.TCPTimedOutError': 2,\n   ...:  'scheduler/dequeued': 7,\n   ...:  'scheduler/dequeued/memory': 7,\n   ...:  'scheduler/enqueued': 7,\n   ...:  'scheduler/enqueued/memory': 7,\n   ...:  'start_time': datetime.datetime(2018, 10, 23, 10, 28, 35, 70938)}\n   ...: 2018-10-23 18:29:42 [scrapy.core.engine] INFO: Spider closed (finished)\"\"\"\n\nIn [3]: odict = parse(log, headlines=1, taillines=1)\n\nIn [4]: odict\nOut[4]:\nOrderedDict([('head',\n              '2018-10-23 18:28:34 [scrapy.utils.log] INFO: Scrapy 1.5.0 started (bot: demo)'),\n             ('tail',\n              '2018-10-23 18:29:42 [scrapy.core.engine] INFO: Spider closed (finished)'),\n             ('first_log_time', '2018-10-23 18:28:34'),\n             ('latest_log_time', '2018-10-23 18:29:42'),\n             ('runtime', '0:01:08'),\n             ('first_log_timestamp', 1540290514),\n             ('latest_log_timestamp', 1540290582),\n             ('datas', []),\n             ('pages', 3),\n             ('items', 2),\n             ('latest_matches',\n              {'telnet_console': '',\n               'resuming_crawl': '',\n               'latest_offsite': '',\n               'latest_duplicate': '',\n               'latest_crawl': '',\n               'latest_scrape': '',\n               'latest_item': '',\n               'latest_stat': ''}),\n             ('latest_crawl_timestamp', 0),\n             ('latest_scrape_timestamp', 0),\n             ('log_categories',\n              {'critical_logs': {'count': 5, 'details': []},\n               'error_logs': {'count': 5, 'details': []},\n               'warning_logs': {'count': 3, 'details': []},\n               'redirect_logs': {'count': 1, 'details': []},\n               'retry_logs': {'count': 2, 'details': []},\n               'ignore_logs': {'count': 1, 'details': []}}),\n             ('shutdown_reason', 'N/A'),\n             ('finish_reason', 'finished'),\n             ('crawler_stats',\n              OrderedDict([('source', 'log'),\n                           ('last_update_time', '2018-10-23 18:29:41'),\n                           ('last_update_timestamp', 1540290581),\n                           ('downloader/exception_count', 3),\n                           ('downloader/exception_type_count/twisted.internet.error.TCPTimedOutError',\n                            3),\n                           ('downloader/request_bytes', 1336),\n                           ('downloader/request_count', 7),\n                           ('downloader/request_method_count/GET', 7),\n                           ('downloader/response_bytes', 1669),\n                           ('downloader/response_count', 4),\n                           ('downloader/response_status_count/200', 2),\n                           ('downloader/response_status_count/302', 1),\n                           ('downloader/response_status_count/404', 1),\n                           ('dupefilter/filtered', 1),\n                           ('finish_reason', 'finished'),\n                           ('finish_time',\n                            'datetime.datetime(2018, 10, 23, 10, 29, 41, 174719)'),\n                           ('httperror/response_ignored_count', 1),\n                           ('httperror/response_ignored_status_count/404', 1),\n                           ('item_scraped_count', 2),\n                           ('log_count/CRITICAL', 5),\n                           ('log_count/DEBUG', 14),\n                           ('log_count/ERROR', 5),\n                           ('log_count/INFO', 75),\n                           ('log_count/WARNING', 3),\n                           ('offsite/domains', 1),\n                           ('offsite/filtered', 1),\n                           ('request_depth_max', 1),\n                           ('response_received_count', 3),\n                           ('retry/count', 2),\n                           ('retry/max_reached', 1),\n                           ('retry/reason_count/twisted.internet.error.TCPTimedOutError',\n                            2),\n                           ('scheduler/dequeued', 7),\n                           ('scheduler/dequeued/memory', 7),\n                           ('scheduler/enqueued', 7),\n                           ('scheduler/enqueued/memory', 7),\n                           ('start_time',\n                            'datetime.datetime(2018, 10, 23, 10, 28, 35, 70938)')])),\n             ('last_update_time', '2019-03-08 16:53:50'),\n             ('last_update_timestamp', 1552035230),\n             ('logparser_version', '0.8.1')])\n\nIn [5]: odict['runtime']\nOut[5]: '0:01:08'\n\nIn [6]: odict['pages']\nOut[6]: 3\n\nIn [7]: odict['items']\nOut[7]: 2\n\nIn [8]: odict['finish_reason']\nOut[8]: 'finished'\n```\n\n</details>\n\n### To run as a service\n1. **Make sure that [*Scrapyd*](https://github.com/scrapy/scrapyd) has been installed and started on the current host.**\n2. Start ***LogParser*** via command `logparser`\n3. Visit http://127.0.0.1:6800/logs/stats.json **(Assuming the Scrapyd service runs on port 6800.)**\n4. Visit http://127.0.0.1:6800/logs/projectname/spidername/jobid.json to get stats of a job in details.\n\n### To work with *ScrapydWeb* for visualization\nCheck out https://github.com/my8100/scrapydweb for more info.\n\n![stats](https://raw.githubusercontent.com/my8100/files/master/scrapydweb/screenshots/stats.gif)\n\n\n",
        "description_content_type": "text/markdown",
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://github.com/my8100/logparser",
        "keywords": "",
        "license": "GNU General Public License v3.0",
        "maintainer": "",
        "maintainer_email": "",
        "name": "logparser",
        "package_url": "https://pypi.org/project/logparser/",
        "platform": "",
        "project_url": "https://pypi.org/project/logparser/",
        "project_urls": {
            "Homepage": "https://github.com/my8100/logparser"
        },
        "release_url": "https://pypi.org/project/logparser/0.8.2/",
        "requires_dist": [
            "pexpect (>=4.7.0)",
            "six (>=1.12.0)"
        ],
        "requires_python": ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*",
        "summary": "A tool for parsing Scrapy log files periodically and incrementally, designed for ScrapydWeb.",
        "version": "0.8.2"
    },
    "last_serial": 5630865,
    "releases": {
        "0.8.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "5875b553e01e39ef843b72a0a8c3fe97",
                    "sha256": "620f5fd7755baeb94e59de973b96af1e00aadf17ebe6b39441def77c88f54c57"
                },
                "downloads": -1,
                "filename": "logparser-0.8.0-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "5875b553e01e39ef843b72a0a8c3fe97",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*",
                "size": 30314,
                "upload_time": "2019-01-20T16:29:55",
                "url": "https://files.pythonhosted.org/packages/da/aa/302d0d52f57fb53323375360ef9805e6c25b7a293050ddb04e095c6e9d13/logparser-0.8.0-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "ff95d41b77b1fe070b29d66e94fa17f0",
                    "sha256": "840bcc27a5c9d10bb5a0e5f322e0b19f17fd6d4e9af70c984f88b87fe6c80215"
                },
                "downloads": -1,
                "filename": "logparser-0.8.0.tar.gz",
                "has_sig": false,
                "md5_digest": "ff95d41b77b1fe070b29d66e94fa17f0",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*",
                "size": 16738,
                "upload_time": "2019-01-20T16:29:59",
                "url": "https://files.pythonhosted.org/packages/f3/b2/ef1157ceaa534b174c0ca5e8d5cd2098b39ed83d299870e497e098a6ccbf/logparser-0.8.0.tar.gz"
            }
        ],
        "0.8.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "d48431484021f7b3fd274e8779fcbf52",
                    "sha256": "b427adddb893df09fad301c905cd7bea8ff5a7ebe10dd8083ed292ad757f2943"
                },
                "downloads": -1,
                "filename": "logparser-0.8.1-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "d48431484021f7b3fd274e8779fcbf52",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*",
                "size": 34343,
                "upload_time": "2019-03-12T09:56:01",
                "url": "https://files.pythonhosted.org/packages/21/84/f81d5af38b742814802773032997c5e08556f2cd4d0f215b4c3c873e576a/logparser-0.8.1-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "e6f7aa916dc7583089ff1f651c797f10",
                    "sha256": "71e40609614a860b5b8187f7aab67419136c9d92832dbbd346ad3aba1c36b4c4"
                },
                "downloads": -1,
                "filename": "logparser-0.8.1.tar.gz",
                "has_sig": false,
                "md5_digest": "e6f7aa916dc7583089ff1f651c797f10",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*",
                "size": 20818,
                "upload_time": "2019-03-12T09:58:39",
                "url": "https://files.pythonhosted.org/packages/3f/08/420f7447f106d01586b0f22e3a4c3b5f845d78e050ac4a907d7e4525f602/logparser-0.8.1.tar.gz"
            }
        ],
        "0.8.2": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "ef00fab72ff523182c5d58e0096a9cb0",
                    "sha256": "9d34701ff8ae22596a987eb10defd81c86003860a9289e74f72ee185f7c01db8"
                },
                "downloads": -1,
                "filename": "logparser-0.8.2-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "ef00fab72ff523182c5d58e0096a9cb0",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*",
                "size": 37312,
                "upload_time": "2019-08-04T15:19:19",
                "url": "https://files.pythonhosted.org/packages/d6/7e/528828509144799c08e903e0f49cbc037cec4128286eaafa19e312263651/logparser-0.8.2-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "11bed4a1de8927840325600137aa6be4",
                    "sha256": "b1720067ba267f13a4c1b08cfdcad5b9ed7e0db95817a46a946de83a5b18f227"
                },
                "downloads": -1,
                "filename": "logparser-0.8.2.tar.gz",
                "has_sig": false,
                "md5_digest": "11bed4a1de8927840325600137aa6be4",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*",
                "size": 23568,
                "upload_time": "2019-08-04T15:19:21",
                "url": "https://files.pythonhosted.org/packages/0e/e1/c2679116e29ff09cb5018635297fbe59ab371eae1d9b913b0e9e22a6e974/logparser-0.8.2.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "ef00fab72ff523182c5d58e0096a9cb0",
                "sha256": "9d34701ff8ae22596a987eb10defd81c86003860a9289e74f72ee185f7c01db8"
            },
            "downloads": -1,
            "filename": "logparser-0.8.2-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "ef00fab72ff523182c5d58e0096a9cb0",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*",
            "size": 37312,
            "upload_time": "2019-08-04T15:19:19",
            "url": "https://files.pythonhosted.org/packages/d6/7e/528828509144799c08e903e0f49cbc037cec4128286eaafa19e312263651/logparser-0.8.2-py3-none-any.whl"
        },
        {
            "comment_text": "",
            "digests": {
                "md5": "11bed4a1de8927840325600137aa6be4",
                "sha256": "b1720067ba267f13a4c1b08cfdcad5b9ed7e0db95817a46a946de83a5b18f227"
            },
            "downloads": -1,
            "filename": "logparser-0.8.2.tar.gz",
            "has_sig": false,
            "md5_digest": "11bed4a1de8927840325600137aa6be4",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": ">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*",
            "size": 23568,
            "upload_time": "2019-08-04T15:19:21",
            "url": "https://files.pythonhosted.org/packages/0e/e1/c2679116e29ff09cb5018635297fbe59ab371eae1d9b913b0e9e22a6e974/logparser-0.8.2.tar.gz"
        }
    ]
}