{ "info": { "author": "Endurant Devs", "author_email": "info@endurantdevs.com", "bugtrack_url": null, "classifiers": [ "Development Status :: 4 - Beta", "Environment :: Console", "Intended Audience :: Developers", "Intended Audience :: System Administrators", "License :: OSI Approved :: MIT License", "Programming Language :: Python :: 2", "Programming Language :: Python :: 2.7", "Programming Language :: Python :: 3", "Programming Language :: Python :: 3.5", "Programming Language :: Python :: 3.6", "Programming Language :: Python :: 3.7" ], "description": "# BotStat\n[![Build Status](https://img.shields.io/travis/EndurantDevs/botstat-seo.svg?logo=travis)](https://travis-ci.org/EndurantDevs/botstat-seo) [![Latest Version](https://pypip.in/version/botstat/badge.svg)](https://pypi.python.org/pypi/botstat/) [![Python Versions](https://img.shields.io/pypi/pyversions/botstat.svg)](https://github.com/EndurantDevs/botstat-seo/blob/master/setup.py) [![Tests Coverage](https://img.shields.io/codecov/c/github/EndurantDevs/botstat-seo/master.svg)](https://codecov.io/gh/EndurantDevs/botstat-seo) [![Downloads](http://pepy.tech/badge/botstat)](http://pepy.tech/project/botstat)\n\nBotStat is a small utility to monitor the crawl rate of your website by search engine bots.\nOnce you run it, it sends bots crawl rate statistics in CSV format to your email.\n\nAlso, it helps to detect the source of possible problems. Crawl rate is dependent on many parameters. We try to give a picture of your site productivity for different search engines: Google, Bing, Yahoo, Baidu, Yandex, Sogou, and others.\n\nFor example, according to Google's Webmaster Blog, if your website responds quickly for a while, the crawl limit goes up, meaning more connections can be used to crawl. If the site slows down or responds with server errors, the limit goes down, and Googlebot crawls less. Other search engines have similar logic in most cases.\n\n### Installing\n\nIt is easy to do from `pip`\n\n```\npip install botstat\n```\n\nor from sources\n\n```\ngit clone git@github.com:EndurantDevs/botstat-seo.git\ncd botstat-seo\npython setup.py install\n```\n\nIf you want to confirm that install was successful, please check for the `botstat` command line utility.\n\nUsually this tool is used with `cron`. To go the same way, please add your configuration and configure your `crontab`.\n\n## Running the tests\n\nTo be sure everything is fine before installation from sources, just run:\n```\npython setup.py test\n```\nOr\n```\npytest tests/\n```\n\n## Usage\n\nIf you have config at ~/.botstat or /etc/botstat.conf you can just do\n```\nbotstat\n```\nor if you have config on custom path\n```\nbotstat -c /path/to/your/config \n```\nor if you like to provide all params from command line\n```\nbotstat --access-log access.log --debug --log-format '$remote_addr $host $remote_user [$time_local] \"$request\" $status $body_bytes_sent \"$http_referer\" \"$http_user_agent\" $request_time -$http_x_forwarded_for-' --smtp-port 10025 --mail-to \"you@gmail.com\" --mail-from \"root@localhost\"\n```\n\n## Help\n\n```\n$ botstat --help\nusage: botstat [-h] [-c MY_CONFIG] [--verbose] [--debug]\n [--log-format LOG_FORMAT] [--nginx-config NGINX_CONFIG]\n [--access-log ACCESS_LOG] [--day-start DAY_START]\n [--date-start DATE_START] [--mail-to MAIL_TO]\n [--mail-from MAIL_FROM] [--mail-subject MAIL_SUBJECT]\n [--smtp-host SMTP_HOST] [--date-format DATE_FORMAT]\n [--smtp-port SMTP_PORT] [--server-type {nginx,apache}]\n [--xlsx-report]\n\nParse web server logs and make bots statistic Args that start with '--' (eg.\n--verbose) can also be set in a config file (/etc/botstat.conf or ~/.botstat\nor specified via -c). Config file syntax allows: key=value, flag=true,\nstuff=[a,b,c] (for details, see syntax at https://goo.gl/R74nmi). If an arg is\nspecified in more than one place, then commandline values override config file\nvalues which override defaults.\n\noptional arguments:\n -h, --help show this help message and exit\n -c MY_CONFIG, --my-config MY_CONFIG\n config file path\n --verbose Verbose output\n --debug Enable debug mode\n --log-format LOG_FORMAT\n Web server log format like 'log_format' in nginx.conf.\n Accept 'combined', 'common' or format string field\n names with $\n --nginx-config NGINX_CONFIG\n Nginx config file name with path\n --access-log ACCESS_LOG\n Access log file name. If not specify used stdin.\n --day-start DAY_START\n Days from the beginning of today, all older records\n skipped\n --date-start DATE_START\n Start date for parsing log, all older records skipped\n --mail-to MAIL_TO Email address to send report\n --mail-from MAIL_FROM\n 'Email FROM' address\n --mail-subject MAIL_SUBJECT\n Report email subject\n --smtp-host SMTP_HOST\n SMTP server host name or ip adddress\n --date-format DATE_FORMAT\n A format string for a date/time field in 'time_local'\n log field.To see the full set of format codes\n supported on your platform, consult the strftime(3)\n documentation.\n --smtp-port SMTP_PORT\n SMTP server port\n --server-type {nginx,apache}\n Web server type, support nginx and apache (default:\n nginx)\n --xlsx-report Report in excel format (it's required for xlsxwriter\n module - run \"pip install xlsxwriter\" to install)\n```\n\n## Built With\n\n* [ConfigArgParse](https://github.com/bw2/ConfigArgParse) - A drop-in replacement for argparse that allows options to also be set via config files and/or environment variables\n* [pytest](https://docs.pytest.org/en/latest/) - Framework makes it easy to write small tests, yet scales to support complex functional testing for applications and libraries\n* [apache-log-parser](https://github.com/rory/apache-log-parser) - Parses log lines from an apache log\n\n## Authors\n[\"Endurant](https://www.EndurantDevs.com)\n\n## License\n\nThis project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details\n\n\n", "description_content_type": "text/markdown", "docs_url": null, "download_url": "", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "https://www.endurantdevs.com", "keywords": "cli apache nginx system", "license": "MIT", "maintainer": "", "maintainer_email": "", "name": "botstat", "package_url": "https://pypi.org/project/botstat/", "platform": "", "project_url": "https://pypi.org/project/botstat/", "project_urls": { "Homepage": "https://www.endurantdevs.com" }, "release_url": "https://pypi.org/project/botstat/0.1.2/", "requires_dist": [ "pyparsing", "dateutils", "ua-parser", "user-agents", "apache-log-parser", "ConfigArgParse" ], "requires_python": "", "summary": "Search engine analyser for nginx/apache web servers", "version": "0.1.2" }, "last_serial": 4331655, "releases": { "0.0.1": [ { "comment_text": "", "digests": { "md5": "77baaf1963385b2167924dbf789cecd2", "sha256": "7652568acc791a9d5626dff5a1d719a01dd988174b7ae329c0574dd301c9d1ec" }, "downloads": -1, "filename": "botstat-0.0.1-py2.7.egg", "has_sig": false, "md5_digest": "77baaf1963385b2167924dbf789cecd2", "packagetype": "bdist_egg", "python_version": "2.7", "requires_python": null, "size": 17259, "upload_time": "2018-07-06T15:44:34", "url": "https://files.pythonhosted.org/packages/a5/d6/6dcd6cb49df9ff0389194346021eaefc7f6130603a0e12674b0bb4178fee/botstat-0.0.1-py2.7.egg" }, { "comment_text": "", "digests": { "md5": "76d67829cef916330c58b3351d077dad", "sha256": "665fbc91ca194340bb2a8d98bc6dae61b6322068ade219c89e31acb318744890" }, "downloads": -1, "filename": "botstat-0.0.1-py2.py3-none-any.whl", "has_sig": false, "md5_digest": "76d67829cef916330c58b3351d077dad", "packagetype": "bdist_wheel", "python_version": "py2.py3", "requires_python": null, "size": 9467, "upload_time": "2018-07-06T15:44:31", "url": "https://files.pythonhosted.org/packages/02/f5/3b5e6447e77a3c7e46ec410078bee0d9c9a792c3413b1c368124965a403b/botstat-0.0.1-py2.py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "214a742917574c085405495015b27d0c", "sha256": "e179cb70cbfd6036b3235ee1eb68579d725e701f61c0acc21c6b263e2a9a14bc" }, "downloads": -1, "filename": "botstat-0.0.1.tar.gz", "has_sig": false, "md5_digest": "214a742917574c085405495015b27d0c", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 8504, "upload_time": "2018-07-06T15:44:35", "url": "https://files.pythonhosted.org/packages/99/92/ac4971ecf583c78f861c9f310e0d839efa591e6303c3f5dc0d2009e8f4a0/botstat-0.0.1.tar.gz" } ], "0.0.2": [ { "comment_text": "", "digests": { "md5": "f1adcd96af68b9653a4621fafb5c88a4", "sha256": "c95981484db6c02bdfb947cef37f39901244d63e72b61695ddba99b07ba03be8" }, "downloads": -1, "filename": "botstat-0.0.2.macosx-10.13-intel.tar.gz", "has_sig": false, "md5_digest": "f1adcd96af68b9653a4621fafb5c88a4", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 15139, "upload_time": "2018-07-06T15:57:27", "url": "https://files.pythonhosted.org/packages/04/fa/166cb373fe54d42bce6776dab30d46d3c90b9fff60e3dce8b3ef55383b8c/botstat-0.0.2.macosx-10.13-intel.tar.gz" }, { "comment_text": "", "digests": { "md5": "adb9a708cbc981b5f48de7bbaaffeb53", "sha256": "2f0e62446f010c442aef56f6b809a9dd3a03187d386836b3d217aa5633d5bdfa" }, "downloads": -1, "filename": "botstat-0.0.2-py2.py3-none-any.whl", "has_sig": false, "md5_digest": "adb9a708cbc981b5f48de7bbaaffeb53", "packagetype": "bdist_wheel", "python_version": "py2.py3", "requires_python": null, "size": 9358, "upload_time": "2018-07-06T15:57:26", "url": "https://files.pythonhosted.org/packages/8b/58/6901b5efee2f5f3949d0457e8252f6b95f6654233676154b20e237e20f5d/botstat-0.0.2-py2.py3-none-any.whl" } ], "0.0.3": [ { "comment_text": "", "digests": { "md5": "50e018b2bc4ace6c3d780b384908bf80", "sha256": "74735f7c5d3e4e390c7801a00aa1e7e8aeb735cf3492266e2dc1b72cfedb2a0e" }, "downloads": -1, "filename": "botstat-0.0.3-py2.py3-none-any.whl", "has_sig": false, "md5_digest": "50e018b2bc4ace6c3d780b384908bf80", "packagetype": "bdist_wheel", "python_version": "py2.py3", "requires_python": null, "size": 10130, "upload_time": "2018-08-31T09:36:05", "url": "https://files.pythonhosted.org/packages/ec/6b/9ea4209a6189743f86a4fc0d6d73cdeeb9154f4b59d8643e9a25fa54b88a/botstat-0.0.3-py2.py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "ae10319e78b3246770302df750a057df", "sha256": "d915700580d28176c11415eec0ed50058db3b28c53794eb35611fb180106c325" }, "downloads": -1, "filename": "botstat-0.0.3.tar.gz", "has_sig": false, "md5_digest": "ae10319e78b3246770302df750a057df", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 9455, "upload_time": "2018-08-31T09:36:41", "url": "https://files.pythonhosted.org/packages/99/6f/46e29e5e112891981b009a8cf71bc16ee9648c6845fa68c49a7e463667c7/botstat-0.0.3.tar.gz" } ], "0.0.4": [ { "comment_text": "", "digests": { "md5": "2ea39f150787fc2bc77f72af60d627f5", "sha256": "0ef757b4e6b9d16238fa6f4f8a45121d5c40dce1822f3b76e7b360e155880adf" }, "downloads": -1, "filename": "botstat-0.0.4-py2.py3-none-any.whl", "has_sig": false, "md5_digest": "2ea39f150787fc2bc77f72af60d627f5", "packagetype": "bdist_wheel", "python_version": "py2.py3", "requires_python": null, "size": 10129, "upload_time": "2018-08-31T16:20:23", "url": "https://files.pythonhosted.org/packages/20/dd/aed0e874092763d4994c0cbbe7da87a26b9a1ba7cc79ce36ba1e5eaa3190/botstat-0.0.4-py2.py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "13c40f78f711457e9ed1976335fc55d8", "sha256": "8baf255f122194e5be42d17052739af7ac1a75ab5104a14b1fac52b54b123ce0" }, "downloads": -1, "filename": "botstat-0.0.4.tar.gz", "has_sig": false, "md5_digest": "13c40f78f711457e9ed1976335fc55d8", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 10073, "upload_time": "2018-08-31T16:20:24", "url": "https://files.pythonhosted.org/packages/6a/6a/8cf289bc8fc9cf17cb3f300635222fdba8aaefe82886646d6026ab533f19/botstat-0.0.4.tar.gz" } ], "0.0.5": [ { "comment_text": "", "digests": { "md5": "7bbf7c5d451336798870d2403f5fc94e", "sha256": "cfe6bf19af9a5c4df69731e09b01f8872c8cba5500a946bd9cf67e26f85be73e" }, "downloads": -1, "filename": "botstat-0.0.5-py2.py3-none-any.whl", "has_sig": false, "md5_digest": "7bbf7c5d451336798870d2403f5fc94e", "packagetype": "bdist_wheel", "python_version": "py2.py3", "requires_python": null, "size": 10257, "upload_time": "2018-09-02T11:07:08", "url": "https://files.pythonhosted.org/packages/3f/7a/e8787ba41d5827ffa8f283bfb9fb67ca624a361db3ad21bdbbb5fd7a33df/botstat-0.0.5-py2.py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "1ee3e40269f4a28d76f062d44d5c258d", "sha256": "3d77ffb3cfcc020de00e72f1ff6f74e556d383fe3806aa5002247c553c3da3c3" }, "downloads": -1, "filename": "botstat-0.0.5.tar.gz", "has_sig": false, "md5_digest": "1ee3e40269f4a28d76f062d44d5c258d", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 9310, "upload_time": "2018-09-02T11:07:09", "url": "https://files.pythonhosted.org/packages/86/9b/80157ba4f4db5192d82b22badb5c44aa46e2b6c6913907c80a395b482ba9/botstat-0.0.5.tar.gz" } ], "0.0.6": [ { "comment_text": "", "digests": { "md5": "97b1705c9bde6b59debd356b5cb5ed65", "sha256": "869864480b8eadb168572bf2e922281572ba6b710cc7abd2ce161fed867ee566" }, "downloads": -1, "filename": "botstat-0.0.6-py2.py3-none-any.whl", "has_sig": false, "md5_digest": "97b1705c9bde6b59debd356b5cb5ed65", "packagetype": "bdist_wheel", "python_version": "py2.py3", "requires_python": null, "size": 10646, "upload_time": "2018-09-03T11:22:04", "url": "https://files.pythonhosted.org/packages/36/d6/4281f714511d0f22463e5c3699e1069c7f4e9395f3e237ca212a0c1b89de/botstat-0.0.6-py2.py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "085571e9fca66bfac98dbd108ce995b3", "sha256": "029518ebb5c3331f8930c5e090c691e8b1988878f82ffed0589ba7e30aa46ab7" }, "downloads": -1, "filename": "botstat-0.0.6.tar.gz", "has_sig": false, "md5_digest": "085571e9fca66bfac98dbd108ce995b3", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 9597, "upload_time": "2018-09-03T11:22:06", "url": "https://files.pythonhosted.org/packages/e6/24/8eab9696ae6f4e1da30dc6c1b9eef480207e0c11b06b3cb174028648ebb4/botstat-0.0.6.tar.gz" } ], "0.1": [ { "comment_text": "", "digests": { "md5": "3e5ed3ad89d2f46b99185e7c7edc67f3", "sha256": "cc4c04ef6fee28963ce2b25e74ebb7e718522cc69c6d6c88b50374d14f88522d" }, "downloads": -1, "filename": "botstat-0.1-py2.py3-none-any.whl", "has_sig": false, "md5_digest": "3e5ed3ad89d2f46b99185e7c7edc67f3", "packagetype": "bdist_wheel", "python_version": "py2.py3", "requires_python": null, "size": 11218, "upload_time": "2018-09-11T18:53:17", "url": "https://files.pythonhosted.org/packages/e0/b8/4d21c13a3b5596eb9dfaa06e88d52efe8e0c3878f41fa1089dc77f77acea/botstat-0.1-py2.py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "20f388d03c57ccdda746389d691f8b13", "sha256": "90034c520951bc536baf696d84f3f8039a7288b5678b896e52060bdd5f59e7a3" }, "downloads": -1, "filename": "botstat-0.1.tar.gz", "has_sig": false, "md5_digest": "20f388d03c57ccdda746389d691f8b13", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 12181, "upload_time": "2018-09-11T18:53:18", "url": "https://files.pythonhosted.org/packages/85/ec/6f397f7dd9af8555ae22a892ca500f7e5bd328b87ab964ae2fcfe1236b8c/botstat-0.1.tar.gz" } ], "0.1.1": [ { "comment_text": "", "digests": { "md5": "a1aaa02af148a33fa6a82d56f57a7f12", "sha256": "38d759b8c692127904f490c1bde1f0473f2231bbd79a2b2e0818d582b7365cdb" }, "downloads": -1, "filename": "botstat-0.1.1-py2.py3-none-any.whl", "has_sig": false, "md5_digest": "a1aaa02af148a33fa6a82d56f57a7f12", "packagetype": "bdist_wheel", "python_version": "py2.py3", "requires_python": null, "size": 11429, "upload_time": "2018-09-11T19:35:26", "url": "https://files.pythonhosted.org/packages/7e/b9/48bec57bd7ef7ab991adc5f445c964d944331c42cc19bcfc92b29914aab5/botstat-0.1.1-py2.py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "3fe4bfa87516f3d7d5d91ed2b64357ff", "sha256": "6512d4cf216b236885bf6a7b3598fbf14a467b689968cc923b5662d21c0052f4" }, "downloads": -1, "filename": "botstat-0.1.1.tar.gz", "has_sig": false, "md5_digest": "3fe4bfa87516f3d7d5d91ed2b64357ff", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 12440, "upload_time": "2018-09-11T19:35:28", "url": "https://files.pythonhosted.org/packages/67/ee/8526717bed6303b8e5e1a9660aa2bc6856d207d2684f12571064aad5362e/botstat-0.1.1.tar.gz" } ], "0.1.2": [ { "comment_text": "", "digests": { "md5": "e0770df71bcfcbf34b59f657bf66a2ef", "sha256": "458af73673942ba3240bc650faf3d916dfbf06270818e52a86cea6d00ac74334" }, "downloads": -1, "filename": "botstat-0.1.2-py2.py3-none-any.whl", "has_sig": false, "md5_digest": "e0770df71bcfcbf34b59f657bf66a2ef", "packagetype": "bdist_wheel", "python_version": "py2.py3", "requires_python": null, "size": 11463, "upload_time": "2018-10-02T08:17:22", "url": "https://files.pythonhosted.org/packages/34/68/27cf0bd816cb17d5872794946406a759cfb43aa190ab45bf9c8a03f34f2c/botstat-0.1.2-py2.py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "dd82f75622d616c7f9382ec7323a2608", "sha256": "d628da117cda7580cfdb578264f8b309d86dedec01227a8126ad613432f1b3b6" }, "downloads": -1, "filename": "botstat-0.1.2.tar.gz", "has_sig": false, "md5_digest": "dd82f75622d616c7f9382ec7323a2608", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 12491, "upload_time": "2018-10-02T08:17:24", "url": "https://files.pythonhosted.org/packages/13/30/12c0131ecf1058bd363c72d696c0853c18a831d4cc32366e24a0f40e478c/botstat-0.1.2.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "e0770df71bcfcbf34b59f657bf66a2ef", "sha256": "458af73673942ba3240bc650faf3d916dfbf06270818e52a86cea6d00ac74334" }, "downloads": -1, "filename": "botstat-0.1.2-py2.py3-none-any.whl", "has_sig": false, "md5_digest": "e0770df71bcfcbf34b59f657bf66a2ef", "packagetype": "bdist_wheel", "python_version": "py2.py3", "requires_python": null, "size": 11463, "upload_time": "2018-10-02T08:17:22", "url": "https://files.pythonhosted.org/packages/34/68/27cf0bd816cb17d5872794946406a759cfb43aa190ab45bf9c8a03f34f2c/botstat-0.1.2-py2.py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "dd82f75622d616c7f9382ec7323a2608", "sha256": "d628da117cda7580cfdb578264f8b309d86dedec01227a8126ad613432f1b3b6" }, "downloads": -1, "filename": "botstat-0.1.2.tar.gz", "has_sig": false, "md5_digest": "dd82f75622d616c7f9382ec7323a2608", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 12491, "upload_time": "2018-10-02T08:17:24", "url": "https://files.pythonhosted.org/packages/13/30/12c0131ecf1058bd363c72d696c0853c18a831d4cc32366e24a0f40e478c/botstat-0.1.2.tar.gz" } ] }