{ "info": { "author": "Brian Wylie", "author_email": "briford@supercowpowers.com", "bugtrack_url": null, "classifiers": [ "Development Status :: 2 - Pre-Alpha", "Intended Audience :: Developers", "License :: OSI Approved :: Apache Software License", "Natural Language :: English", "Programming Language :: Python :: 2.7", "Programming Language :: Python :: 3.5", "Programming Language :: Python :: 3.6", "Programming Language :: Python :: Implementation :: CPython", "Programming Language :: Python :: Implementation :: PyPy" ], "description": "## Bro Analysis Tools (BAT) [![travis](https://travis-ci.org/SuperCowPowers/bat.svg?branch=master)](https://travis-ci.org/SuperCowPowers/bat) [![Coverage Status](https://coveralls.io/repos/github/SuperCowPowers/bat/badge.svg?branch=master)](https://coveralls.io/github/SuperCowPowers/bat?branch=master) [![supported-versions](https://img.shields.io/pypi/pyversions/bat.svg)](https://pypi.python.org/pypi/bat) [![license](https://img.shields.io/badge/License-Apache%202.0-green.svg)](https://choosealicense.com/licenses/apache-2.0)\n\n\nThe BAT Python package supports the processing and analysis of Bro data\nwith Pandas, scikit-learn, and Spark\n\n## BroCon 2017 Presentation\n\nData Analysis, Machine Learning, Bro, and You!\n([Video](https://www.youtube.com/watch?v=pG5lU9CLnIU))\n\n## Why BAT?\n\nBro already has a flexible, powerful scripting language why should I use\nBAT?\n\n**Offloading:** Running complex tasks like statistics, state machines,\nmachine learning, etc.. should be offloaded from Bro so that Bro can\nfocus on the efficient processing of high volume network traffic.\n\n**Data Analysis:** We have a large set of support classes that help\nbridge from raw Bro data to packages like Pandas, scikit-learn, and\nSpark. We also have example notebooks that show step-by-step how to get\nfrom here to there.\n\n### Example: Pull in Bro Logs as Python Dictionaries\n\n```python\n from bat import bro_log_reader\n ...\n # Run the bro reader on a given log file\n reader = bro_log_reader.BroLogReader('dhcp.log')\n for row in reader.readrows():\n pprint(row)\n```\n\n**Output:** Each row is a nice Python Dictionary with timestamps and\ntypes properly converted.\n\n {'assigned_ip': '192.168.84.10',\n 'id.orig_h': '192.168.84.10',\n 'id.orig_p': 68,\n 'id.resp_h': '192.168.84.1',\n 'id.resp_p': 67,\n 'lease_time': datetime.timedelta(49710, 23000),\n 'mac': '00:20:18:eb:ca:54',\n 'trans_id': 495764278,\n 'ts': datetime.datetime(2012, 7, 20, 3, 14, 12, 219654),\n 'uid': 'CJsdG95nCNF1RXuN5'}\n ...\n\n### Example: Bro log to Pandas DataFrame (in one line of code)\n\n```python\nfrom bat.log_to_dataframe import LogToDataFrame\n...\n # Create a Pandas dataframe from a Bro log\n bro_df = LogToDataFrame('/path/to/dns.log')\n\n # Print out the head of the dataframe\n print(bro_df.head())\n```\n\n**Output:** All the Bro log data is in a Pandas DataFrame with proper\ntypes and timestamp as the index\n\n```\n query id.orig_h id.orig_p id.resp_h\nts\n2013-09-15 17:44:27.631940 guyspy.com 192.168.33.10 1030 4.2.2.3\n2013-09-15 17:44:27.696869 www.guyspy.com 192.168.33.10 1030 4.2.2.3\n2013-09-15 17:44:28.060639 devrubn8mli40.cloudfront.net 192.168.33.10 1030 4.2.2.3\n2013-09-15 17:44:28.141795 d31qbv1cthcecs.cloudfront.net 192.168.33.10 1030 4.2.2.3\n2013-09-15 17:44:28.422704 crl.entrust.net 192.168.33.10 1030 4.2.2.3\n```\n\n## More Examples\n\n- Easy ingestion of any Bro Log into Python (dynamic tailing and log\n rotations are handled)\n- Bro Logs to Pandas Dataframes and Scikit-Learn\n- Dynamically monitor files.log and make VirusTotal Queries\n- Dynamically monitor http.log and show 'uncommon' User Agents\n- Running Yara Signatures on Extracted Files\n- Checking x509 Certificates\n- Anomaly Detection\n- See [BAT\n Examples](https://bat-tools.readthedocs.io/en/latest/examples.html)\n for more details.\n\n## Analysis Notebooks\n\nBAT enables the processing, analysis, and machine learning of realtime\ndata coming from Bro.\n\n- Bro to Scikit-Learn: [Bro to\n Scikit](https://nbviewer.jupyter.org/github/SuperCowPowers/bat/blob/master/notebooks/Bro_to_Scikit_Learn.ipynb)\n- Bro to Matplotlib: [Bro to\n Plot](https://nbviewer.jupyter.org/github/SuperCowPowers/bat/blob/master/notebooks/Bro_to_Plot.ipynb)\n- Bro to Parquet to Spark:\n [Bro->Parquet->Spark](https://nbviewer.jupyter.org/github/SuperCowPowers/bat/blob/master/notebooks/Bro_to_Parquet_to_Spark.ipynb)\n- Bro to Kafka to Spark:\n [Bro->Kafka->Spark](https://nbviewer.jupyter.org/github/SuperCowPowers/bat/blob/master/notebooks/Bro_to_Kafka_to_Spark.ipynb)\n- Clustering: Picking K (or not): [Clustering K\n Hyperparameter](https://nbviewer.jupyter.org/github/SuperCowPowers/bat/blob/master/notebooks/Clustering_Picking_K.ipynb)\n- Anomaly Detection Exploration: [Anomaly\n Detection](https://nbviewer.jupyter.org/github/SuperCowPowers/bat/blob/master/notebooks/Anomaly_Detection.ipynb)\n- Risky Domains Stats and Deployment: [Risky\n Domains](https://nbviewer.jupyter.org/github/SuperCowPowers/bat/blob/master/notebooks/Risky_Domains.ipynb)\n\nInstall\n-------\n\n $ pip install bat\n\nDocumentation\n-------------\n\n[bat-tools.readthedocs.org](https://bat-tools.readthedocs.org/)\n\nThanks\n------\n\n- The DummyEncoder is based on Tom Augspurger's great PyData Chicago\n 2016 [Talk](https://youtu.be/KLPtEBokqQ0)\n\n[![SuperCowPowers Logo](https://www.kitware.com/img/small_logo_over.png)](https://www.kitware.com)\n\n\n", "description_content_type": "", "docs_url": null, "download_url": "", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "https://github.com/SuperCowPowers/bat", "keywords": "Bro,Python,Networking,Security,Scikit-Learn,Spark,Kafka,Parquet", "license": "Apache", "maintainer": "", "maintainer_email": "", "name": "zat", "package_url": "https://pypi.org/project/zat/", "platform": "", "project_url": "https://pypi.org/project/zat/", "project_urls": { "Homepage": "https://github.com/SuperCowPowers/bat" }, "release_url": "https://pypi.org/project/zat/0.3.4/", "requires_dist": [ "requests", "watchdog", "numpy", "scipy", "pandas", "scikit-learn", "pyarrow", "yara-python; extra == 'all'", "tldextract; extra == 'all'" ], "requires_python": "", "summary": "Bro Analysis Tools", "version": "0.3.4" }, "last_serial": 4388387, "releases": { "0.3.4": [ { "comment_text": "", "digests": { "md5": "34d1066c3335ab432ebd567e6775a0d7", "sha256": "a7b288c895931ee02d44bbc6d7ea49b77850da0f7648ede44b333ffe1e961570" }, "downloads": -1, "filename": "zat-0.3.4-py2.py3-none-any.whl", "has_sig": false, "md5_digest": "34d1066c3335ab432ebd567e6775a0d7", "packagetype": "bdist_wheel", "python_version": "py2.py3", "requires_python": null, "size": 162259, "upload_time": "2018-10-18T00:56:09", "url": "https://files.pythonhosted.org/packages/75/bb/38005661b4362833b78dfb97ddc39be2d099985867e5682939ff5df84682/zat-0.3.4-py2.py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "8025a655ef07308ae4be5a50daa89c38", "sha256": "b53d466626349b50eede9e3e5a73d8356d5ad31be7cff0e7e223671ee0437caa" }, "downloads": -1, "filename": "zat-0.3.4.tar.gz", "has_sig": false, "md5_digest": "8025a655ef07308ae4be5a50daa89c38", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 142874, "upload_time": "2018-10-18T00:56:10", "url": "https://files.pythonhosted.org/packages/82/f6/8dd50986527c1beb2f6e2f5f15d1dc56f9fe0946eab051fc1056d2ab49a1/zat-0.3.4.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "34d1066c3335ab432ebd567e6775a0d7", "sha256": "a7b288c895931ee02d44bbc6d7ea49b77850da0f7648ede44b333ffe1e961570" }, "downloads": -1, "filename": "zat-0.3.4-py2.py3-none-any.whl", "has_sig": false, "md5_digest": "34d1066c3335ab432ebd567e6775a0d7", "packagetype": "bdist_wheel", "python_version": "py2.py3", "requires_python": null, "size": 162259, "upload_time": "2018-10-18T00:56:09", "url": "https://files.pythonhosted.org/packages/75/bb/38005661b4362833b78dfb97ddc39be2d099985867e5682939ff5df84682/zat-0.3.4-py2.py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "8025a655ef07308ae4be5a50daa89c38", "sha256": "b53d466626349b50eede9e3e5a73d8356d5ad31be7cff0e7e223671ee0437caa" }, "downloads": -1, "filename": "zat-0.3.4.tar.gz", "has_sig": false, "md5_digest": "8025a655ef07308ae4be5a50daa89c38", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 142874, "upload_time": "2018-10-18T00:56:10", "url": "https://files.pythonhosted.org/packages/82/f6/8dd50986527c1beb2f6e2f5f15d1dc56f9fe0946eab051fc1056d2ab49a1/zat-0.3.4.tar.gz" } ] }