{ "info": { "author": "InQuest Labs", "author_email": "labs@inquest.net", "bugtrack_url": null, "classifiers": [ "Intended Audience :: Developers", "Intended Audience :: Information Technology", "Intended Audience :: System Administrators", "License :: OSI Approved :: GNU General Public License v2 (GPLv2)", "Operating System :: OS Independent", "Programming Language :: Python :: 3.6", "Programming Language :: Python :: 3.7", "Programming Language :: Python :: 3 :: Only", "Topic :: Internet", "Topic :: Security" ], "description": "ThreatIngestor\n==============\n\n.. image:: https://inquest.net/images/inquest-badge.svg\n :target: https://inquest.net/\n :alt: Developed by InQuest\n.. image:: https://travis-ci.org/InQuest/ThreatIngestor.svg?branch=master\n :target: https://travis-ci.org/InQuest/ThreatIngestor\n :alt: Build Status\n.. image:: https://readthedocs.org/projects/threatingestor/badge/?version=latest\n :target: http://inquest.readthedocs.io/projects/threatingestor/en/latest/?badge=latest\n :alt: Documentation Status\n.. image:: https://api.codacy.com/project/badge/Grade/a989bb12e9604d5a9577ce71848e7a2a\n :target: https://app.codacy.com/app/InQuest/ThreatIngestor\n :alt: Code Health\n.. image:: https://api.codacy.com/project/badge/Coverage/a989bb12e9604d5a9577ce71848e7a2a\n :target: https://app.codacy.com/app/InQuest/ThreatIngestor\n :alt: Test Coverage\n.. image:: http://img.shields.io/pypi/v/ThreatIngestor.svg\n :target: https://pypi.python.org/pypi/ThreatIngestor\n :alt: PyPi Version\n\nAn extendable tool to extract and aggregate IOCs_ from threat feeds.\n\nIntegrates out-of-the-box with ThreatKB_ and MISP_, and can fit seamlessly into any existing worflow with SQS_, Beanstalk_, and `custom plugins`_.\n\nOverview\n--------\n\nThreatIngestor can be configured to watch Twitter, RSS feeds, or other sources, extract meaningful information such as malicious IPs/domains and YARA signatures, and send that information to another system for analysis.\n\n.. image:: https://inquest.readthedocs.io/projects/threatingestor/en/latest/_images/mermaid-multiple-operators.png\n :target: https://inquest.readthedocs.io/projects/threatingestor/en/latest/workflows.html\n :alt: ThreatIngestor flowchart with several sources feeding into multiple operators.\n\nTry it out now with this `quick walkthrough`_, or see more `ThreatIngestor walkthroughs`_ on the InQuest blog.\n\nInstallation\n------------\n\nThreatIngestor requires Python 3.6+, with development headers.\n\nInstall ThreatIngestor from PyPI::\n\n pip install threatingestor\n\nInstall optional dependencies for using some plugins, as needed::\n\n pip install threatingestor[all]\n\nView the `full installation instructions`_ for more information.\n\nUsage\n-----\n\nCreate a new ``config.yml`` file, and configure each source and operator module you want to use. (See ``config.example.yml`` for layout.) Then run the script::\n\n threatingestor config.yml\n\nBy default, it will run forever, polling each configured source every 15 minutes.\n\nView the `full ThreatIngestor documentation`_ for more information.\n\nPlugins\n-------\n\nThreatIngestor uses a plugin architecture with \"source\" (input) and \"operator\" (output) plugins. The currently supported integrations are:\n\nSources\n~~~~~~~\n\n* `Beanstalk work queues `__\n* `Git repositories `__\n* `GitHub repository search `__\n* `RSS feeds `__\n* `Amazon SQS queues `__\n* `Twitter `__\n* `Generic web pages `__\n\nOperators\n~~~~~~~~~\n\n* `Beanstalk work queues `__\n* `CSV files `__\n* `MISP `__\n* `MySQL table `__\n* `SQLite database `__\n* `Amazon SQS queues `__\n* `ThreatKB `__\n* `Twitter `__\n\nView the `full ThreatIngestor documentation`_ for more information on included plugins, and how to create your own.\n\nThreat Intel Sources\n--------------------\n\nLooking for some threat intel sources to get started? InQuest has a Twitter List with several accounts that post C2 domains and IPs: https://twitter.com/InQuest/lists/c2-feed. Note that you will need to apply for a Twitter developer account to use the ThreatIngestor Twitter Source. Take a look at ``config.example.yml`` to see how to set this list up as a source.\n\nFor quicker setup, RSS feeds can be a great source of intelligence. Check out this example `RSS config file`_ for a few pre-configured security blogs.\n\nSupport\n-------\n\nIf you need help getting set up, or run into any issues, feel free to open an Issue_. You can also reach out to `@InQuest`_ on Twitter.\n\nWe'd love to hear any feedback you have on ThreatIngestor, its documentation, or how you're putting it to work for you!\n\nContributing\n------------\n\nIssues and pull requests are welcomed. Please keep Python code PEP8 compliant. By submitting a pull request you agree to release your submissions under the terms of the LICENSE_.\n\n.. _ThreatKB: https://github.com/InQuest/ThreatKB\n.. _LICENSE: https://github.com/InQuest/threat-ingestors/blob/master/LICENSE\n.. _full ThreatIngestor Documentation: https://inquest.readthedocs.io/projects/threatingestor/\n.. _SQS: https://aws.amazon.com/sqs/\n.. _Beanstalk: https://beanstalkd.github.io/\n.. _MISP: https://www.misp-project.org/\n.. _custom plugins: https://inquest.readthedocs.io/projects/threatingestor/en/latest/developing.html\n.. _IOCs: https://en.wikipedia.org/wiki/Indicator_of_compromise\n.. _full installation instructions: https://inquest.readthedocs.io/projects/threatingestor/en/latest/installation.html\n.. _Issue: https://github.com/InQuest/ThreatIngestor/issues\n.. _@InQuest: https://twitter.com/InQuest\n.. _quick walkthrough: https://inquest.readthedocs.io/projects/threatingestor/en/latest/welcome.html#try-it-out\n.. _ThreatIngestor walkthroughs: https://inquest.net/taxonomy/term/42\n.. _RSS config file: https://github.com/InQuest/ThreatIngestor/blob/master/rss.example.yml\n\n\n", "description_content_type": "", "docs_url": null, "download_url": "", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "https://github.com/InQuest/ThreatIngestor", "keywords": "", "license": "GPL", "maintainer": "", "maintainer_email": "", "name": "threatingestor", "package_url": "https://pypi.org/project/threatingestor/", "platform": "", "project_url": "https://pypi.org/project/threatingestor/", "project_urls": { "Homepage": "https://github.com/InQuest/ThreatIngestor" }, "release_url": "https://pypi.org/project/threatingestor/1.0.0b6/", "requires_dist": [ "beautifulsoup4 (>=4.4.1)", "ipaddress (>=1.0.18)", "iocextract (>=1.12.1)", "PyYAML (>=3.13)", "jsonpath-rw (>=1.4.0)", "sgmllib3k", "loguru", "statsd", "requests", "beautifulsoup4 (>=4.4.1); extra == 'all'", "ipaddress (>=1.0.18); extra == 'all'", "iocextract (>=1.12.1); extra == 'all'", "PyYAML (>=3.13); extra == 'all'", "jsonpath-rw (>=1.4.0); extra == 'all'", "sgmllib3k; extra == 'all'", "loguru; extra == 'all'", "statsd; extra == 'all'", "requests; extra == 'all'", "greenstalk; extra == 'beanstalk'", "hug; extra == 'extras'", "boto3; extra == 'extras'", "greenstalk; extra == 'extras'", "watchdog; extra == 'extras'", "PyMISP; extra == 'misp'", "pymysql; extra == 'mysql'", "feedparser; extra == 'rss'", "boto3; extra == 'sqs'", "threatkb; extra == 'threatkb'", "twitter; extra == 'twitter'" ], "requires_python": "", "summary": "Extract and aggregate IOCs from threat feeds.", "version": "1.0.0b6" }, "last_serial": 5233775, "releases": { "1.0.0a1": [ { "comment_text": "", "digests": { "md5": "2a0963bd2d989c90c318eaaba9a32400", "sha256": "cb3b24ad33b0eecbf0287e430142615f17f4334328e8df6fb45c3649fc3dfaca" }, "downloads": -1, "filename": "threatingestor-1.0.0a1-py2.py3-none-any.whl", "has_sig": true, "md5_digest": "2a0963bd2d989c90c318eaaba9a32400", "packagetype": "bdist_wheel", "python_version": "py2.py3", "requires_python": null, "size": 30468, "upload_time": "2019-03-12T15:55:54", "url": "https://files.pythonhosted.org/packages/4f/8b/d7a70faa3d116843caa955d14e184c9ad263955906536eb72fce64a8b256/threatingestor-1.0.0a1-py2.py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "ab27169f86f8da2ccae4401aefb80621", "sha256": "b2a66082ec7b544ea52d88b9b3124ac97a9b770275f70d1a5400360cefa1f77c" }, "downloads": -1, "filename": "threatingestor-1.0.0a1.tar.gz", "has_sig": true, "md5_digest": "ab27169f86f8da2ccae4401aefb80621", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 27259, "upload_time": "2019-03-12T15:55:56", "url": "https://files.pythonhosted.org/packages/9e/02/8339c1d175bdae468152016ca94e4a63f7ddddb4ff0d016988813e31226e/threatingestor-1.0.0a1.tar.gz" } ], "1.0.0a2": [ { "comment_text": "", "digests": { "md5": "ed7f5ce8cd719a87e480950245fb0821", "sha256": "c30f363d1783c6457729357a7c4773130f0787604d7bd737a1c83e9992b4ffd8" }, "downloads": -1, "filename": "threatingestor-1.0.0a2-py2.py3-none-any.whl", "has_sig": true, "md5_digest": "ed7f5ce8cd719a87e480950245fb0821", "packagetype": "bdist_wheel", "python_version": "py2.py3", "requires_python": null, "size": 33217, "upload_time": "2019-03-22T20:27:24", "url": "https://files.pythonhosted.org/packages/84/46/961c851c17909d42aa1c26943e6ab27106857509356f69bff7068c139a0a/threatingestor-1.0.0a2-py2.py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "624aa39095f28b4661b3dd667284af96", "sha256": "718f45391290762267a2d1b9542c46171936e32d0da7212d323e208ad91b1844" }, "downloads": -1, "filename": "threatingestor-1.0.0a2.tar.gz", "has_sig": true, "md5_digest": "624aa39095f28b4661b3dd667284af96", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 29231, "upload_time": "2019-03-22T20:27:25", "url": "https://files.pythonhosted.org/packages/ee/f9/c70a54859df9c79c4a1acfa8ba74fb3662e092a77c9f7901d686b465af41/threatingestor-1.0.0a2.tar.gz" } ], "1.0.0b1": [ { "comment_text": "", "digests": { "md5": "c6c07f47dcf1606ee747348a9babae2b", "sha256": "31404dfee87daa9d22439248f224819237e1a0bf3bb59d8b1ad1a510f85fa361" }, "downloads": -1, "filename": "threatingestor-1.0.0b1-py2.py3-none-any.whl", "has_sig": true, "md5_digest": "c6c07f47dcf1606ee747348a9babae2b", "packagetype": "bdist_wheel", "python_version": "py2.py3", "requires_python": null, "size": 36238, "upload_time": "2019-03-26T18:13:42", "url": "https://files.pythonhosted.org/packages/8f/f8/245a5710262d4fb9cd6853f399f9cdebcaf6419d3af5901177a79f5c8adc/threatingestor-1.0.0b1-py2.py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "09840b3f715e8c81240de52a58fa9edf", "sha256": "d2103351409219c5355ebb1b82f1e1b8959cbc262be7cb0d2cfde37f217e914d" }, "downloads": -1, "filename": "threatingestor-1.0.0b1.tar.gz", "has_sig": true, "md5_digest": "09840b3f715e8c81240de52a58fa9edf", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 29282, "upload_time": "2019-03-26T18:13:44", "url": "https://files.pythonhosted.org/packages/e1/79/60241104fd270b368b5a332af9c22f31ed02e97e451315c39e15660f9907/threatingestor-1.0.0b1.tar.gz" } ], "1.0.0b2": [ { "comment_text": "", "digests": { "md5": "e14b459b4cc1e253f9b8a42527acd615", "sha256": "40a292a078b5a72a863f549e9c2bea8e03d02c1f5f68d7058f332d24a68367a9" }, "downloads": -1, "filename": "threatingestor-1.0.0b2-py2.py3-none-any.whl", "has_sig": true, "md5_digest": "e14b459b4cc1e253f9b8a42527acd615", "packagetype": "bdist_wheel", "python_version": "py2.py3", "requires_python": null, "size": 33571, "upload_time": "2019-03-28T14:25:08", "url": "https://files.pythonhosted.org/packages/e6/91/602169958d697fc52aacec923dcde56269d380dd848e9daf7fdbe93a9f46/threatingestor-1.0.0b2-py2.py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "fe670d1046986d259aa743010d01e5b1", "sha256": "b1f4336de72c6937ad6617dc7cf4325df469b07389e6dbf16cd2c931b503f2d8" }, "downloads": -1, "filename": "threatingestor-1.0.0b2.tar.gz", "has_sig": true, "md5_digest": "fe670d1046986d259aa743010d01e5b1", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 29581, "upload_time": "2019-03-28T14:25:10", "url": "https://files.pythonhosted.org/packages/35/c0/a2c4f55971feb613caf94f02a69b3264a9aad23ccc4c1fb2989fe574d283/threatingestor-1.0.0b2.tar.gz" } ], "1.0.0b3": [ { "comment_text": "", "digests": { "md5": "0238a36db7c36fe0c02a25fa85c6121e", "sha256": "3a1e12a1a6283e9ff6b2c22f992cd5af6e39ffb4f8ee36158565dcd88996db27" }, "downloads": -1, "filename": "threatingestor-1.0.0b3-py2.py3-none-any.whl", "has_sig": true, "md5_digest": "0238a36db7c36fe0c02a25fa85c6121e", "packagetype": "bdist_wheel", "python_version": "py2.py3", "requires_python": null, "size": 33880, "upload_time": "2019-04-09T16:05:08", "url": "https://files.pythonhosted.org/packages/0e/d8/e6af64e87777bc38bfe3f44247c5ee09762ef68027391f304f997464902a/threatingestor-1.0.0b3-py2.py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "6aa48fa77ed95f8cd87558bb530c20b4", "sha256": "3dd31531b185104445f042d69024b321a4a08995e0c13a68d5ac873f0247f066" }, "downloads": -1, "filename": "threatingestor-1.0.0b3.tar.gz", "has_sig": true, "md5_digest": "6aa48fa77ed95f8cd87558bb530c20b4", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 29922, "upload_time": "2019-04-09T16:05:10", "url": "https://files.pythonhosted.org/packages/38/72/5466332b8358a49e265a6ad5f91f6bb619e68c5408882026b12850132f6e/threatingestor-1.0.0b3.tar.gz" } ], "1.0.0b4": [ { "comment_text": "", "digests": { "md5": "0ab4052980f4636b2d79b8840ae308a5", "sha256": "f469b706b751a8e41758c0eb2d347aa4db1ff071e994b2b106b2c84ce9e1ea05" }, "downloads": -1, "filename": "threatingestor-1.0.0b4-py2.py3-none-any.whl", "has_sig": true, "md5_digest": "0ab4052980f4636b2d79b8840ae308a5", "packagetype": "bdist_wheel", "python_version": "py2.py3", "requires_python": null, "size": 33915, "upload_time": "2019-04-18T16:16:07", "url": "https://files.pythonhosted.org/packages/bd/4d/0d7451f24a0aee737f1d24135a0df6553015d2bdabdc43e824db599e61de/threatingestor-1.0.0b4-py2.py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "4ed1b992935276a4c81f3cdda0cf34bf", "sha256": "50b2d7efec983df1230b437d110469c9d355f4bfdcd348fe0707ab3cd862fff9" }, "downloads": -1, "filename": "threatingestor-1.0.0b4.tar.gz", "has_sig": true, "md5_digest": "4ed1b992935276a4c81f3cdda0cf34bf", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 29985, "upload_time": "2019-04-18T16:16:08", "url": "https://files.pythonhosted.org/packages/5a/1c/a9bc28642a851dd7d0bf7f9d02b24ab121cc0cae761012c7d4e8f87ea05d/threatingestor-1.0.0b4.tar.gz" } ], "1.0.0b5": [ { "comment_text": "", "digests": { "md5": "e7ca5f4da75edc828cf3666388a04e94", "sha256": "a42d7ac52036bcbeca7decec048636a5f57bf448e8157bf84dbd8f74b816139d" }, "downloads": -1, "filename": "threatingestor-1.0.0b5-py2.py3-none-any.whl", "has_sig": true, "md5_digest": "e7ca5f4da75edc828cf3666388a04e94", "packagetype": "bdist_wheel", "python_version": "py2.py3", "requires_python": null, "size": 34016, "upload_time": "2019-04-19T22:19:59", "url": "https://files.pythonhosted.org/packages/b1/59/926f909464b3c2004d6ae499630edff0fed8aa2370c5f40e2bba2f8af237/threatingestor-1.0.0b5-py2.py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "ab9918ae8f3f3da9385bbd5824eb424d", "sha256": "01090dd9eb9bce39a0639c30e6cca5e617ad36a2523b9a1ab976bc69e38ffd0e" }, "downloads": -1, "filename": "threatingestor-1.0.0b5.tar.gz", "has_sig": true, "md5_digest": "ab9918ae8f3f3da9385bbd5824eb424d", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 30021, "upload_time": "2019-04-19T22:20:00", "url": "https://files.pythonhosted.org/packages/96/75/bf02f34456e798ad36b1f5f6948e92e18d8af6ce460f70c7fb251cd99e55/threatingestor-1.0.0b5.tar.gz" } ], "1.0.0b6": [ { "comment_text": "", "digests": { "md5": "f6030d2be0f8a1877b0cb9edcfe2b1b2", "sha256": "b45c9d448a2cf1bed14a0d325607e9fc4fc1a0dc9b5f2601ad2be703eba6ba99" }, "downloads": -1, "filename": "threatingestor-1.0.0b6-py2.py3-none-any.whl", "has_sig": true, "md5_digest": "f6030d2be0f8a1877b0cb9edcfe2b1b2", "packagetype": "bdist_wheel", "python_version": "py2.py3", "requires_python": null, "size": 37280, "upload_time": "2019-05-06T15:57:28", "url": "https://files.pythonhosted.org/packages/93/a9/b8d36c257800e81f7918a0f299f786390f6a5630dec93b5464dc4d56602e/threatingestor-1.0.0b6-py2.py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "fd150af9ee8a7962da3a909b48b7ea9e", "sha256": "fea7d788504550c53c1a374adc65ec6ab7b7b91ac8185d9f0fbe989c92aa5e97" }, "downloads": -1, "filename": "threatingestor-1.0.0b6.tar.gz", "has_sig": true, "md5_digest": "fd150af9ee8a7962da3a909b48b7ea9e", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 30008, "upload_time": "2019-05-06T15:57:30", "url": "https://files.pythonhosted.org/packages/4f/86/17a2d7b1b390a363e7b31b9297c83cbe7f501af52920f464525efbec2275/threatingestor-1.0.0b6.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "f6030d2be0f8a1877b0cb9edcfe2b1b2", "sha256": "b45c9d448a2cf1bed14a0d325607e9fc4fc1a0dc9b5f2601ad2be703eba6ba99" }, "downloads": -1, "filename": "threatingestor-1.0.0b6-py2.py3-none-any.whl", "has_sig": true, "md5_digest": "f6030d2be0f8a1877b0cb9edcfe2b1b2", "packagetype": "bdist_wheel", "python_version": "py2.py3", "requires_python": null, "size": 37280, "upload_time": "2019-05-06T15:57:28", "url": "https://files.pythonhosted.org/packages/93/a9/b8d36c257800e81f7918a0f299f786390f6a5630dec93b5464dc4d56602e/threatingestor-1.0.0b6-py2.py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "fd150af9ee8a7962da3a909b48b7ea9e", "sha256": "fea7d788504550c53c1a374adc65ec6ab7b7b91ac8185d9f0fbe989c92aa5e97" }, "downloads": -1, "filename": "threatingestor-1.0.0b6.tar.gz", "has_sig": true, "md5_digest": "fd150af9ee8a7962da3a909b48b7ea9e", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 30008, "upload_time": "2019-05-06T15:57:30", "url": "https://files.pythonhosted.org/packages/4f/86/17a2d7b1b390a363e7b31b9297c83cbe7f501af52920f464525efbec2275/threatingestor-1.0.0b6.tar.gz" } ] }