{ "info": { "author": "Fedele Mantuano", "author_email": "mantuano.fedele@gmail.com", "bugtrack_url": null, "classifiers": [ "Intended Audience :: Developers", "License :: OSI Approved :: Apache Software License", "Operating System :: OS Independent", "Programming Language :: Python", "Programming Language :: Python :: 2", "Programming Language :: Python :: 2.6", "Programming Language :: Python :: 2.7" ], "description": "`PyPI version `__ `Build\nStatus `__ `Coverage\nStatus `__\n`BCH compliance `__\n\n.. figure:: https://raw.githubusercontent.com/SpamScope/spamscope/develop/docs/logo/spamscope.png\n :alt: SpamScope\n\n SpamScope\n\nOverview\n========\n\nSpamScope is an advanced spam analysis tool that use `Apache\nStorm `__ with\n`streamparse `__ to process a\nstream of mails. To understand how SpamScope works, I suggest to read\nthese overviews: - `Apache Storm\nConcepts `__ -\n`Streamparse\nQuickstart `__\n\nIn general the first step is run Apache Storm, then you can run the\ntopologies on it. SpamScope has some topologies in `topologies\nfolder <./topologies/>`__, but you can make others topologies.\n\n.. figure:: docs/images/schema_topology.png?raw=true\n :alt: Schema topology\n\n Schema topology\n\nWhat Does SpamScope do?\n=======================\n\nSpamScope gets the raw emails (both RFC822 and Outlook formats) in input\nand returns an JSON object. Then it extracts urls and attachments (if\nthey are zipped extracts the content files). All informations are saved\nin JSON objects. This is the first analysis. After that SpamScope runs a\n*phishing* module, that gives a *phishing score* to the emails.\n\nThen you can enable/disable post processing modules, that connect\nSpamScope with third party tools. There are three main categories: - raw\nemails analysis - attachments analysis - sender emails analysis\n\nIt\u2019s possible to add new modules in these three categories, if you want\nconnect SpamScope with others tools.\n\nRaw emails analysis\n-------------------\n\nThese modules (see `here <./src/modules/mails>`__) analyze the raw\nemails: - SMTP dialect - SpamAssassin\n\nAttachments analysis\n--------------------\n\nThese modules (see `here <./src/modules/attachments>`__) analyze the\nattachments of emails: - Apache Tika - Store sample on disk (as default\nSpamScope saves samples in JSON objects) - Thug - VirusTotal - Zemana\n\nSender emails analysis\n----------------------\n\nSpamScope can detects the exact sender IP and then it can analyze it\n(see `here <./src/modules/networks>`__): - Shodan - VirusTotal\n\nWhy should I use SpamScope\n==========================\n\n- It\u2019s very fast: the job is splitted in functionalities that work in\n parallel.\n- It\u2019s flexible: you can choose what SpamScope has to do.\n- It\u2019s distributed: SpamScope uses Apache Storm, free and open source\n distributed realtime computation system.\n- It makes JSON output that you can save where you want.\n- It\u2019s easy to setup: there are docker images and docker-compose ready\n for use.\n- It\u2019s integrated with Apache Tika, VirusTotal, Thug, Shodan and\n SpamAssassin (for now).\n- It\u2019s free and open source (for special functions you can contact me).\n- It can analyze Outlook msg.\n\nDistributed\n-----------\n\nSpamScope uses Apache Storm that allows you to start small and scale\nhorizontally as you grow. Simply add more workers.\n\nFlexibility\n-----------\n\nYou can choose your mails input sources (with **spouts**) and your\nfunctionalities (with **bolts**).\n\nSpamScope comes with the following bolts: - **tokenizer** splits mail in\ntoken like headers, body, attachments and it can filter emails,\nattachments and ip addresses already seen - **phishing** looks for your\nkeywords in email and connects email to targets (bank, your customers,\netc.) - **raw_mail** is for all third party tools that analyze raw mails\nlike SpamAssassin - **attachments** analyzes all mail attachments and\nuses third party tools like VirusTotal - **network** analyzes all sender\nip addresses with third party tools like Shodan - **urls** extracts all\nurls in email and attachments - **json_maker** and **outputs** make the\njson report and save it\n\nStore where you want\n--------------------\n\nYou can build your custom output bolts and store your data in\nElasticsearch, MongoDB, filesystem, etc.\n\nBuild your topology\n-------------------\n\nWith streamparse tecnology you can build your topology in Python, add\nand/or remove spouts and bolts.\n\nAPI\n---\n\nFor now SpamScope doesn\u2019t have its own API, because it isn\u2019t tied to any\ntecnology. If you use ``Redis`` as spout (input), you\u2019ll use Redis API\nto put mails in topology. If you use ``Elasticsearch`` as output, you\u2019ll\nuse Elasticsearch API to get results.\n\nIt\u2019s possible to develop a middleware API that it talks with input,\noutput and changes the configuration, but now there isn\u2019t.\n\nApache 2 Open Source License\n============================\n\nSpamScope can be downloaded, used, and modified free of charge. It is\navailable under the Apache 2 license.\n\n`Donate `__\n\nSpamScope on Web\n================\n\n- `Shodan Applications &\n Integrations `__\n- `The Honeynet Project `__\n- `securityonline.info `__\n- `jekil/awesome-hacking `__\n- `Linux Security\n Expert `__\n\nAuthors\n=======\n\nMain Author\n-----------\n\nFedele Mantuano (**LinkedIn**: `Fedele\nMantuano `__)\n\nRequirements\n============\n\nFor operating system requirements you can read `Ansible\nplaybooks <./ansible>`__, that go into details.\n\nFor Python requirements you can read: \\* `mandatory\nrequirements <./requirements.txt>`__ \\* `optional\nrequirements <./requirements_optional.txt>`__\n\n*Thug* is another optional requirement, that it\u2019s not in requirements.\nSee `Thug section <#thug-optional>`__ for more details.\n\nApache Storm\n------------\n\n`Apache Storm `__ is a free and open source\ndistributed realtime computation system.\n\nstreamparse\n-----------\n\n`streamparse `__ lets you run\nPython code against real-time streams of data via Apache Storm.\n\nmail-parser\n-----------\n\n`mail-parser `__ is the\nparsing for raw email of SpamScope.\n\nFaup\n----\n\n`Faup `__ stands for Finally An Url\nParser and is a library and command line tool to parse URLs and\nnormalize fields.\n\nrarlinux (optional)\n-------------------\n\n`rarlinux `__ unarchives rar file.\n\nSpamAssassin (optional)\n-----------------------\n\nSpamScope can use `SpamAssassin `__ an\nopen source anti-spam to analyze every mails.\n\nApache Tika (optional)\n----------------------\n\nSpamScope can use `Apache Tika `__ to parse\nevery attachments. The Apache Tika toolkit detects and extracts metadata\nand text from over a thousand different file types (such as PPT, XLS,\nand PDF). To use Apache Tika in SpamScope you must install\n`tika-app-python `__\nwith ``pip`` and `Apache\nTika `__.\n\nThug (optional)\n---------------\n\nFrom release v1.3 SpamScope can analyze Javascript and HTML attachments\nwith `Thug `__. If you want to analyze\nthe attachments with Thug, follow `these\ninstructions `__ to install\nit. Enable it in ``attachments`` section of `main configuration\nfile <./conf/spamscope.example.yml>`__.\n\nWhat is Thug? From README project: > Thug is a Python low-interaction\nhoneyclient aimed at mimicing the behavior of a web browser in order to\ndetect and emulate malicious contents.\n\nYou can see a complete SpamScope report with Thug analysis\n`here `__.\n\nThug analysis can be very slow and you can have ``heartbeat timeout``\nerrors in Apache Storm. To avoid any issue set\n``supervisor.worker.timeout.secs``:\n\n::\n\n nr. user agents * timeout_thug < supervisor.worker.timeout.secs\n\nThe best value for ``threshold`` is 1.\n\nVirusTotal (optional)\n---------------------\n\nIt\u2019s possible add to results (for mail attachments and sender ip\naddress) the VirusTotal report. You need a private API key.\n\nShodan (optional)\n-----------------\n\nIt\u2019s possible add to results the Shodan report for sender ip address.\nYou need a private API key.\n\nElasticsearch (optional)\n------------------------\n\nIt\u2019s possible to store the results in Elasticsearch. In this case you\nshould install ``elasticsearch`` package.\n\nRedis (optional)\n----------------\n\nIt\u2019s possible to store the results in Redis. In this case you should\ninstall ``redis`` package.\n\nConfiguration\n=============\n\nRead the `example of main configuration\nfile <./conf/spamscope.example.yml>`__. The default value where\nSpamScope will search the configuration file is\n``/etc/spamscope/spamscope.yml``, but it\u2019s possible to set the\nenvironment variable ``SPAMSCOPE_CONF_FILE``:\n\n::\n\n $ export SPAMSCOPE_CONF_FILE=/etc/spamscope/spamscope.yml\n\nWhen you change the configuration file, SpamScope automatically reloads\nthe new changes.\n\nInstallation\n============\n\nYou can use: \\* `Docker images <./docker/README.md>`__ to run SpamScope\nwith docker engine \\* `Ansible <./ansible/README.md>`__: to install and\nrun SpamScope on server\n\nTopologies\n==========\n\nSpamScope comes with six topologies: -\n`spamscope_debug <./topologies/spamscope_debug.py>`__: the output are\nJSON files on file system. -\n`spamscope_elasticsearch <./topologies/spamscope_elasticsearch.py>`__:\nthe output are stored in Elasticsearch indexes. -\n`spamscope_redis <./topologies/spamscope_redis.py>`__: the output are\nstored in Redis. -\n`spamscope_debug_iter <./topologies/spamscope_debug_iter.py>`__: It uses\ngenerator to send mails in topology. The output are JSON files on file\nsystem. -\n`spamscope_elasticsearch_iter <./topologies/spamscope_elasticsearch_iter.py>`__:\nIt uses generator to send mails in topology. The output are stored in\nElasticsearch indexes. -\n`spamscope_redis_iter <./topologies/spamscope_redis_iter.py>`__: It uses\ngenerator to send mails in topology. The output are stored in Redis.\n\nIf you want submit SpamScope topology use ``spamscope-topology submit``\ntool. For more details `see SpamScope cli tools `__:\n\n::\n\n $ spamscope-topology submit --topology {spamscope_debug,spamscope_elasticsearch,spamscope_redis}\n\nIt\u2019s possible to change the default settings for all Apache Storm\noptions. I suggest to change these options:\n\n- **topology.tick.tuple.freq.secs**: reload configuration of all bolts\n- **topology.max.spout.pending**: Apache Storm framework will then\n throttle your spout as needed to meet the\n ``topology.max.spout.pending`` requirement\n- **topology.sleep.spout.wait.strategy.time.ms**: max sleep for emit\n new tuple (mail)\n\nYou can use ``spamscope-topology submit`` to do these changes.\n\nImportant\n=========\n\nIf you are using Elasticsearch output, I suggest you to use\n`Elasticsearch templates <./conf/templates>`__ that comes with\nSpamScope.\n\nUnittest\n========\n\nSpamScope comes with unittests for each modules. In bolts and spouts\nthere are no special features, all intelligence is in external modules.\nAll unittests are in `tests folder `__.\n\nTo have complete tests you should set the followings enviroment\nvariables:\n\n::\n\n $ export THUG_ENABLED=True\n $ export VIRUSTOTAL_ENABLED=True\n $ export VIRUSTOTAL_APIKEY=\"your key\"\n $ export ZEMANA_ENABLED=True\n $ export ZEMANA_APIKEY=\"your key\"\n $ export ZEMANA_PARTNERID=\"your partner id\"\n $ export ZEMANA_USERID=\"your userid\"\n $ export SHODAN_ENABLED=True\n $ export SHODAN_APIKEY=\"your key\"\n $ export SPAMASSASSIN_ENABLED=True\n\nOutput example\n==============\n\nThis is a `raw email `__ that I analyzed with\nSpamScope: - `SpamScope output `__.\n\nThis is another example with `Thug analysis `__.\n\nScreenshots\n===========\n\n.. figure:: docs/images/Docker00.png?raw=true\n :alt: Apache Storm\n\n Apache Storm\n\n.. figure:: docs/images/Docker01.png?raw=true\n :alt: SpamScope\n\n SpamScope\n\n.. figure:: docs/images/Docker02.png?raw=true\n :alt: SpamScope Topology\n\n SpamScope Topology\n\n.. figure:: docs/images/map.png?raw=true\n :alt: SpamScope Map\n\n SpamScope Map\n\n\n", "description_content_type": "", "docs_url": null, "download_url": "", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "https://github.com/SpamScope/spamscope", "keywords": "spam,email,mail,apache,apache-storm", "license": "Apache License, Version 2.0", "maintainer": "Fedele Mantuano", "maintainer_email": "mantuano.fedele@gmail.com", "name": "SpamScope", "package_url": "https://pypi.org/project/SpamScope/", "platform": "Linux", "project_url": "https://pypi.org/project/SpamScope/", "project_urls": { "Homepage": "https://github.com/SpamScope/spamscope" }, "release_url": "https://pypi.org/project/SpamScope/2.8.1/", "requires_dist": [ "PyYAML", "astropy (==1.3.3)", "backports.functools-lru-cache (>=1.3)", "chainmap", "lxml", "mail-parser (>=3.9.0)", "patool", "pyparsing", "python-magic", "simplejson", "six", "ssdeep", "streamparse (==3.15.1)" ], "requires_python": "", "summary": "Fast Advanced Spam Analysis tool", "version": "2.8.1" }, "last_serial": 4947855, "releases": { "2.0.0": [ { "comment_text": "", "digests": { "md5": "ae9f6e125c85497fc93d61ad13c9343b", "sha256": "b8323972ddd90aa63aa3078ac8b656f84a837eb53366f8f0866093f97e55b8f8" }, "downloads": -1, "filename": "SpamScope-2.0.0.zip", "has_sig": false, "md5_digest": "ae9f6e125c85497fc93d61ad13c9343b", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 71298, "upload_time": "2017-12-05T00:32:33", "url": "https://files.pythonhosted.org/packages/9f/8b/56fb63f3fef22eae3031c9f6cbdba505d14eb500eaf34dba15521cfe5c6a/SpamScope-2.0.0.zip" } ], "2.1.0": [ { "comment_text": "", "digests": { "md5": "7cee3fbd6dc6717fb9bc73e91428a313", "sha256": "7a051f1ec03aec9fb7ad1be4841486e5e6a46641172afaef6f67218604db6a33" }, "downloads": -1, "filename": "SpamScope-2.1.0.tar.gz", "has_sig": false, "md5_digest": "7cee3fbd6dc6717fb9bc73e91428a313", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 36296, "upload_time": "2017-12-08T17:14:33", "url": "https://files.pythonhosted.org/packages/a3/e7/030bc4221cb1f5ed06c43b030b54ce102f3297672c851e79f38a2d764202/SpamScope-2.1.0.tar.gz" } ], "2.1.2": [ { "comment_text": "", "digests": { "md5": "fb72217bc2ba38eecb7252b94d32efbd", "sha256": "0a22846a879a5eef0ea847736601c86a72106f3d7aeea7208f0e556f797dd3c8" }, "downloads": -1, "filename": "SpamScope-2.1.2.tar.gz", "has_sig": false, "md5_digest": "fb72217bc2ba38eecb7252b94d32efbd", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 36301, "upload_time": "2017-12-09T08:30:14", "url": "https://files.pythonhosted.org/packages/c2/4c/a2c56ad4a64b83a743f501bec9f3a7f7607fe00e3e0ee4030f21bf3267c9/SpamScope-2.1.2.tar.gz" } ], "2.2.0": [ { "comment_text": "", "digests": { "md5": "128bfa5448baa7cba7585b32bf0f9624", "sha256": "9f157674068833ae4e827a0b2672d3333b761e9bcad25a71c57376812fc9733b" }, "downloads": -1, "filename": "SpamScope-2.2.0.tar.gz", "has_sig": false, "md5_digest": "128bfa5448baa7cba7585b32bf0f9624", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 36624, "upload_time": "2018-01-13T16:39:21", "url": "https://files.pythonhosted.org/packages/3b/78/15ff39f972e054a026b3796d25888cb5a4b191838fa04b49b94bd3efcf5e/SpamScope-2.2.0.tar.gz" } ], "2.3.0": [ { "comment_text": "", "digests": { "md5": "090514bb70bf0837ff58257784f5908d", "sha256": "4a2c8835abc8ede2baa3adfa506c4ed0f580c61fd286744545e298578e8f9745" }, "downloads": -1, "filename": "SpamScope-2.3.0.tar.gz", "has_sig": false, "md5_digest": "090514bb70bf0837ff58257784f5908d", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 33315, "upload_time": "2018-03-07T22:53:56", "url": "https://files.pythonhosted.org/packages/9f/8a/dcd7c833eb88e920ac245c5c510ddac0b6bf4b9fb03c85098417b4a6093b/SpamScope-2.3.0.tar.gz" } ], "2.4.0": [ { "comment_text": "", "digests": { "md5": "2baeea19526402ed05b774062d9c5934", "sha256": "299533fd16cebf60d5a94daf4dd1efe0ddf28f5d387065d52ddafe6322c35ceb" }, "downloads": -1, "filename": "SpamScope-2.4.0.tar.gz", "has_sig": false, "md5_digest": "2baeea19526402ed05b774062d9c5934", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 33673, "upload_time": "2018-04-05T12:53:21", "url": "https://files.pythonhosted.org/packages/8e/81/28b222f47ffb21c94409580d2e906b5551a0217932134711286f316a6036/SpamScope-2.4.0.tar.gz" } ], "2.5.0": [ { "comment_text": "", "digests": { "md5": "b48988841dcb3e0d68e7c739b3c5cdc3", "sha256": "4030948cb97b8b5f63fa68e2af1d2d977443eac99afda07c2630f4d01d7022a9" }, "downloads": -1, "filename": "SpamScope-2.5.0.tar.gz", "has_sig": false, "md5_digest": "b48988841dcb3e0d68e7c739b3c5cdc3", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 34721, "upload_time": "2018-07-14T08:12:12", "url": "https://files.pythonhosted.org/packages/da/61/d07c04661927f36673d8ddc9b4ffd3ed9b03a8d9bc8cd12756f8a7ed2dbf/SpamScope-2.5.0.tar.gz" } ], "2.6.0": [ { "comment_text": "", "digests": { "md5": "0180e9f646167d35c8db76e270c38880", "sha256": "174a4f13be50ca42cc7b2d567092e38b180751b1fd534c9896179fbf1961e6fa" }, "downloads": -1, "filename": "SpamScope-2.6.0.tar.gz", "has_sig": false, "md5_digest": "0180e9f646167d35c8db76e270c38880", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 37757, "upload_time": "2018-09-15T17:14:41", "url": "https://files.pythonhosted.org/packages/08/11/83099f930a2b5c76fc7e281c2a685d9d0787e0b1c116efffeeb51a85bef0/SpamScope-2.6.0.tar.gz" } ], "2.7.0": [ { "comment_text": "", "digests": { "md5": "da24aa1fb608f00644e3da7c3892c9bf", "sha256": "532557c8188e82155222e7e7c48ed03281cd0ad2afe2974e5c68eb3d866f51c6" }, "downloads": -1, "filename": "SpamScope-2.7.0.tar.gz", "has_sig": false, "md5_digest": "da24aa1fb608f00644e3da7c3892c9bf", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 37920, "upload_time": "2019-02-09T10:51:10", "url": "https://files.pythonhosted.org/packages/bd/0a/029b5b1255d32ff57899405be1650433d4f40a6006ea2db1d93c63e98690/SpamScope-2.7.0.tar.gz" } ], "2.8.0": [ { "comment_text": "", "digests": { "md5": "a35aed441a6ba4598759e6a385e60e65", "sha256": "e45c22be933dc9721c0b5f3e6761167dee6bdf421644a6e875703c93ecff0540" }, "downloads": -1, "filename": "SpamScope-2.8.0-py2-none-any.whl", "has_sig": false, "md5_digest": "a35aed441a6ba4598759e6a385e60e65", "packagetype": "bdist_wheel", "python_version": "py2", "requires_python": null, "size": 72041, "upload_time": "2019-03-16T14:12:58", "url": "https://files.pythonhosted.org/packages/b4/d7/a07e01cc88e9ec8d7155d396e80267cae19052a3da78f6600545cd71fd12/SpamScope-2.8.0-py2-none-any.whl" }, { "comment_text": "", "digests": { "md5": "cfb59b94a1c3afdb45efa8eab0a428f1", "sha256": "cc9224503c4d10f62378bf0ae56b256c5068a73242bb6b8b2b8b46947bbfbc63" }, "downloads": -1, "filename": "SpamScope-2.8.0.tar.gz", "has_sig": false, "md5_digest": "cfb59b94a1c3afdb45efa8eab0a428f1", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 43339, "upload_time": "2019-03-16T14:13:00", "url": "https://files.pythonhosted.org/packages/38/2c/78dbb8e45d65ea081b32b23b22f9bffa3911e1a60e26010d21e1e9ba5023/SpamScope-2.8.0.tar.gz" } ], "2.8.1": [ { "comment_text": "", "digests": { "md5": "4535e11319752edcd219abdc5c7ae515", "sha256": "cbc61ed8278c8ee59d4c20acc3231a724f21dd307434223d51c0823f790f4406" }, "downloads": -1, "filename": "SpamScope-2.8.1-py2-none-any.whl", "has_sig": false, "md5_digest": "4535e11319752edcd219abdc5c7ae515", "packagetype": "bdist_wheel", "python_version": "py2", "requires_python": null, "size": 72097, "upload_time": "2019-03-16T14:55:28", "url": "https://files.pythonhosted.org/packages/4d/f1/2e67a47256bf3311a6d907973d0519d9b09f12f9b4bfe712d9797eee287c/SpamScope-2.8.1-py2-none-any.whl" }, { "comment_text": "", "digests": { "md5": "c0ae267f32ef936219c01f88cc2538e5", "sha256": "f5170f4b5f21a0d4befafd16333087cab2b6e064c98074aaf8c958fdc7eccb32" }, "downloads": -1, "filename": "SpamScope-2.8.1.tar.gz", "has_sig": false, "md5_digest": "c0ae267f32ef936219c01f88cc2538e5", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 43387, "upload_time": "2019-03-16T14:55:30", "url": "https://files.pythonhosted.org/packages/9f/38/2e9bc66245653647b99d43975370672f325da3d3868ac952f011e6d25064/SpamScope-2.8.1.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "4535e11319752edcd219abdc5c7ae515", "sha256": "cbc61ed8278c8ee59d4c20acc3231a724f21dd307434223d51c0823f790f4406" }, "downloads": -1, "filename": "SpamScope-2.8.1-py2-none-any.whl", "has_sig": false, "md5_digest": "4535e11319752edcd219abdc5c7ae515", "packagetype": "bdist_wheel", "python_version": "py2", "requires_python": null, "size": 72097, "upload_time": "2019-03-16T14:55:28", "url": "https://files.pythonhosted.org/packages/4d/f1/2e67a47256bf3311a6d907973d0519d9b09f12f9b4bfe712d9797eee287c/SpamScope-2.8.1-py2-none-any.whl" }, { "comment_text": "", "digests": { "md5": "c0ae267f32ef936219c01f88cc2538e5", "sha256": "f5170f4b5f21a0d4befafd16333087cab2b6e064c98074aaf8c958fdc7eccb32" }, "downloads": -1, "filename": "SpamScope-2.8.1.tar.gz", "has_sig": false, "md5_digest": "c0ae267f32ef936219c01f88cc2538e5", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 43387, "upload_time": "2019-03-16T14:55:30", "url": "https://files.pythonhosted.org/packages/9f/38/2e9bc66245653647b99d43975370672f325da3d3868ac952f011e6d25064/SpamScope-2.8.1.tar.gz" } ] }