{
"info": {
"author": "Andres J. Diaz",
"author_email": "ajdiaz@connectical.com",
"bugtrack_url": null,
"classifiers": [
"Development Status :: 4 - Beta",
"Intended Audience :: Developers",
"License :: OSI Approved :: GNU General Public License (GPL)",
"Operating System :: OS Independent",
"Programming Language :: Python"
],
"description": "Mole: A flexible operational log analyzer.\n==========================================\n\nMole is a log analyzer with parse your logs file (any kind of log), using\nspecified definitions (usually as regular expressions) and magically\ninterpret some fields (numbers, dates ...). Mole provide you a set of\nfunctions to analyze that data.\n\nInstallation\n------------\nJust as usual for each python package::\n\n pip install mole\n\nGetting started\n---------------\n\nIn this example we will use an access log file generated by apache (or any\nother HTTP server). Let's suppose that this file is located in\n/var/log/apache/access.log.\n\n.. note:: Don't worry about log rotations, mole can handle it.\n\n1. Configure mole\n~~~~~~~~~~~~~~~~~\n\nEdit the ``/etc/mole/input.conf``, just adding\n\n.. code-block:: ini\n\n [apache_log]\n type = tail\n source = /var/log/apache/access.log\n\nWe are defining a new input called *apache_log*, of type tail (that means\nthat we read the new lines in the file when written and handle rotate logs),\npointing to our log file in ``/var/log/apache/access.log``\n\nEdit the ``/etc/mole/index.conf``, just adding\n\n.. code-block:: ini\n\n [apache_log]\n path = /var/db/mole/apache_log\n\nWe are defining a new index. The index is the mole database where logs will\nbe stored in a proper format, so we can perform faster searches.\n\n2. Start daemons\n~~~~~~~~~~~~~~~~~~~~\n\n.. code-block:: bash\n\n $ mole-indexer -C /etc/mole\n $ mole-seeker -C /etc/mole\n\n3. Enjoy some searches\n~~~~~~~~~~~~~~~~~~~~~~\n\nFor example, get the top IP addresses which requested more traffic\n\n.. code-block:: bash\n\n $ mole 'input apache_log | sum bytes by src_ip | top'\n\n\nUnderstanding Mole Components\n-----------------------------\n\nThe mole pipeline is the responsible to read log items from a source,\nprocess then (and transform them if required) and, finally, return an\noutput. If output is not explicitly defined, use the best output format for\ncurrent console (serialize in network, just an printf in console).\n\n.. image:: http://yuml.me/diagram/scruffy;/class/[element]++-0..*%3E[input],%20[element]++-0..*%3E[index],%20[element]++-0..*%3E[parser],%20[index]-%3E[schema]\n :align: center\n\nThere are a few components which are interesting to know:\n\n**input:** The input are the responsible to read the log source, sources can\nbe of different kinds, such normal files, network stream, index file and so\non.\n\n**plotter:** The plotter main function is to split the source in logical\nlines. In a normal log file, each line in log is usually a new log entry,\nbut some other logs could be use a couple of lines to define the same\nlogical entry (i.e. java exceptions are usually in a number of lines).\n\n**parser:** Once the logical line is got, you need to known what is the\nmeaning of each field. The parser just assign names to fields using regular\nexpressions for that.\n\n**actions:** The actions are transformations, filters and in general any\nother action to take over the log dataset.\n\n**output:** The output just encapsulate the results of the actions in\na human (or machine) readable form. You can think the output as some kind of\nserialization.\n\nSo, the final pipeline in mole is something like that::\n\n | | | | ... |