{ "info": { "author": "tumbler", "author_email": "zimbler@gmail.com", "bugtrack_url": null, "classifiers": [], "description": "ALCO - Autonomous Log Collector and Observer\n============================================\n\n|PyPI version|\n\nWhat's the problem\n------------------\n\nThere is a widely used stack of technologies for parsing, collecting and\nanalysing logs - `ELK Stack `__. It has\nvery functional web interface, search cluster and a log transformation\ntool. Very cool, but:\n\n- It's Java with well-known requirements for memory and CPUs\n- It's ElasticSearch with it's requirements for disk space\n- It's Logstash which suddenly stops processing logs in some\n conditions.\n- It's Kibana with very cool RICH interface which looses on all counts\n to ``grep`` and ``less`` in a task of log reading and searching.\n\nIntroducing ALCO\n----------------\n\nALCO is a simple ELK analog which primary aim is to provide a online\nreplacement for ``grep`` and ``less``. Main features are:\n\n- Django application for incident analysis in distributed systems\n- schemeless full-text index with filtering and searching\n- configurable log collection and rotation from RabbitMQ messaging\n server\n- not a all-purpose monster\n\nTechnology stack\n----------------\n\nLet's trace log message path from some distributed system to ALCO web\ninterface.\n\n1. Python-based project calls ``logger.debug()`` method with text 'hello\n world'\n2. At startup time\n `Logcollect `__ library\n automatically configures python logging (or even\n `Django `__ and\n `Celery `__ one's) to send log\n messages to RabbitMQ server in JSON format readable both with ELK and\n ALCO projects.\n3. ALCO log collector binds a queue to RabbitMQ exchange and processes\n messages in a batch.\n4. It uses Redis to collect unique values for filterable fields and\n SphinxSearch to store messages in a realtime index.\n5. When a message is inserted to sphinxsearch, it contains indexed\n ``message`` field, timestamp information and schemeless JSON field\n named ``js`` with all log record attributes sent by python log.\n6. Django-based web interface provides API and client-side app for\n searching collected logs online.\n\nRequirements\n------------\n\n- Python 2.7 or 3.3+\n- `Logcollect `__ for python\n projects which logs are collected\n- `RabbitMQ `__ server for distributed log\n collection\n- `SphinxSearch `__ server 2.3 or later for\n log storage\n- `Redis `__ for SphinxSearch docid management and\n field values storage\n- `django-sphinxsearch `__\n as a database backend for ``Django>=1.8`` (will be available from\n PyPI)\n\nSetup\n-----\n\n1. You need to configure logcollect in analyzed projects (see\n `README `__).\n If RabbitMQ admin interface shows non-zero message flow in\n ``logstash`` exchange - \"It works\" :-)\n\n2. Install alco and it's requirements from PyPi\n\n .. code:: sh\n\n pip install alco\n\n3. Next, create django project, add ``sphinxsearch`` database\n connection and configure ``settings.py`` to enable alco applications\n\n .. code:: python\n\n # For SphinxRouter\n SPHINX_DATABASE_NAME = 'sphinx'\n\n DATABASES[SPHINX_DATABASE_NAME] = {\n 'ENGINE': 'sphinxsearch.backend.sphinx',\n 'HOST': '127.0.0.1',\n 'PORT': 9306,\n }\n }\n\n # Auto routing log models to SphinxSearch database\n DATABASE_ROUTERS = (\n 'sphinxsearch.routers.SphinxRouter',\n )\n\n INSTALLED_APPS += [\n 'rest_framework', # for API to work\n 'alco.collector',\n 'alco.grep'\n ]\n\n ROOT_URLCONF = 'alco.urls'\n\n4. Configure ALCO resources in ``settings.py``:\n\n .. code:: python\n\n ALCO_SETTINGS = {\n # log messaging server\n 'RABBITMQ': {\n 'host': '127.0.0.1',\n 'userid': 'guest',\n 'password': 'guest',\n 'virtual_host': '/'\n },\n\n # redis server\n 'REDIS': {\n 'host': '127.0.0.1',\n 'db': 0\n },\n # url for fetching sphinx.conf dynamically\n 'SPHINX_CONF_URL': 'http://127.0.0.1:8000/collector/sphinx.conf',\n # name of django.db.connection for SphinxSearch\n 'SPHINX_DATABASE_NAME': 'sphinx',\n # number of results in log view API\n 'LOG_PAGE_SIZE': 100\n }\n\n # override defaults for sphinx.conf template\n ALCO_SPHINX_CONF = {\n # local index definition defaults override \n 'index': {\n 'min_word_len': 8\n },\n # searchd section defaults override\n 'searchd': {\n 'dist_threads': 8\n }\n }\n\n5. Run ``syncdb`` or better ``migrate`` management command to create\n database tables.\n\n6. Run webserver and create a LoggerIndex from `django\n admin `__.\n\n7. Created directories for sphinxsearch:\n\n ::\n\n /var/log/sphinx/\n /var/run/sphinx/\n /data/sphinx/\n\n8. Next, configure sphinxsearch to use generated config:\n\n .. code:: sh\n\n\n searchd -c sphinx_conf.py\n\n ``sphinx_conf.py`` is a simple script that imports\n ``alco.sphinx_conf`` module which fetches generated ``sphinx.conf``\n from alco http api and created directories for SphinxSearch indices:\n\n .. code:: python\n\n #!/data/alco/virtualenv/bin/python\n\n # coding: utf-8\n import os\n os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'settings')\n\n from alco import sphinx_conf\n\n9. Run log collectors:\n\n .. code:: sh\n\n python manage.py start_collectors --no-daemon\n\n If it shows number of collected messages periodically - then log\n collecting is set up correctly.\n\n10. Configure system services to start subsystems automatically:\n\n - nginx or apache http server\n - django uwsgi backend\n - alco collectors (``start_collectors`` management command)\n - sphinxsearch, redis, default database for Django\n\n11. Open ``http://127.0.0.1:8000/grep//`` to read and\n search logs online.\n\nVirtualenv\n----------\n\nWe successfully configured SphinxSearch to use python from\n``virtualenv``, adding some environment variables to start script (i.e.\nFreeBSD rc.d script):\n\n.. code:: sh\n\n\n sphinxsearch_prestart ()\n {\n # nobody user has no HOME\n export PYTHON_EGG_CACHE=/tmp/.python-eggs\n # python path for virtualenv interpreter should be redeclared\n export PYTHONPATH=${venv_path}/lib/python3.4/:${venv_path}/lib/python3.4/site-packages/\n . \"${virtualenv_path}/bin/activate\" || err 1 \"Virtualenv is not found\"\n echo \"Virtualenv ${virtualenv_path} activated: `which python`\"\n\n }\n\nIn this case *shebang* for ``sphinx_conf.py`` must point virtualenv's\npython interpreter.\n\nProduction usage\n----------------\n\nFor now ALCO stack is tested in preproduction environment in our company\nand is actively developed. There are no reasons to say that it's not\nready for production usage.\n\n.. |PyPI version| image:: https://badge.fury.io/py/alco.svg\n :target: http://badge.fury.io/py/alco", "description_content_type": null, "docs_url": null, "download_url": "UNKNOWN", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "https://github.com/rutube/alco", "keywords": null, "license": "Beerware", "maintainer": null, "maintainer_email": null, "name": "alco", "package_url": "https://pypi.org/project/alco/", "platform": "UNKNOWN", "project_url": "https://pypi.org/project/alco/", "project_urls": { "Download": "UNKNOWN", "Homepage": "https://github.com/rutube/alco" }, "release_url": "https://pypi.org/project/alco/2.5.2/", "requires_dist": null, "requires_python": null, "summary": "Autonomous Log Collector and Observer", "version": "2.5.2" }, "last_serial": 2128848, "releases": { "0.5.0": [ { "comment_text": "", "digests": { "md5": "5b08e9c023d2a0d5a5a6e2230245581e", "sha256": "f9c3a3718a9920db41dd1f70fd12cb6e336de0714471a0b2769c521ffd0525cb" }, "downloads": -1, "filename": "alco-0.5.0.tar.gz", "has_sig": false, "md5_digest": "5b08e9c023d2a0d5a5a6e2230245581e", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 99731, "upload_time": "2015-07-31T15:04:22", "url": "https://files.pythonhosted.org/packages/ac/7f/83cd8d021697077e002e0516f66d159ee8c2334c28bce669b24cb8146b44/alco-0.5.0.tar.gz" } ], "0.5.1": [ { "comment_text": "", "digests": { "md5": "75c8f486a72ee8ff173ab4f8b9d5f3a6", "sha256": "122d1adf6716ccc1c14d354e063dd0a00b5aebff15b31e98cc022c115a0ff642" }, "downloads": -1, "filename": "alco-0.5.1.tar.gz", "has_sig": false, "md5_digest": "75c8f486a72ee8ff173ab4f8b9d5f3a6", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 99624, "upload_time": "2015-08-03T07:49:02", "url": "https://files.pythonhosted.org/packages/64/94/69d5726984e176260dc5e13d7e83f334adec815dd6ca4b0d630ed158ddc9/alco-0.5.1.tar.gz" } ], "0.5.2": [ { "comment_text": "", "digests": { "md5": "ae9e02a3d69a9b7d80cd094318fc7b4a", "sha256": "76ca82e875a515023c62a0b805c8ed51c6a0be0130dd1532630d9c7d4f5fd68f" }, "downloads": -1, "filename": "alco-0.5.2.tar.gz", "has_sig": false, "md5_digest": "ae9e02a3d69a9b7d80cd094318fc7b4a", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 99629, "upload_time": "2015-08-03T15:04:36", "url": "https://files.pythonhosted.org/packages/d3/ea/c925ff9f06734f2d9a7ab123a0c170037b05b87ba4e622483e83f0d0ca7a/alco-0.5.2.tar.gz" } ], "0.6.0": [ { "comment_text": "", "digests": { "md5": "012bf5a791965b63eeb5a026c9e01dfb", "sha256": "1662a63c3ba1c01ae807c046a23921a4a94017533cad96010e16442b5117f977" }, "downloads": -1, "filename": "alco-0.6.0.tar.gz", "has_sig": false, "md5_digest": "012bf5a791965b63eeb5a026c9e01dfb", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 99818, "upload_time": "2015-08-04T08:19:00", "url": "https://files.pythonhosted.org/packages/01/95/71aa9621b68e9496b8a0dd39ff28619457a38158cd5ebea250946fb52fe2/alco-0.6.0.tar.gz" } ], "0.6.1": [ { "comment_text": "", "digests": { "md5": "7aa86cf0a347cea6f19c020951653bbf", "sha256": "7e8bd29fa41db79a6f8958911a68e70a1c05390e03b9a3c45727d9571723ed3c" }, "downloads": -1, "filename": "alco-0.6.1.tar.gz", "has_sig": false, "md5_digest": "7aa86cf0a347cea6f19c020951653bbf", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 99904, "upload_time": "2015-08-04T09:51:18", "url": "https://files.pythonhosted.org/packages/17/8f/c1f343058f4117a14f36c0c6d4d69cf968183346e61c1bdc1fa7a6b27d98/alco-0.6.1.tar.gz" } ], "0.6.2": [ { "comment_text": "", "digests": { "md5": "d24910bf3382bdb05a1ef93aea9b4e97", "sha256": "4ed06b8ef939d5c3e0e047f46621f55cf2d73bd5d570763d57d4eaa8591ef258" }, "downloads": -1, "filename": "alco-0.6.2.tar.gz", "has_sig": false, "md5_digest": "d24910bf3382bdb05a1ef93aea9b4e97", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 99897, "upload_time": "2015-08-04T10:05:55", "url": "https://files.pythonhosted.org/packages/b5/24/a7b425b1084805250bbab9aebbc0542b978a1cfdcc63d742486dcc6ebeb2/alco-0.6.2.tar.gz" } ], "0.6.3": [ { "comment_text": "", "digests": { "md5": "74dc2f8c6c570bb0e7284ec540ba46c0", "sha256": "68e5425b99a8fe42049e046c5d6afb74dac8b348e7f41b0b1c3c400b59263b0e" }, "downloads": -1, "filename": "alco-0.6.3.tar.gz", "has_sig": false, "md5_digest": "74dc2f8c6c570bb0e7284ec540ba46c0", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 99900, "upload_time": "2015-08-04T10:08:38", "url": "https://files.pythonhosted.org/packages/79/4f/64b718fccf8a611c5aa811506a1d4d6eedb4b135541e74ac3a3e18abe8ff/alco-0.6.3.tar.gz" } ], "0.6.5": [ { "comment_text": "", "digests": { "md5": "53a31accc71b94ef4a621ad8edbe77aa", "sha256": "1c02b2b1891491df7f20b34afb59d825c216965a67bcef9e7035663a932faa62" }, "downloads": -1, "filename": "alco-0.6.5.tar.gz", "has_sig": false, "md5_digest": "53a31accc71b94ef4a621ad8edbe77aa", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 99923, "upload_time": "2015-08-04T11:24:44", "url": "https://files.pythonhosted.org/packages/fd/ea/2afb4054fa96b235dda5decdcb47d9591c0eb0dc87b6b79146feadd0f0a4/alco-0.6.5.tar.gz" } ], "0.7.0": [ { "comment_text": "", "digests": { "md5": "3725757b518ec7fe08e2f8bbfb2b6d4e", "sha256": "79f1043678c81d6b978cc2859a11bcadae618d0a45f4955ea60d338390f88f51" }, "downloads": -1, "filename": "alco-0.7.0.tar.gz", "has_sig": false, "md5_digest": "3725757b518ec7fe08e2f8bbfb2b6d4e", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 100408, "upload_time": "2015-08-24T06:32:41", "url": "https://files.pythonhosted.org/packages/ff/8c/92f14a24551644f509463904016554a789336f9eb58516d99af6e38946ae/alco-0.7.0.tar.gz" } ], "0.7.1": [ { "comment_text": "", "digests": { "md5": "884c2b92177a38ab7142dae34c7611bc", "sha256": "8c3839ab69cb6b67936a63b3fdb1ffbc746bf8a41bdaf8691e2ba8382342e460" }, "downloads": -1, "filename": "alco-0.7.1.tar.gz", "has_sig": false, "md5_digest": "884c2b92177a38ab7142dae34c7611bc", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 100564, "upload_time": "2015-08-24T07:06:05", "url": "https://files.pythonhosted.org/packages/09/88/f5a32f5bdee360fde2a76d97186d3d2661a3334f149137b07e7d29ca9bef/alco-0.7.1.tar.gz" } ], "0.7.2": [ { "comment_text": "", "digests": { "md5": "131f3d3edcb28ed6004c982aa174708d", "sha256": "2ce75260183477457469324150c13aa8a45cae226b24f2c73a6e3411bad3e827" }, "downloads": -1, "filename": "alco-0.7.2.tar.gz", "has_sig": false, "md5_digest": "131f3d3edcb28ed6004c982aa174708d", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 100875, "upload_time": "2015-09-07T10:23:55", "url": "https://files.pythonhosted.org/packages/e7/1d/40b836aa31fe9837559e1e63bb50c7c6352dd86e2a9b91d810404cca7394/alco-0.7.2.tar.gz" } ], "0.8.0": [ { "comment_text": "", "digests": { "md5": "1e04e6addf2a108557759ac971852ed0", "sha256": "24c22b7baac803a8206bf45914ce24c15254d0d3f3e74c20eea6a9810affc80e" }, "downloads": -1, "filename": "alco-0.8.0.tar.gz", "has_sig": false, "md5_digest": "1e04e6addf2a108557759ac971852ed0", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 100961, "upload_time": "2015-10-06T12:07:17", "url": "https://files.pythonhosted.org/packages/35/fe/21708ff4d5979026c3d4689ff5d28e68a82eb3aef4b1335b503babe5e9cf/alco-0.8.0.tar.gz" } ], "1.0.0": [ { "comment_text": "", "digests": { "md5": "e7f147842ba683223fca3402bf29dfab", "sha256": "1188bcdf7c605cd6bae86b957b3aace2a067aea6e944133e438fb01757fa4a7c" }, "downloads": -1, "filename": "alco-1.0.0.tar.gz", "has_sig": false, "md5_digest": "e7f147842ba683223fca3402bf29dfab", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 101260, "upload_time": "2016-03-16T10:43:17", "url": "https://files.pythonhosted.org/packages/44/25/ff4bc986182228f873ce0c8b1b5573055874b985ada1ee053239c8c94341/alco-1.0.0.tar.gz" } ], "1.0.1": [ { "comment_text": "", "digests": { "md5": "4100355f3c206dc6401bd6bf777033c5", "sha256": "902338644d7add88dab6d7440b211bb6db93e26630f61274a56b0e102a3fdba5" }, "downloads": -1, "filename": "alco-1.0.1.tar.gz", "has_sig": false, "md5_digest": "4100355f3c206dc6401bd6bf777033c5", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 101398, "upload_time": "2016-03-16T10:54:38", "url": "https://files.pythonhosted.org/packages/fe/e3/443f2dea9e18eaff0d001a7ea67ed87feb4273e34167de0afe1f656ae273/alco-1.0.1.tar.gz" } ], "2.0.0": [ { "comment_text": "", "digests": { "md5": "305e97ea0324ef330c34646d3a8d3818", "sha256": "47ce2ff4539f5e0bd53658b9ed52337a8a5fa285350690faf3572a5abe2efc37" }, "downloads": -1, "filename": "alco-2.0.0.tar.gz", "has_sig": false, "md5_digest": "305e97ea0324ef330c34646d3a8d3818", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 218272, "upload_time": "2016-04-08T16:07:25", "url": "https://files.pythonhosted.org/packages/6d/fb/d2a81c28a8343e7bf1baaf4d8ff82481d1de5d1be34f5e3aa46cb5a4068b/alco-2.0.0.tar.gz" } ], "2.1.0": [ { "comment_text": "", "digests": { "md5": "2154552c5a6eebb7904941b38083c979", "sha256": "b9bb2cde0ef39e2dff73fda413c2f838b3d6a7b2447642104528c038ea13d1ba" }, "downloads": -1, "filename": "alco-2.1.0.tar.gz", "has_sig": false, "md5_digest": "2154552c5a6eebb7904941b38083c979", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 218289, "upload_time": "2016-04-11T06:50:27", "url": "https://files.pythonhosted.org/packages/84/a7/c9253b4af6244ff65a45f7593c12f6f302e24aa00a84ef8a0f5c4f81a452/alco-2.1.0.tar.gz" } ], "2.2.0": [ { "comment_text": "", "digests": { "md5": "aaad5736d56a33eee8470d158fed782b", "sha256": "c84a93614be9d999b654083b5898df28aafe43330ca9f9d68b260ffb9ebf5ca6" }, "downloads": -1, "filename": "alco-2.2.0.tar.gz", "has_sig": false, "md5_digest": "aaad5736d56a33eee8470d158fed782b", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 218411, "upload_time": "2016-04-11T11:55:42", "url": "https://files.pythonhosted.org/packages/ad/f8/eb777ebe0ae80a79f3570b67da66d2fc87bd857373eac72273e5001851de/alco-2.2.0.tar.gz" } ], "2.3.0": [ { "comment_text": "", "digests": { "md5": "3418f01ef5400e8f6b0cd419be5e52cd", "sha256": "6e7a713b5e7b1c5812c11136f06739b8689202cf35a2d41fd7794b0ba72ac69b" }, "downloads": -1, "filename": "alco-2.3.0.tar.gz", "has_sig": false, "md5_digest": "3418f01ef5400e8f6b0cd419be5e52cd", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 218451, "upload_time": "2016-04-12T13:30:23", "url": "https://files.pythonhosted.org/packages/b6/e0/02250b9eedcb887df169f34ba380c8ca6efb078bac66162b05574521bf13/alco-2.3.0.tar.gz" } ], "2.4.0": [ { "comment_text": "", "digests": { "md5": "76114909f3fec02cc299a975c077f910", "sha256": "05f0c0213b07d506c9b964374ab554c784c87539d3722831a5d0c08bc7b9e8ca" }, "downloads": -1, "filename": "alco-2.4.0.tar.gz", "has_sig": false, "md5_digest": "76114909f3fec02cc299a975c077f910", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 218670, "upload_time": "2016-05-06T13:54:54", "url": "https://files.pythonhosted.org/packages/a1/db/45247ac9658d3b0b66e76b94d0069b4b06fa9da1ab4259aba3a0414aed57/alco-2.4.0.tar.gz" } ], "2.5.0": [ { "comment_text": "", "digests": { "md5": "eb8a926ce33c0565b515a364f0337658", "sha256": "c760947439a5802a77e535ffea741d225659e7d39c12145cf987003d84c4ed73" }, "downloads": -1, "filename": "alco-2.5.0.tar.gz", "has_sig": false, "md5_digest": "eb8a926ce33c0565b515a364f0337658", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 218684, "upload_time": "2016-05-23T09:50:47", "url": "https://files.pythonhosted.org/packages/6e/72/55c5b51a5d021a563e1c8cddf6abd42ed81ab59dcc49731fa996c99ecd2f/alco-2.5.0.tar.gz" } ], "2.5.1": [ { "comment_text": "", "digests": { "md5": "92110b497d24a03bf33c1f1cc1dabfd1", "sha256": "4f9ed4827a6bee63603472c076c77a9213787367ae3edb89847ee13510cc42a4" }, "downloads": -1, "filename": "alco-2.5.1.tar.gz", "has_sig": false, "md5_digest": "92110b497d24a03bf33c1f1cc1dabfd1", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 218689, "upload_time": "2016-05-23T09:53:47", "url": "https://files.pythonhosted.org/packages/48/63/33582aba8173c117e9a1f9d088fa72d3346f14a3395d5e693a28405b7ee7/alco-2.5.1.tar.gz" } ], "2.5.2": [ { "comment_text": "", "digests": { "md5": "55c2daba9563a40a064eb1a37397790f", "sha256": "f326145ccde9716a8401a2eb4ac2e5e8531856e91d70f5525e85451751445944" }, "downloads": -1, "filename": "alco-2.5.2.tar.gz", "has_sig": false, "md5_digest": "55c2daba9563a40a064eb1a37397790f", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 218682, "upload_time": "2016-05-23T09:54:49", "url": "https://files.pythonhosted.org/packages/32/0f/fb8727fbe399b412b1469280409c072400255fd661bc586b801409b60b1e/alco-2.5.2.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "55c2daba9563a40a064eb1a37397790f", "sha256": "f326145ccde9716a8401a2eb4ac2e5e8531856e91d70f5525e85451751445944" }, "downloads": -1, "filename": "alco-2.5.2.tar.gz", "has_sig": false, "md5_digest": "55c2daba9563a40a064eb1a37397790f", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 218682, "upload_time": "2016-05-23T09:54:49", "url": "https://files.pythonhosted.org/packages/32/0f/fb8727fbe399b412b1469280409c072400255fd661bc586b801409b60b1e/alco-2.5.2.tar.gz" } ] }