{
    "info": {
        "author": "Tristan Cacqueray",
        "author_email": "tdecacqu@redhat.com",
        "bugtrack_url": null,
        "classifiers": [
            "Intended Audience :: Developers",
            "Intended Audience :: System Administrators",
            "License :: OSI Approved :: Apache Software License",
            "Programming Language :: Python",
            "Programming Language :: Python :: 3.5",
            "Topic :: Scientific/Engineering"
        ],
        "description": "logreduce - extract anomaly from log files\n==========================================\n\nBased on success logs, logreduce highlights useful text in failed logs.\nThe goal is to save time in finding a failure's root cause.\n\nOn average, learning run at 2000 lines per second, and\ntesting run at 1300 lines per seconds.\n\n\nHow it works\n------------\n\nlogreduce uses a *model* to learn successful logs and detect novelties in\nfailed logs:\n\n* Random words are manually removed using regular expression\n* Then lines are converted to a matrix of token occurrences\n  (using **HashingVectorizer**),\n* An unsupervised learner implements neighbor searches\n  (using **NearestNeighbors**).\n\n\nCaveats\n-------\n\nThis method doesn't work when debug content is only included in failed logs.\nTo successfully detect anomalies, failed and success logs needs to be similar,\notherwise the extra informations in failed logs will be considered anomalous.\n\nFor example this happens with testr where success logs only contains 'SUCCESS'.\n\n\nInstall\n-------\n\n* Fedora:\n\n.. code-block:: console\n\n  sudo dnf install -y python3-scikit-learn\n  git clone https://softwarefactory-project.io/r/logreduce\n  pushd logreduce\n  python3 setup.py develop --user\n  popd\n\n\n* openSUSE:\n\n.. code-block:: console\n\n  sudo zypper install python3-scikit-learn\n  git clone https://softwarefactory-project.io/r/logreduce\n  pushd logreduce\n  python3 setup.py develop --user\n  popd\n\n\n* Pip:\n\n.. code-block:: console\n\n  pip install --user logreduce\n\n\nUsage\n-----\n\nLogreduce needs a **baseline** for success log training, and a **target**\nfor the log to reduce.\n\nLogreduce prints anomalies on the console, the log files are not modified:\n\n.. code-block:: console\n\n  \"%(distance)f | %(log_path)s:%(line_number)d: %(log_line)s\"\n\nLocal file usage\n................\n\n* Compare two files or directories without building a model:\n\n.. code-block:: console\n\n  $ logreduce diff testr-nodepool-01/output.good testr-nodepool-01/output.fail\n  0.232 | testr-nodepool-01/output.fail:0677:  File \"voluptuous/schema_builder.py\", line 370, in validate_mapping\n  0.462 | testr-nodepool-01/output.fail:0678:    raise er.MultipleInvalid(errors)\n  0.650 | testr-nodepool-01/output.fail:0679:  voluptuous.error.MultipleInvalid: required key not provided @ data['providers'][2]['cloud']\n\n* Compare two files or directories:\n\n.. code-block:: console\n\n  $ logreduce dir preprod-logs/ /var/log/\n\n\n* Or build a model first and run it separately:\n\n.. code-block:: console\n\n  $ logreduce dir-train sosreport.clf old-sosreport/ good-sosreport/\n  $ logreduce dir-run sosreport.clf new-sosreport/\n\n\nZuul job usage\n..............\n\nLogreduce can query Zuul build database to train a model.\n\n* Extract novelty from a job logs:\n\n.. code-block:: console\n\n  $ logreduce job http://logs.openstack.org/...\n\n  # Reduce comparaison to a single project (e.g. for tox jobs)\n  $ logreduce job --project openstack/nova http://logs.openstack.org/...\n\n  # Compare using many baselines\n  $ logreduce job --count 10 http://logs.openstack.org/...\n\n  # Include job artifacts\n  $ logreduce job --include-path logs/ http:/logs.openstack.org/...\n\n* Or build a model first and run it separately:\n\n.. code-block:: console\n\n  $ logreduce job-train --job job_name job_name.clf\n  $ logreduce job-run job_name.clf http://logs.openstack.org/.../\n\n\nJournald usage\n..............\n\nLogreduce can look for anomaly in journald, comparing the last day/week/month\nto the previous one:\n\n* Extract novelty from last day journal:\n\n.. code-block:: console\n\n  $ logreduce journal --range day\n\n* Build a model using journal of last month and look for novelty in last week:\n\n.. code-block:: console\n\n  $ logreduce journal-train --range month good-journal.clf\n  $ logreduce journal-run --range week good-journal.clf\n\n\nServer component\n----------------\n\nA server component may be deployed to build an anomaly database and produce\ndataset. This initial implementation is focused on Zuul builds and it\ndoesn't support importing arbritary files yet.\nMore details in this specification https://review.openstack.org/#/c/581214:\n\nComponents list:\n\n* logreduce-server: the REST and Gearman server\n* logreduce-worker: job executor\n* logreduce-client: client cli\n* logreduce-webui: logreduce web interface\n\nAPI\n...\n\n* PUT /anomaly/new: receive user report request from os_loganalyze\n* PUT /anomaly: import an anomaly report (json file generated by standalone cli)\n* GET /anomaly/{anomaly_id}: return an anomaly details\n* POST /anomaly/{anomaly_id}: update anomaly status\n* POST /anomaly/{anomaly_id}/logfile/{logfile_id}: update scores\n* DELETE /anomaly/{anomaly_id}/logfile/{logfile_id}: remove a file\n* GET /anomalies: return the list of anomalies\n* GET /status: return the list of worker jobs\n\n\nlogreduce-tests\n---------------\n\nThis package contains tests data for different type of log such as testr\nor syslog. Each tests includes a pre-computed list of the anomalies in log\nfailures.\n\nThis package also includes a command line utility to run logreduce against all\ntests data and print a summary of its performance.\n\n\nTest format\n...........\n\nEach tests case is composed of:\n\n* A *.good* file (or directory) that holds the baseline\n* A *.fail* file (or directory)\n* A *info.yaml* file that describe expected output:\n\n.. code-block:: yaml\n\n  threshold: float # set the distance threshold for the test\n  anomalies:\n    - optional: bool  # to define minor anomalies not considered false positive\n      lines: |        # the expected lines to be highlighted\n        Traceback...\n        RuntimeError...\n\n\nEvaluate\n........\n\nTo run the evaluation, first install logreduce-tests:\n\n.. code-block:: console\n\n  git clone https://softwarefactory-project.io/r/logreduce-tests\n  pushd logreduce-tests\n  python3 setup.py develop --user\n\nlogreduce-tests expect tests directories as argument:\n\n.. code-block:: console\n\n  $ logreduce-tests tests/testr-zuul-[0-9]*\n  [testr-zuul-01]: 100.00% accuracy,  5.00% false-positive\n  [testr-zuul-02]:  80.00% accuracy,  0.00% false-positive\n  ...\n  Summary:  90.00% accuracy,  2.50% false-positive\n\nAdd --debug to display false positive and missing chunks.\n\n\nTODOs\n-----\n\n* Add terminal colors output\n* Add progress bar\n* Better differentiate training debug from testing debug\n* Add a starting log line and report written\n* Add tarball traversal in utils.files_iterator\n* Add logstash filter module\n* Improve tokenization tests\n\n\nRoadmap\n-------\n* Add daemon worker mode with MQTT event listener\n* Discard files that are 100% anomalous\n* Report mean diviation instead of absolute distances\n* Investigate second stage model\n\n\nContribute\n----------\n\nContribution are most welcome, use **git-review** to propose a change.\nSetup your ssh keys after sign in https://softwarefactory-project.io/auth/login\n\n\n\n",
        "description_content_type": "",
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://logreduce.softwarefactory-project.io/",
        "keywords": "machine learning,ci,anomaly detection",
        "license": "",
        "maintainer": "",
        "maintainer_email": "",
        "name": "logreduce",
        "package_url": "https://pypi.org/project/logreduce/",
        "platform": "",
        "project_url": "https://pypi.org/project/logreduce/",
        "project_urls": {
            "Homepage": "https://logreduce.softwarefactory-project.io/"
        },
        "release_url": "https://pypi.org/project/logreduce/0.4.0/",
        "requires_dist": [
            "PyYAML",
            "aiohttp",
            "alembic",
            "cherrypy",
            "gear",
            "numpy",
            "requests",
            "routes",
            "scikit-learn",
            "scipy",
            "sqlalchemy",
            "voluptuous"
        ],
        "requires_python": "",
        "summary": "Extract anomalies from log files",
        "version": "0.4.0"
    },
    "last_serial": 4464546,
    "releases": {
        "0.0.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "6ee4588e7c42ea50cc4944272f2e165a",
                    "sha256": "6d734e27f4bb4359377d648e1113e5c89f1b517bb3d1310dbf38183402e1715c"
                },
                "downloads": -1,
                "filename": "logreduce-0.0.1-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "6ee4588e7c42ea50cc4944272f2e165a",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 24714,
                "upload_time": "2017-12-27T05:40:27",
                "url": "https://files.pythonhosted.org/packages/17/a2/d9908bad5fa277cac726476f7b132f4490ace22996c58d8166a4ac656011/logreduce-0.0.1-py2.py3-none-any.whl"
            }
        ],
        "0.1.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "8feff8627c9eb7f1ee484dff3ea623a5",
                    "sha256": "d9e9699a0d8932c0ef262e52bec6d3d9971ed3f2f51e2650dddb2fe7ab9df804"
                },
                "downloads": -1,
                "filename": "logreduce-0.1.0-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "8feff8627c9eb7f1ee484dff3ea623a5",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 28296,
                "upload_time": "2018-03-02T04:40:48",
                "url": "https://files.pythonhosted.org/packages/8b/26/688f7cd5c5879a5001e648a29ac7dd000704e7f1657cbdd34d4df5f9f7eb/logreduce-0.1.0-py2.py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "25ad067a6763d18db6ef37dbfb2b52f3",
                    "sha256": "7a84dcdbd46fba68d74621e9a24d1b1447d77b2a74ac21b17eb747f6f5e7e81f"
                },
                "downloads": -1,
                "filename": "logreduce-0.1.0.tar.gz",
                "has_sig": false,
                "md5_digest": "25ad067a6763d18db6ef37dbfb2b52f3",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 33521,
                "upload_time": "2018-03-02T04:40:51",
                "url": "https://files.pythonhosted.org/packages/f7/76/ace4201382333ecc6ea6001f2831fcfb189f0fea8629bd85318bd4ad086f/logreduce-0.1.0.tar.gz"
            }
        ],
        "0.1.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "18b57a4efafae2668b077355a4df48c6",
                    "sha256": "4543e842feb66bd9783233d8e275dcbb37ff98a5a200a50b1dc3446024254339"
                },
                "downloads": -1,
                "filename": "logreduce-0.1.1-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "18b57a4efafae2668b077355a4df48c6",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 25329,
                "upload_time": "2018-04-04T06:03:22",
                "url": "https://files.pythonhosted.org/packages/b0/37/07f0f1774c602500f235eca56a162e39b116e2e126bd4d2f623252861dc1/logreduce-0.1.1-py2.py3-none-any.whl"
            }
        ],
        "0.1.2": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "e6a99c9cea3a752503b32577efa17de6",
                    "sha256": "105cd1807f72c5626087d0fb87228fe264416c87af87fe436ac54a332e820707"
                },
                "downloads": -1,
                "filename": "logreduce-0.1.2-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "e6a99c9cea3a752503b32577efa17de6",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 29235,
                "upload_time": "2018-04-26T04:07:33",
                "url": "https://files.pythonhosted.org/packages/68/40/c2652675e170c2f49f74a9745eb393ab71d37e24357ababec0367db0ddcf/logreduce-0.1.2-py2.py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "c3736c4955130f469e26792061784c0e",
                    "sha256": "3f31691f9b2a741bb381e96551346504133b0c4e0a6047e3d11f3a389fac1ec1"
                },
                "downloads": -1,
                "filename": "logreduce-0.1.2.tar.gz",
                "has_sig": false,
                "md5_digest": "c3736c4955130f469e26792061784c0e",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 34971,
                "upload_time": "2018-04-26T04:07:35",
                "url": "https://files.pythonhosted.org/packages/f9/ef/0720c3a82258bdd5eb3f7876babea01ffef920c2ebeae6eb86f65b0fe235/logreduce-0.1.2.tar.gz"
            }
        ],
        "0.1.3": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "7553cf7d07b34e4c1bf67c7e39bfe06c",
                    "sha256": "889bff4f3b035a465957b3333c24e780752c82f6bfec478b86c2c4b21ccc6309"
                },
                "downloads": -1,
                "filename": "logreduce-0.1.3-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "7553cf7d07b34e4c1bf67c7e39bfe06c",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 29508,
                "upload_time": "2018-07-04T06:41:35",
                "url": "https://files.pythonhosted.org/packages/66/cf/abfc9d386e3bcc9048584cea1883d805ff0e6033b57ffa9ed37fa2867669/logreduce-0.1.3-py2.py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "1cc11d18a4baf9f03ee216a005c76098",
                    "sha256": "74ae79e40d422b21a0a1f9ed1cbfe06b2236a22bfe47e2ccab9f023c4f2d1e32"
                },
                "downloads": -1,
                "filename": "logreduce-0.1.3.tar.gz",
                "has_sig": false,
                "md5_digest": "1cc11d18a4baf9f03ee216a005c76098",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 35552,
                "upload_time": "2018-07-04T06:41:39",
                "url": "https://files.pythonhosted.org/packages/d7/13/c5b3f61a5ed3b217b80f135d6e1276ce48bdc9cde18b5393e054f69bc474/logreduce-0.1.3.tar.gz"
            }
        ],
        "0.2.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "58bdd4f29411a0bafbd7a3919415807c",
                    "sha256": "b83fcb87b28d1b53836d56b957abaa0df1f2513d69b3b30a6034bc6b8108ac48"
                },
                "downloads": -1,
                "filename": "logreduce-0.2.0-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "58bdd4f29411a0bafbd7a3919415807c",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 36008,
                "upload_time": "2018-08-27T02:25:55",
                "url": "https://files.pythonhosted.org/packages/19/21/492302615ab98c4129c60e56921b9edadeb452c7297ea68e58ecb899ffb8/logreduce-0.2.0-py2.py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "976fa500275101a39f481cc02e5e4b3f",
                    "sha256": "2c2fd6f3e634f54e385f0b7154dcfc0dd5830e9876a526699f5570f601f72a1b"
                },
                "downloads": -1,
                "filename": "logreduce-0.2.0.tar.gz",
                "has_sig": false,
                "md5_digest": "976fa500275101a39f481cc02e5e4b3f",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 41830,
                "upload_time": "2018-08-27T02:25:58",
                "url": "https://files.pythonhosted.org/packages/6d/57/030c1c4699e1ec9557714f46a3851bfbe9d989a054a9bd87867146c7e55e/logreduce-0.2.0.tar.gz"
            }
        ],
        "0.3.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "e3488851d18f3e354958345ef4f5c210",
                    "sha256": "4bc846af5329f63047d1f68e0c0d631b5418e197c62225011f6461ac706e8ef5"
                },
                "downloads": -1,
                "filename": "logreduce-0.3.0-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "e3488851d18f3e354958345ef4f5c210",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 62116,
                "upload_time": "2018-10-25T09:24:05",
                "url": "https://files.pythonhosted.org/packages/75/9b/38241e24435b44d21275d57a23b47ad22589d090f10c47db71fc384b4608/logreduce-0.3.0-py2.py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "d4118588b9ad6a0c6e490e727c135b7b",
                    "sha256": "0a8d5830c1c5317f8ff62ffd905241af1ebacd5103e1a42fadfe49a5d66dd7bb"
                },
                "downloads": -1,
                "filename": "logreduce-0.3.0.tar.gz",
                "has_sig": false,
                "md5_digest": "d4118588b9ad6a0c6e490e727c135b7b",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 155180,
                "upload_time": "2018-10-25T09:24:08",
                "url": "https://files.pythonhosted.org/packages/bd/32/f65732cbb0146ea5258057f55904fba9c52c17bb11c6237f71924c278f66/logreduce-0.3.0.tar.gz"
            }
        ],
        "0.4.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "b0e72a85517f7fe9bbb5c20d120208f6",
                    "sha256": "258c6773c6a79bd1ff4b80b5cebca852cb75ac29deb770897d1971d93c88c745"
                },
                "downloads": -1,
                "filename": "logreduce-0.4.0-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "b0e72a85517f7fe9bbb5c20d120208f6",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 63942,
                "upload_time": "2018-11-08T07:40:16",
                "url": "https://files.pythonhosted.org/packages/2e/cd/7d8dd0421d3a8475d81711f1080b47f5a66ce89a62ccfc63df0adf2259e4/logreduce-0.4.0-py2.py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "12cc63aed1b2593a034c96895fd5e024",
                    "sha256": "ce5eaa80e1ac37f740bb82413a0cf156429e9e32446f3e4363ae82988315d532"
                },
                "downloads": -1,
                "filename": "logreduce-0.4.0.tar.gz",
                "has_sig": false,
                "md5_digest": "12cc63aed1b2593a034c96895fd5e024",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 157878,
                "upload_time": "2018-11-08T07:40:19",
                "url": "https://files.pythonhosted.org/packages/6e/1c/cfe60594bf0de4459be46d3bb7d09f8ba1128a82c1d9340b93c8b0985e97/logreduce-0.4.0.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "b0e72a85517f7fe9bbb5c20d120208f6",
                "sha256": "258c6773c6a79bd1ff4b80b5cebca852cb75ac29deb770897d1971d93c88c745"
            },
            "downloads": -1,
            "filename": "logreduce-0.4.0-py2.py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "b0e72a85517f7fe9bbb5c20d120208f6",
            "packagetype": "bdist_wheel",
            "python_version": "py2.py3",
            "requires_python": null,
            "size": 63942,
            "upload_time": "2018-11-08T07:40:16",
            "url": "https://files.pythonhosted.org/packages/2e/cd/7d8dd0421d3a8475d81711f1080b47f5a66ce89a62ccfc63df0adf2259e4/logreduce-0.4.0-py2.py3-none-any.whl"
        },
        {
            "comment_text": "",
            "digests": {
                "md5": "12cc63aed1b2593a034c96895fd5e024",
                "sha256": "ce5eaa80e1ac37f740bb82413a0cf156429e9e32446f3e4363ae82988315d532"
            },
            "downloads": -1,
            "filename": "logreduce-0.4.0.tar.gz",
            "has_sig": false,
            "md5_digest": "12cc63aed1b2593a034c96895fd5e024",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 157878,
            "upload_time": "2018-11-08T07:40:19",
            "url": "https://files.pythonhosted.org/packages/6e/1c/cfe60594bf0de4459be46d3bb7d09f8ba1128a82c1d9340b93c8b0985e97/logreduce-0.4.0.tar.gz"
        }
    ]
}