{ "info": { "author": "Marek Cermak", "author_email": "macermak@redhat.com", "bugtrack_url": null, "classifiers": [ "Development Status :: 2 - Pre-Alpha", "Framework :: IPython", "Framework :: Jupyter", "License :: OSI Approved :: MIT License", "Natural Language :: English", "Operating System :: OS Independent", "Programming Language :: JavaScript", "Programming Language :: Python :: 3.5", "Programming Language :: Python :: 3.6", "Programming Language :: Python :: 3.7", "Topic :: Scientific/Engineering :: Visualization", "Topic :: Utilities" ], "description": "# Thoth: Build Analysers\n\nThis is a set of analysers that will retrieve information from build logs.\n\n
\n\n### Lead by Example:\n\nConsider a log produced by the [pip](https://pypi.org/project/pip/) package installer:\n\n> We can also use [pipenv](https://pypi.org/project/pipenv/), see the [How it works](how-it-works) section\n\n
\n Log file content \n
\nProcessing /home/macermak/RedHat/aicoe/lab\nCollecting networkx (from thoth-lab==0.0.3)\nCollecting requests (from thoth-lab==0.0.3)\n  Using cached https://files.pythonhosted.org/packages/7d/e3/20f3d364d6c8e5d2353c72a67778eb189176f08e873c9900e10c0287b84b/requests-2.21.0-py2.py3-none-any.whl\nCollecting pandas (from thoth-lab==0.0.3)\n  Using cached https://files.pythonhosted.org/packages/e6/de/a0d3defd8f338eaf53ef716e40ef6d6c277c35d50e09b586e170169cdf0d/pandas-0.24.1-cp36-cp36m-manylinux1_x86_64.whl\nCollecting plotly (from thoth-lab==0.0.3)\n  Using cached https://files.pythonhosted.org/packages/fd/db/003b5cfbc710f4d4982440451185b952269e4080a57ae7e760a2ceb8ce0c/plotly-3.6.1-py2.py3-none-any.whl\nCollecting decorator>=4.3.0 (from networkx->thoth-lab==0.0.3)\n  Using cached https://files.pythonhosted.org/packages/f1/cd/7c8240007e9716b14679bc217a1baefa4432aa30394f7e2ec40a52b1a708/decorator-4.3.2-py2.py3-none-any.whl\nCollecting idna<2.9,>=2.5 (from requests->thoth-lab==0.0.3)\n  Using cached https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl\nCollecting urllib3<1.25,>=1.21.1 (from requests->thoth-lab==0.0.3)\n  Using cached https://files.pythonhosted.org/packages/62/00/ee1d7de624db8ba7090d1226aebefab96a2c71cd5cfa7629d6ad3f61b79e/urllib3-1.24.1-py2.py3-none-any.whl\nCollecting certifi>=2017.4.17 (from requests->thoth-lab==0.0.3)\n  Using cached https://files.pythonhosted.org/packages/9f/e0/accfc1b56b57e9750eba272e24c4dddeac86852c2bebd1236674d7887e8a/certifi-2018.11.29-py2.py3-none-any.whl\nCollecting chardet<3.1.0,>=3.0.2 (from requests->thoth-lab==0.0.3)\n  Using cached https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl\nCollecting python-dateutil>=2.5.0 (from pandas->thoth-lab==0.0.3)\n  Using cached https://files.pythonhosted.org/packages/41/17/c62faccbfbd163c7f57f3844689e3a78bae1f403648a6afb1d0866d87fbb/python_dateutil-2.8.0-py2.py3-none-any.whl\nCollecting pytz>=2011k (from pandas->thoth-lab==0.0.3)\n  Using cached https://files.pythonhosted.org/packages/61/28/1d3920e4d1d50b19bc5d24398a7cd85cc7b9a75a490570d5a30c57622d34/pytz-2018.9-py2.py3-none-any.whl\nCollecting numpy>=1.12.0 (from pandas->thoth-lab==0.0.3)\n  Using cached https://files.pythonhosted.org/packages/f5/bf/4981bcbee43934f0adb8f764a1e70ab0ee5a448f6505bd04a87a2fda2a8b/numpy-1.16.1-cp36-cp36m-manylinux1_x86_64.whl\nCollecting six (from plotly->thoth-lab==0.0.3)\n  Using cached https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl\nCollecting retrying>=1.3.3 (from plotly->thoth-lab==0.0.3)\nCollecting nbformat>=4.2 (from plotly->thoth-lab==0.0.3)\n  Using cached https://files.pythonhosted.org/packages/da/27/9a654d2b6cc1eaa517d1c5a4405166c7f6d72f04f6e7eea41855fe808a46/nbformat-4.4.0-py2.py3-none-any.whl\nCollecting traitlets>=4.1 (from nbformat>=4.2->plotly->thoth-lab==0.0.3)\n  Using cached https://files.pythonhosted.org/packages/93/d6/abcb22de61d78e2fc3959c964628a5771e47e7cc60d53e9342e21ed6cc9a/traitlets-4.3.2-py2.py3-none-any.whl\nCollecting jupyter-core (from nbformat>=4.2->plotly->thoth-lab==0.0.3)\n  Using cached https://files.pythonhosted.org/packages/1d/44/065d2d7bae7bebc06f1dd70d23c36da8c50c0f08b4236716743d706762a8/jupyter_core-4.4.0-py2.py3-none-any.whl\nCollecting jsonschema!=2.5.0,>=2.4 (from nbformat>=4.2->plotly->thoth-lab==0.0.3)\n  Using cached https://files.pythonhosted.org/packages/77/de/47e35a97b2b05c2fadbec67d44cfcdcd09b8086951b331d82de90d2912da/jsonschema-2.6.0-py2.py3-none-any.whl\nCollecting ipython-genutils (from nbformat>=4.2->plotly->thoth-lab==0.0.3)\n  Using cached https://files.pythonhosted.org/packages/fa/bc/9bd3b5c2b4774d5f33b2d544f1460be9df7df2fe42f352135381c347c69a/ipython_genutils-0.2.0-py2.py3-none-any.whl\nBuilding wheels for collected packages: thoth-lab\n  Running setup.py bdist_wheel for thoth-lab: started\n  Running setup.py bdist_wheel for thoth-lab: finished with status 'done'\n  Stored in directory: /tmp/pip-ephem-wheel-cache-_4menpyv/wheels/c6/6e/7d/ef751ae03b7d549816c6658c7f8065cb889a8de3f0c2b3ca6c\nSuccessfully built thoth-lab\nselinon 0.1.0rc9 requires celery>=4, which is not installed.\nthoth-analyzer 0.1.0 requires distro, which is not installed.\nthoth-storages 0.9.5 requires aiogremlin==3.2.6rc1, which is not installed.\ntensorflow-tensorboard 1.5.1 has requirement bleach==1.5.0, but you'll have bleach 2.1.4 which is incompatible.\ntensorflow-tensorboard 1.5.1 has requirement html5lib==0.9999999, but you'll have html5lib 1.0.1 which is incompatible.\ntensorboard 1.6.0 has requirement bleach==1.5.0, but you'll have bleach 2.1.4 which is incompatible.\ntensorboard 1.6.0 has requirement html5lib==0.9999999, but you'll have html5lib 1.0.1 which is incompatible.\nnotebook 5.7.0 has requirement jupyter-client>=5.2.0, but you'll have jupyter-client 5.1.0 which is incompatible.\nnotebook 5.7.0 has requirement pyzmq>=17, but you'll have pyzmq 16.0.3 which is incompatible.\nkaggle 1.4.7.1 has requirement urllib3<1.23.0,>=1.15, but you'll have urllib3 1.24.1 which is incompatible.\nbotocore 1.11.3 has requirement urllib3<1.24,>=1.20, but you'll have urllib3 1.24.1 which is incompatible.\nawscli 1.14.25 has requirement botocore==1.8.29, but you'll have botocore 1.11.3 which is incompatible.\nawscli 1.14.25 has requirement colorama<=0.3.7,>=0.2.5, but you'll have colorama 0.3.9 which is incompatible.\nawscli 1.14.25 has requirement PyYAML<=3.12,>=3.10, but you'll have pyyaml 3.13 which is incompatible.\nthoth-storages 0.9.5 has requirement yarl<1.2, but you'll have yarl 1.3.0 which is incompatible.\nspyder-kernels 0.2.6 has requirement jupyter-client>=5.2.3, but you'll have jupyter-client 5.1.0 which is incompatible.\nspyder-kernels 0.2.6 has requirement pyzmq>=17, but you'll have pyzmq 16.0.3 which is incompatible.\nrstcheck 2.2 has requirement sphinx<1.5,>=1.3, but you'll have sphinx 1.8.1 which is incompatible.\ngremlinpython 3.2.6 has requirement six==1.10.0, but you'll have six 1.12.0 which is incompatible.\ngremlinpython 3.2.6 has requirement tornado==4.4.1, but you'll have tornado 4.5.2 which is incompatible.\nInstalling collected packages: decorator, networkx, idna, urllib3, certifi, chardet, requests, six, python-dateutil, pytz, numpy, pandas, retrying, ipython-genutils, traitlets, jupyter-core, jsonschema, nbformat, plotly, thoth-lab\n  Found existing installation: decorator 4.3.2\n    Uninstalling decorator-4.3.2:\n      Successfully uninstalled decorator-4.3.2\n  Found existing installation: networkx 2.2\n    Uninstalling networkx-2.2:\n      Successfully uninstalled networkx-2.2\n  Found existing installation: idna 2.8\n    Uninstalling idna-2.8:\n      Successfully uninstalled idna-2.8\n  Found existing installation: urllib3 1.24.1\n    Uninstalling urllib3-1.24.1:\n      Successfully uninstalled urllib3-1.24.1\n  Found existing installation: certifi 2018.11.29\nCannot uninstall 'certifi'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.\nYou are using pip version 18.1, however version 19.0.2 is available.\nYou should consider upgrading via the 'pip install --upgrade pip' command.\n  
\n
\n\nIn the log file, there is a line 76 which contains the information about the package which broke the installation: `certifi`. We want to extract this information. In the most common case, we want to get a **report** containing the observations about this particular build log.\n\n```python\nfrom thoth.build_analysers.analysis import build_breaker_report\n\nlog: str = ...\nreport: dict = build_breaker_report(log)\n```\n\nThe result is a dictionary which can further be formatted by the convenient `build_breaker_format_report` function.\n\n```python\nfrom thoth.build_analysers.analysis import build_breaker_format_report\n\nprint(build_breaker_format_report(report))\n```\n\nAnd you should see something like this:\n```\nBuild breaker:\n\n {\n \"already_satisfied\": null,\n \"source\": \"thoth-lab\",\n \"target\": \"certifi\",\n \"version_installed\": null,\n \"version_specified\": \">=2017.4.17\"\n }\n\nProbable reason:\n\n 76: Cannot uninstall 'certifi'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.\n\nCandidates:\n\n [\n \"76: Cannot uninstall 'certifi'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.\"\n ]\n\n```\n\n
\n\n### How it works\n\nThe logic is quite simple (and also a bit naive). Really, no rocket science, just a lot of string matching.\n\n##### The intuition\n\nThe approach currently handles logs produced by pip and pipenv package installers. For both of them, there is a set of patterns which, if found in a log, potentially indicate an error. These patterns are strings formatted according to the [PEP3101](https://www.python.org/dev/peps/pep-3101/) string formatting syntax and are stored in the [`resources`](/thoth/build_analysers/resources) folder as [pandas DataFrames](https://pandas.pydata.org/pandas-docs/version/0.23.4/generated/pandas.DataFrame.html).\n\nWe evaluate the 'goodness' of a match for each line of the log file with respect to the relevant set of patterns and label them based on the given *thresholds*. Currently, the `ERROR` threshold is set to the score of *>=0.6* on the scale from *0* through *1.0*. Finally, we try to search for package names in the log file to construct a dependency table. We filter the `ERROR`s containing a package name and we've got ourselves a list of *build breakers*.\n\n
\n\n### Installation\n\nThe package is available on [PyPI](https://pypi.org/project/thoth-build-analysers/):\n\n```bash\npip install thoth-build-analysers\n```\n\nAnd you're good to go :thumbsup:\n\n
\n\n---\n\n> Author: Marek Cermak , @AICoE - Project Thoth\n\n\n", "description_content_type": "text/markdown", "docs_url": null, "download_url": "", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "https://github.com/thoth-station/build-analysers", "keywords": "", "license": "GPLv3", "maintainer": "", "maintainer_email": "", "name": "thoth-build-analysers", "package_url": "https://pypi.org/project/thoth-build-analysers/", "platform": "", "project_url": "https://pypi.org/project/thoth-build-analysers/", "project_urls": { "Homepage": "https://github.com/thoth-station/build-analysers" }, "release_url": "https://pypi.org/project/thoth-build-analysers/0.1.0/", "requires_dist": [ "astpath", "parse", "numpy", "pandas", "networkx", "thoth-lab" ], "requires_python": "", "summary": "Build analysis library and tools to handle and process build logs.", "version": "0.1.0" }, "last_serial": 5322548, "releases": { "0.1.0": [ { "comment_text": "", "digests": { "md5": "eb8783f59060fd92d1c4e64bbf705cd9", "sha256": "7c9766828c6132e6dad8ae56db1db0af668ab07aa7370b6ea5ff0a65f64303c3" }, "downloads": -1, "filename": "thoth_build_analysers-0.1.0-py3-none-any.whl", "has_sig": false, "md5_digest": "eb8783f59060fd92d1c4e64bbf705cd9", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": null, "size": 42963, "upload_time": "2019-05-27T14:21:24", "url": "https://files.pythonhosted.org/packages/d9/5e/67fd6b6a1a0392083b306764c0f18c65e415c26d75c61173628f01409115/thoth_build_analysers-0.1.0-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "268b1a381eebe904e71564b119b35140", "sha256": "375faf23f0faca723710b78abaacbb8bc62e2b903792ded1a362d6d7ad3b4303" }, "downloads": -1, "filename": "thoth-build-analysers-0.1.0.tar.gz", "has_sig": false, "md5_digest": "268b1a381eebe904e71564b119b35140", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 744974, "upload_time": "2019-05-27T14:21:26", "url": "https://files.pythonhosted.org/packages/73/45/d36b403f07a89f20c9441cfbf67e178330f4c6291e46792b0efcc5058658/thoth-build-analysers-0.1.0.tar.gz" } ], "0.1.0rc0": [ { "comment_text": "", "digests": { "md5": "a09afe39df6a36b13be7d24a3585fc77", "sha256": "d9c6c4baeb9c29ced6180601e161837d44d6ccb3f590a9603d7947f9457328b1" }, "downloads": -1, "filename": "thoth_build_analysers-0.1.0rc0-py3-none-any.whl", "has_sig": false, "md5_digest": "a09afe39df6a36b13be7d24a3585fc77", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": null, "size": 43000, "upload_time": "2019-05-27T13:57:10", "url": "https://files.pythonhosted.org/packages/7d/4c/1278619f79ea0b5809ca07a02e35fd1d3d07d9c54c7ede465810208bf1c9/thoth_build_analysers-0.1.0rc0-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "e5d1cc142b4c368bf3111e39ff68f8a4", "sha256": "f5227263751dbdc4ec96caf9f2b40b99737ae564ff8c39bcde48d9c34b77ab39" }, "downloads": -1, "filename": "thoth-build-analysers-0.1.0rc0.tar.gz", "has_sig": false, "md5_digest": "e5d1cc142b4c368bf3111e39ff68f8a4", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 744985, "upload_time": "2019-05-27T13:57:13", "url": "https://files.pythonhosted.org/packages/ef/d5/43661bd81eeac6ae3710eecf979934e353bfa6766d20d941a276d7bf402d/thoth-build-analysers-0.1.0rc0.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "eb8783f59060fd92d1c4e64bbf705cd9", "sha256": "7c9766828c6132e6dad8ae56db1db0af668ab07aa7370b6ea5ff0a65f64303c3" }, "downloads": -1, "filename": "thoth_build_analysers-0.1.0-py3-none-any.whl", "has_sig": false, "md5_digest": "eb8783f59060fd92d1c4e64bbf705cd9", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": null, "size": 42963, "upload_time": "2019-05-27T14:21:24", "url": "https://files.pythonhosted.org/packages/d9/5e/67fd6b6a1a0392083b306764c0f18c65e415c26d75c61173628f01409115/thoth_build_analysers-0.1.0-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "268b1a381eebe904e71564b119b35140", "sha256": "375faf23f0faca723710b78abaacbb8bc62e2b903792ded1a362d6d7ad3b4303" }, "downloads": -1, "filename": "thoth-build-analysers-0.1.0.tar.gz", "has_sig": false, "md5_digest": "268b1a381eebe904e71564b119b35140", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 744974, "upload_time": "2019-05-27T14:21:26", "url": "https://files.pythonhosted.org/packages/73/45/d36b403f07a89f20c9441cfbf67e178330f4c6291e46792b0efcc5058658/thoth-build-analysers-0.1.0.tar.gz" } ] }