{ "info": { "author": "Vladimir Shulyak", "author_email": "vladimir@shulyak.net", "bugtrack_url": null, "classifiers": [ "Development Status :: 4 - Beta", "License :: OSI Approved :: MIT License", "Operating System :: OS Independent", "Programming Language :: Python :: 3.6", "Programming Language :: Python :: 3.7", "Programming Language :: Python :: 3.8" ], "description": "# ```ts-eval``` Time Series analysis and evaluation tools\n\n[![pypi](https://img.shields.io/pypi/v/ts-eval)](https://pypi.org/project/ts-eval/)\n[![Build Status](https://travis-ci.org/vshulyak/ts-eval.svg?branch=master)](https://travis-ci.org/vshulyak/ts-eval)\n[![codecov](https://codecov.io/github/vshulyak/ts-eval/branch/master/graph/badge.svg)](https://codecov.io/github/vshulyak/ts-eval)\n[![python3](https://img.shields.io/pypi/pyversions/ts-eval)](https://www.python.org/downloads/release/python-374/)\n[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)\n[![License: MIT](https://img.shields.io/pypi/l/ts-eval)](https://github.com/vshulyak/ts-eval/blob/master/LICENSE)\n[![Contributions welcome](https://img.shields.io/badge/contributions-welcome-brightgreen.svg?style=flat)](https://github.com/vshulyak/ts-eval/issues)\n\n---\nA set of tools to make time series analysis easier.\n\n## \ud83e\udde9 Current features\n\n* **N-step ahead rolling origin time series evaluation** \u2013 using a Jupyter widget.\n* **Friedman / Nemenyi rank test (posthoc)** \u2013 to see which model statistically performs better.\n* **Relative Metrics** \u2013 rMSE, rMAE + Forecasted Value analogues.\n* **Prediction Interval Metrics** \u2013 MIS, rMIS, FVrMIS\n* **Fixed fourier series generation** \u2013 fixed in time according to pandas index\n* **Naive/Seasonal models for baseline predictions** (with prediction intervals)\n* **Statsmodels n-step evaluation** \u2013 helper functions to evaluate n-step ahead forecasts using Statsmodels models or naive/seasonal naive models.\n\n## \ud83d\udc69\ud83c\udffe\u200d\ud83c\udfa8 Widget Preview\n\nIn:\n```\nTSMetrics(target, sm_seas, default)\n.use_reference(snaive)\n.for_horizons(0, 1, 5, 23)\n.for_time_slices(time_slices.all, time_slices.weekend)\n.with_description()\n.with_prediction_rankings(mtx.FVrMSE, mtx.FVrMIS)\n.with_predictions_plot()\n.show()\n```\n\nOut:\n![Demo Screenshot](images/demo_screenshot.png)\n\n## \ud83d\udc69\ud83c\udffe\u200d\ud83d\ude92 Demo\nFor a more elaborate example, please check out the [Demo Notebook](https://nbviewer.jupyter.org/github/vshulyak/ts-eval/blob/master/examples/basic_usage.ipynb).\n\nAlternatively, check out [interactive Binder demo](https://mybinder.org/v2/gh/vshulyak/ts-eval/master?filepath=examples%2Fbasic_usage.ipynb)\n\n## \ud83e\udd26\ud83c\udffe\u200d Motivation\n\nWhile working on a long term time series analysis project, I had a need to summarize and store performance metrics\nof different models and compare them. As it's daunting to do this across dozens of notebooks, I huddled over some code\nto do what I want in a few lines of code.\n\n## \ud83d\udc69\ud83c\udffe\u200d\ud83d\ude80 Installation\n\n pip install ts-eval\n\n\n## \ud83d\udccb Release Planning:\n\n* **Release 0.3**\n * use pandas better for dataframe styles / viz https://pandas.pydata.org/pandas-docs/stable/user_guide/style.html\n * api like (viz1 | viz2 | viz3 ) / viz4 (patchwork R package)\n * CRPS evaluation\n * dynamic insample forecast for statsmodels\n * PI coverage estimation is really needed (in %)\n * predictions based on loess decomposition https://github.com/jrmontag/STLDecompose\n * dataset description: first index - last index sample (dates or ids)\n * transform with callback\n * altair-like API where you can combine components with +\n * success rate of prediction intervals like here http://freerangestats.info/blog/2016/01/30/hybrid-forecasts\n * describe case when MIS rankings are better for one dataset, but its mean is worse (due to huge outliers)\n * wrapper around xarray datasets, which always returns non-NaN data and/or statistics that I need are computed\n inside of this wrapper. NaNs are always inside. Doable?\n * boxplots by timestep visualization (with boxplot, outliers for each step)\n * remove collection of deps in style [tests_and_bla_bla] to [tests,bla]\n * links to papers \u2013 AvgRelMAE (Davydenko and Fildes, 2013); link to Nemenyi paper / implementations\n * make graphs with PIs more narrow on 0,1,.. steps as there's too much space left (with an option to turn this off).\n * better API for the end user \u2013 minimize interaction with xarray\n * pep517 build / wheels / better setup.py as per Hynek\n * travis: add 3.8 default python when it's available\n * docs: supported metrics & API options\n * Maybe use api like Summary in statsmodels MLEModel class, it has extend methods and warn/info messages\n * pretty legend for lots like here https://studywolf.wordpress.com/2017/11/21/matplotlib-legends-for-mean-and-confidence-interval-plots/\n * Look for TODOs\n * changable colors\n * turn off colored display option\n * a nicer API for raw metrics container\n * codacy badge\n * boxplots to compare models (as an alternative)\n * violin plots to compare predictions \u2013 areas can be colored, different metrics on left and right (like relative...)\n* **Release 0.4**\n * holiday/fourier features model\n * fix viz module to have less of important stuff\n * a gif with project visualization\n * check shapes of input arrays (target vs preds), now it doesn't raise an error\n * Baseline prediction using target dataset (without explicit calculation, but losing some time points)\n\n## \ud83d\udca1 Ideas\n\n* components\n * Graph: Visualize outliers from confidence interval\n * Multi-comparison component: scikit_posthocs lib or homecooked?\n * inspect true confidence interval coverage via sampling (was done in postings around bayesian dropout sampling)\n * xarrays: compare if compared datasets are actually equal (offets by dates, shapes, maybe even hashing)\n * bin together step performance, like steps 0-1, 2-5, 6-12, 13-24\n * highlight regions using a mask (holidays, etc.)\n * option to view interactively points using widget (plotly)?\n * diagnostics: bias to over / underestimate points\n * animated graphs for change in seasonality\n* features\n * example notebook for fourier?\n * tests for fourier\n * nint generation\n* utils:\n * model adaptor (for different models, generic) which generates 3d prediction dataset. For stastmodels using dyn forecast or kalman filter\n * future importance calculator, but only if I can manipulate input features\n * feature selection using PACF / prewhiten?\n* project\n * more defensive style (add arg checks, so it's easier to understand what is going on)\n * docstrings\n * circleci\n * https://timothycrosley.github.io/portray/ for docs\n* sMAPE & MASE can be added for the jupyter evaluation tables\n* ? Residual stats: since I have residuals => Ljung-Box, Heteroscedasticity test, Jarque-Bera \u2013 like in statsmodels results,\n but probably these stats were inspected already by the user... and on which step should they be computed then?\n\n## See also\n\n* https://arch.readthedocs.io/en/latest/multiple-comparison/multiple-comparison_examples.html\n* RMC as an anlogue for nemenyi https://github.com/config-i1/greybox/blob/master/R/rmc.R\n\n## \ud83e\udd39\ud83c\udffc\u200d\u2642\ufe0f Development\n\nRecommended development workflow:\n```\npipenv install -e .[dev]\npipenv shell\n```\nThe library doesn't use Flit/Poetry, so the suggested workflow is based on Pipenv (as per https://github.com/pypa/pipenv/issues/1911).\nPipfile* are ignored in the .gitignore.\n# Changelog\n\n## 0.2.3 (2021-06-03)\n\n### Fixes (unreleased changes from 2019, doh)\n\n- Fix results for 1 time series (noop)\n\n\n## 0.2.2 (2019-10-22)\n\n### Fixes\n\n- Fix nan values propagated to Friedman Nemenyi test.\n- Critical distance is returned alongside Friedman Nemenyi test.\n\n\n## 0.2.1 (2019-10-18)\n\n### Fixes\n\nOutdated import in wheel version of the package.\n\n\n## 0.2.0 (2019-10-16)\n\n### Features\n\n- Multiple prediction ranking with Friedman Nemenyi posthoc.\n- Visualization of prediction intervals\n- Indication of prediction ranking in a colorful table\n- Rewrite of the internal computation machinery\n\n\n## 0.1.0 (2019-10-04)\n\n### Features\n\n- N-step ahead evaluation widget for Jupyter\n- Absolute & relative metrics for point forecasts and prediction intervals (MSE, MAE, rMSE, rMAE, MIS, rMIS)\n- Naive/Seasonal models for baseline (with prediction intervals)\n- Helper functions to evaluate n-step ahead forecasts using Statsmodels models or naive/seasonal naive models.\n- Holiday features generation and model evaluation on holiday datetimes.\n\n\n## 0.0.1 (2019-09-18)\n\n### Features\n\n- Fixed fourier series generation (fixed in time according to pandas index)\n\n\n", "description_content_type": "text/markdown", "docs_url": null, "download_url": "", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "https://github.com/vshulyak/ts-eval", "keywords": "", "license": "", "maintainer": "", "maintainer_email": "", "name": "ts-eval", "package_url": "https://pypi.org/project/ts-eval/", "platform": "", "project_url": "https://pypi.org/project/ts-eval/", "project_urls": { "Homepage": "https://github.com/vshulyak/ts-eval" }, "release_url": "https://pypi.org/project/ts-eval/0.2.3/", "requires_dist": [ "pandas (>=0.23.0)", "numpy (>=1.16.0)", "xarray (>=0.13)", "scipy (>=1.3)", "statsmodels (>=0.10)", "jupyterlab (>=1.1)", "matplotlib (>=3.0)", "dataclasses", "ipykernel", "jupyter-contrib-nbextensions", "pdbpp ; extra == 'debug'", "hypothesis[numpy] ; extra == 'dev'", "pytest ; extra == 'dev'", "tox ; extra == 'dev'", "pre-commit ; extra == 'dev'", "coverage ; extra == 'dev'", "twine ; extra == 'dev'", "pdbpp ; extra == 'dev'", "holidays (>=0.9) ; extra == 'dev'", "holidays (>=0.9) ; extra == 'extra_runtime_libs'", "twine ; extra == 'pypi'", "hypothesis[numpy] ; extra == 'tests'", "pytest ; extra == 'tests'", "tox ; extra == 'tests'", "pre-commit ; extra == 'tests'", "coverage ; extra == 'tests'", "hypothesis[numpy] ; extra == 'tests_and_extra_runtime_libs'", "pytest ; extra == 'tests_and_extra_runtime_libs'", "tox ; extra == 'tests_and_extra_runtime_libs'", "pre-commit ; extra == 'tests_and_extra_runtime_libs'", "coverage ; extra == 'tests_and_extra_runtime_libs'", "holidays (>=0.9) ; extra == 'tests_and_extra_runtime_libs'" ], "requires_python": "", "summary": "Time Series analysis and evaluation tools", "version": "0.2.3", "yanked": false, "yanked_reason": null }, "last_serial": 10549840, "releases": { "0.0.1": [ { "comment_text": "", "digests": { "md5": "7a21d1a34efbfe05d603ac9219675e31", "sha256": "e943dce761e064c661fc9b8e16f14268c0b0471b78307d6baeee48e88bf10b58" }, "downloads": -1, "filename": "ts_eval-0.0.1-py3-none-any.whl", "has_sig": false, "md5_digest": "7a21d1a34efbfe05d603ac9219675e31", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": null, "size": 3443, "upload_time": "2019-09-18T18:29:23", "upload_time_iso_8601": "2019-09-18T18:29:23.310862Z", "url": "https://files.pythonhosted.org/packages/90/c9/cd34cce1bd4895fbad69289f39c66bc3c223f5f99a1029ee241c58a6cb3c/ts_eval-0.0.1-py3-none-any.whl", "yanked": false, "yanked_reason": null }, { "comment_text": "", "digests": { "md5": "0882013c99c74e1aec966eb2c3bb1214", "sha256": "8505d6c1936fee6bb57190fa45670771e63276e283fa3758d85c1b28af5819e9" }, "downloads": -1, "filename": "ts-eval-0.0.1.tar.gz", "has_sig": false, "md5_digest": "0882013c99c74e1aec966eb2c3bb1214", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 2090, "upload_time": "2019-09-18T18:29:25", "upload_time_iso_8601": "2019-09-18T18:29:25.615732Z", "url": "https://files.pythonhosted.org/packages/78/5b/090aca9a6567e61e0297aa4b377364bc25ed7a2347b7183db1b0497bdd62/ts-eval-0.0.1.tar.gz", "yanked": false, "yanked_reason": null } ], "0.1.0": [ { "comment_text": "", "digests": { "md5": "fb2e10ae63e1e6e971999035531640dc", "sha256": "ba89fe47e92b2caf42ca496fefa29d9e95e12db010e54db9ade9fc1c01684f9d" }, "downloads": -1, "filename": "ts_eval-0.1.0-py2.py3-none-any.whl", "has_sig": false, "md5_digest": "fb2e10ae63e1e6e971999035531640dc", "packagetype": "bdist_wheel", "python_version": "py2.py3", "requires_python": null, "size": 28368, "upload_time": "2019-10-04T08:58:23", "upload_time_iso_8601": "2019-10-04T08:58:23.477012Z", "url": "https://files.pythonhosted.org/packages/67/a8/f74f32c63d8ce2c1074340c39e00764fbab1a8f06a834bed8836a70afa17/ts_eval-0.1.0-py2.py3-none-any.whl", "yanked": false, "yanked_reason": null }, { "comment_text": "", "digests": { "md5": "4264bba4106f91ab2266694915a458b5", "sha256": "b85cf4e01c34ea60fa673233359d6a5742093f7bcfc9ea03dfca38414576578a" }, "downloads": -1, "filename": "ts-eval-0.1.0.tar.gz", "has_sig": false, "md5_digest": "4264bba4106f91ab2266694915a458b5", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 20556, "upload_time": "2019-10-04T08:58:26", "upload_time_iso_8601": "2019-10-04T08:58:26.838900Z", "url": "https://files.pythonhosted.org/packages/d6/0a/6e120d5d6d7e8b247ed028f28460667dc1cf5d0690b1ee67f0767c96df19/ts-eval-0.1.0.tar.gz", "yanked": false, "yanked_reason": null } ], "0.2.0": [ { "comment_text": "", "digests": { "md5": "e1bd4dc571aaff254f95e1e494af0489", "sha256": "c9ae5054636a088c35abe074a21b716df74a943b496b814a8ec87a9de51e68ee" }, "downloads": -1, "filename": "ts_eval-0.2.0-py2.py3-none-any.whl", "has_sig": false, "md5_digest": "e1bd4dc571aaff254f95e1e494af0489", "packagetype": "bdist_wheel", "python_version": "py2.py3", "requires_python": null, "size": 40888, "upload_time": "2019-10-16T11:51:05", "upload_time_iso_8601": "2019-10-16T11:51:05.359719Z", "url": "https://files.pythonhosted.org/packages/38/ac/87da01952aa68e640a478575dcd4522a9ee4eaad41751676eb4a74d704d9/ts_eval-0.2.0-py2.py3-none-any.whl", "yanked": false, "yanked_reason": null }, { "comment_text": "", "digests": { "md5": "3cf76f49b3fa5dc4c0aa17607902c98d", "sha256": "2e559783411464e51f4115a847b7738b76d7c448c9b1952710eed20d7dd8c240" }, "downloads": -1, "filename": "ts-eval-0.2.0.tar.gz", "has_sig": false, "md5_digest": "3cf76f49b3fa5dc4c0aa17607902c98d", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 1175047, "upload_time": "2019-10-16T11:51:51", "upload_time_iso_8601": "2019-10-16T11:51:51.556499Z", "url": "https://files.pythonhosted.org/packages/d5/3c/1c07e2ea89ddaafba9c9285af1d301e8ac8f2e5b4132abddbac198bdb25f/ts-eval-0.2.0.tar.gz", "yanked": false, "yanked_reason": null } ], "0.2.1": [ { "comment_text": "", "digests": { "md5": "d510f55821e3496fa75b42f2cfedb009", "sha256": "fd9b5e7d33200bc42615a239ba67a6a35dbd0771494f71c90cd48a28d24d9417" }, "downloads": -1, "filename": "ts_eval-0.2.1-py2.py3-none-any.whl", "has_sig": false, "md5_digest": "d510f55821e3496fa75b42f2cfedb009", "packagetype": "bdist_wheel", "python_version": "py2.py3", "requires_python": null, "size": 39082, "upload_time": "2019-10-18T12:55:41", "upload_time_iso_8601": "2019-10-18T12:55:41.102739Z", "url": "https://files.pythonhosted.org/packages/bf/e4/58611f3330f4cf4be1214d6f422c9689c5c091d007d20bf17568f8c38386/ts_eval-0.2.1-py2.py3-none-any.whl", "yanked": false, "yanked_reason": null }, { "comment_text": "", "digests": { "md5": "fcaa47cee646b54482704c7b63c72a5c", "sha256": "c59c728ddc1fa6d6120532fca37120380f8e7e0d23ba7f2f66da0418f2f41170" }, "downloads": -1, "filename": "ts-eval-0.2.1.tar.gz", "has_sig": false, "md5_digest": "fcaa47cee646b54482704c7b63c72a5c", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 1176445, "upload_time": "2019-10-18T12:56:50", "upload_time_iso_8601": "2019-10-18T12:56:50.014412Z", "url": "https://files.pythonhosted.org/packages/14/f3/6476b3838d6d4c66d1b6c142fdd8ee9ccf599414b2cd9ab8319d65998cb5/ts-eval-0.2.1.tar.gz", "yanked": false, "yanked_reason": null } ], "0.2.2": [ { "comment_text": "", "digests": { "md5": "76ce58044ca2fcceb18a06c71111da2e", "sha256": "c5858846c2924095ff5eff4c281033f0247f0e34853b3f327d478018d470bf0a" }, "downloads": -1, "filename": "ts_eval-0.2.2-py2.py3-none-any.whl", "has_sig": false, "md5_digest": "76ce58044ca2fcceb18a06c71111da2e", "packagetype": "bdist_wheel", "python_version": "py2.py3", "requires_python": null, "size": 39398, "upload_time": "2019-10-22T08:41:13", "upload_time_iso_8601": "2019-10-22T08:41:13.931511Z", "url": "https://files.pythonhosted.org/packages/fe/02/09e9a5808d3c91c7e3e9aa2f48cee13daebff5881aa570ee95aae0407c70/ts_eval-0.2.2-py2.py3-none-any.whl", "yanked": false, "yanked_reason": null }, { "comment_text": "", "digests": { "md5": "30397cd5f496c2661f7461c1459bd74b", "sha256": "122495c538366583ef25f76395893e42fd09593f7a24004b98a1269e525fda35" }, "downloads": -1, "filename": "ts-eval-0.2.2.tar.gz", "has_sig": false, "md5_digest": "30397cd5f496c2661f7461c1459bd74b", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 1177030, "upload_time": "2019-10-22T08:42:19", "upload_time_iso_8601": "2019-10-22T08:42:19.954781Z", "url": "https://files.pythonhosted.org/packages/3f/46/8e314f63fe595487f72595b53861107fd88ecf75cfd2958c0509455b12e0/ts-eval-0.2.2.tar.gz", "yanked": false, "yanked_reason": null } ], "0.2.3": [ { "comment_text": "", "digests": { "md5": "15919d0a12c93e984a6220026e7d5d57", "sha256": "57f1b1fa22aa12560b7fbb5585c4796e4a4bcde3dccf8e2fd1508255917a2719" }, "downloads": -1, "filename": "ts_eval-0.2.3-py2.py3-none-any.whl", "has_sig": false, "md5_digest": "15919d0a12c93e984a6220026e7d5d57", "packagetype": "bdist_wheel", "python_version": "py2.py3", "requires_python": null, "size": 40038, "upload_time": "2021-06-03T16:39:17", "upload_time_iso_8601": "2021-06-03T16:39:17.533281Z", "url": "https://files.pythonhosted.org/packages/7f/d3/1632b1690422c7f4c25fd4fb22d79ac0a8be32b3e2cdfdd2d7d16561539c/ts_eval-0.2.3-py2.py3-none-any.whl", "yanked": false, "yanked_reason": null }, { "comment_text": "", "digests": { "md5": "7cd94e28fde60e2487bd8e083491d244", "sha256": "10044e85079d3e79432838eddd566ee5be6832f96e45b4431021000748cd174a" }, "downloads": -1, "filename": "ts-eval-0.2.3.tar.gz", "has_sig": false, "md5_digest": "7cd94e28fde60e2487bd8e083491d244", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 1178369, "upload_time": "2021-06-03T16:39:20", "upload_time_iso_8601": "2021-06-03T16:39:20.064716Z", "url": "https://files.pythonhosted.org/packages/f2/be/c916e8116d607a423dc9dd139034ce4e77b69ef720b4c2e4e254b431571c/ts-eval-0.2.3.tar.gz", "yanked": false, "yanked_reason": null } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "15919d0a12c93e984a6220026e7d5d57", "sha256": "57f1b1fa22aa12560b7fbb5585c4796e4a4bcde3dccf8e2fd1508255917a2719" }, "downloads": -1, "filename": "ts_eval-0.2.3-py2.py3-none-any.whl", "has_sig": false, "md5_digest": "15919d0a12c93e984a6220026e7d5d57", "packagetype": "bdist_wheel", "python_version": "py2.py3", "requires_python": null, "size": 40038, "upload_time": "2021-06-03T16:39:17", "upload_time_iso_8601": "2021-06-03T16:39:17.533281Z", "url": "https://files.pythonhosted.org/packages/7f/d3/1632b1690422c7f4c25fd4fb22d79ac0a8be32b3e2cdfdd2d7d16561539c/ts_eval-0.2.3-py2.py3-none-any.whl", "yanked": false, "yanked_reason": null }, { "comment_text": "", "digests": { "md5": "7cd94e28fde60e2487bd8e083491d244", "sha256": "10044e85079d3e79432838eddd566ee5be6832f96e45b4431021000748cd174a" }, "downloads": -1, "filename": "ts-eval-0.2.3.tar.gz", "has_sig": false, "md5_digest": "7cd94e28fde60e2487bd8e083491d244", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 1178369, "upload_time": "2021-06-03T16:39:20", "upload_time_iso_8601": "2021-06-03T16:39:20.064716Z", "url": "https://files.pythonhosted.org/packages/f2/be/c916e8116d607a423dc9dd139034ce4e77b69ef720b4c2e4e254b431571c/ts-eval-0.2.3.tar.gz", "yanked": false, "yanked_reason": null } ], "vulnerabilities": [] }