\n
\n A Pythonic query language for time series data\n
\n\n\n\n\n\n## Table of Contents\n\n* [About the Project](#about-the-project)\n * [Built With](#built-with)\n* [Getting Started](#getting-started)\n * [Prerequisites](#prerequisites)\n * [Installation](#installation)\n* [Usage](#usage)\n * [CSV Backend](#csv-backend-usage)\n * [AppOptics Backend](#appoptics-backend-usage)\n * [TimeSeries](#timeseries-usage)\n* [Roadmap](#roadmap)\n* [Contributing](#contributing)\n* [License](#license)\n* [Contact](#contact)\n* [Acknowledgements](#acknowledgements)\n\n\n\n\n## About The Project\n\nThere are many time series databases and each have their own query language. Each platform takes time to invest in \nlearning the structure and keywords of that language and often the skills learned don't translate to other \nplatforms. The goal of this project is to create a time series specific library that can be used across many different time series databases as well as easy to learn because it uses Python syntax. \n\n\n### Built With\n\n* [Numpy](https://numpy.org)\n* [Pandas](https://pandas.pydata.org)\n* [Matplotlib](https://matplotlib.org)\n\n\n\n\n## Getting Started\n\nTo get a local copy up and running follow these simple steps.\n\n### Prerequisites\n\nThe requirements are in the [requirements.txt](requirements.txt) file.\n\n### Installation\n\n#### pip\n\n```sh\npip install timeseriesql\n```\n\n#### manual\n\n1. Clone the timeseriesql\n```sh\ngit clone https:://github.com/mbeale/timeseriesql.git\n```\n2. Install library\n```sh\ncd timeseriesql\npython setup.py install \n```\n\n\n## Usage\n\nThe way this project works is to provide a general framework for querying a time series with pluggable\nbackends that communicate with specific time series databases. The queries are created using Python\ngenerators, a formatt familiar to Pythonistas.\n\n```python\ndata = Query(x for x in \"metric.name\" if x.some_label = \"some_value\").by(\"a_label\")[start:end:resolution]\n```\n\nThe return value is a ``TimeSeries`` object that uses a Numpy array as backend. That object can have \n``ufuncs`` and other numpy functions applied against it. More examples to come.\n\n### CSV Backend Usage\n\nOften time series data is loaded from a CSV file. The backend expects the first column to be the time index in \neither a numerical timestamp or strings in ISO 8601 date or datetime format. The filters are applied to the\nheaders of the CSV. If labels are not in the CSV and are supplied as part of the query, then filters will not\nbe applied.\n\nIf any columns are empty or don't contain a numeric value, the value becomes a ``np.nan``.\n\n\n#### Basic CSV Usage\n\n```python\nfrom timeseriesql.backends import CSVBackend\n\ndata = CSVBackend(x for x in \"path/to.csv\")[:]\n```\n\n#### Basic CSV Filtering\n\nFor CSV files the labels are the column headers. If there are columns that are not needed, they can be filtered out.\n\n```python\nfrom timeseriesql.backends import CSVBackend\n\ndata = CSVBackend(x for x in \"path/to.csv\" if x.label == \"A\")[:]\ndata = CSVBackend(x for x in \"path/to.csv\" if x.label != \"B\")[:]\ndata = CSVBackend(x for x in \"path/to.csv\" if x.label in [\"B\", \"C\", \"G\"])[:]\ndata = CSVBackend(x for x in \"path/to.csv\" if x.label not in [\"B\", \"C\", \"G\"])[:]\n```\n\n#### Set the Labels\n\n```python\nfrom timeseriesql.backends import CSVBackend\n\ndata = CSVBackend(x for x in \"path/to.csv\").labels(\n [\n {\"label\": \"one\"},\n {\"label\": \"two\"},\n {\"label\": \"three\"},\n {\"label\": \"four\"},\n {\"label\": \"five\"},\n {\"label\": \"six\"},\n {\"label\": \"seven\"},\n ]\n)[:]\n```\n\n### AppOptics Backend Usage\n\n[Appoptics](www.appoptics.com) is a commercial time series database product. The backend converts a query into an \nAPI call.\n\nThe backend expects a ``APPOPTICS_TOKEN`` environment variable to be set in order to authenticate to AppOptics.\n\n#### AppOptics Query\n\n```python\nfrom timeseriesql.backends import AOBackend\n\ndata = AOBackend(x for x in \"metric.name\")[3600:] #basic\ndata = AOBackend(x * 100 for x in \"metric.name\")[3600:] #binary operations (+, -, /, *)\ndata = AOBackend(x * 1.8 + 32 for x in \"metric.name\")[3600:] #multiple binary operations (\u00b0C to \u00b0F)\ndata = AOBackend(x.max for x in \"metric.name\")[3600:] #get max value\n```\n\n#### AppOptics Filtering\n\nCurrently only ``==`` is supported.\n\n```python\nfrom timeseriesql.backends import AOBackend\n\ndata = AOBackend(x for x in \"metric.name\" if x.environment == 'production')[3600:]\n```\n\n#### AppOptics Grouping\n\n```python\nfrom timeseriesql.backends import AOBackend\n\ndata = AOBackend(x for x in \"metric.name\").group('environment')[3600:]\ndata = AOBackend(x - y for x,y in AOBackend((x.max for x in \"metric1\"), (x.min for x in \"metric2\")).by('tag1'))[3600:]\n```\n\n#### AppOptics Time\n```python\nfrom timeseriesql.backends import AOBackend\n\ndata = AOBackend(x for x in \"metric.name\")[:] #no start or end time (not recommended)\ndata = AOBackend(x for x in \"metric.name\")[3600:] #from now - 3600 seconds until now, resolution of 1\ndata = AOBackend(x for x in \"metric.name\")[3600:1800] #from now - 3600 seconds until now - 1800 seconds, resolution of 1\ndata = AOBackend(x for x in \"metric.name\")[3600::300] #from now - 3600 seconds until now resoultion of 300 seconds\n```\n\n#### AppOptics Functions\n```python\ndata = AOBackend(sum(derive(x)) for x in \"metric.name\")[3600:] #get the sums of the derivatives\ndata = AOBackend(zero_fill(x) for x in \"metric.name\")[3600::60] #zero_fill\n```\n\n### TimeSeries Usage\n\nThe `TimeSeries` object is allows for manipulation of the time series data after the it's been queried from the \nbackend. There are also helper functions to convert to a pandas `DataFrame` and plot using matplotlib. \n\nIn the following examples, the variables starting with `ts_` are assumed to be queried data from a backend.\n\n#### TimeSeries Operations\n\n```python\n\n# Basic mathematical operations (+, -, /, *)\nts_1 + 5 # will return a new series\nts_1 += 5 #will perform operation in place\nts_1 += ts_2 #add together two TimeSeries\n\n```\n\n#### TimeSeries Time Index\n\nThe time index is a array of floats but there is a built in method to convert the floats into `np.datetime64`.\n\n```python\nts_1.time # array of floats\nts_1.time.dt #array of np.datetime64\n```\n\n#### TimeSeries Merging\n\n`TimeSeries` objects can be combined but the ending time indexes must be the same. This may require empty values \nto be created where the indexes don't align.\n\n```python\nnew_t = ts_1.merge([ts_2, ts_3])\n```\n\n#### TimeSeries Grouping/Reducing\n\nIf there are multiple streams, they can be grouped and merged by the labels.\n\n```python\nreduced = ts_1.group([\"hostname\", \"name\"]).add() \nreduced = ts_1.group(\"env\").mean()\nreduced = ts_1.group(\"env\").mean(axis=None) #setting the access to None will get the mean of the entire object\n```\n\n#### TimeSeries Special Indexing\n\n```python\nimport numpy as np\n\nbeg = np.datetime64('2019-02-25T03:00')\nend = np.datetime64('2019-02-25T04:00')\n\nts_1[beg:end] # set a time range\nts_1[beg : np.timedelta64(3, \"m\")] # fetch from beginning + 3 minutes\nts_1[np.timedelta64(3, \"m\") :] #start from beginning + 3 minutes\nts_1[: np.timedelta64(3, \"m\")] #end at the end - 3 minutes\n\n\nts_1[{\"hostname\": \"host2\"}] # by labels\n\n```\n\n#### TimeSeries Rolling Windows\n\nThe `rolling_window` method assumes that the data is filled and at a fixed resolution. Number of periods is \nan integer and not a time range.\n\n```python\nrolling_cum_sum = ts_1.rolling_window(12).add() #rolling cumsum\nrolling_mean = ts_1.rolling_window(12).mean() # rolling mean\nrolling = ts_1.rolling_window(12).median() #rolling median\n```\n\n#### TimeSeries Resample\n\nThe `resample` method allows a smaller period to be aggregated into a larger period.\n\n```python\nresampled = ts_1.resample(300).mean() #resamples to 5 minutes and takes the mean\n```\n\n#### TimeSeries to Pandas\n\nThe conversion returns 2 pandas DataFrames, one for the labels and the other for the data.\n\n```python\ndata, labels = ts_1.to_pandas()\n```\n\n#### TimeSeries Matplotlib\n\nThere is a helper function that will apply some sane defaults to a plotting function for a TimeSeries object.\n\n```python\nts_1.plot(legend=True)\nplt.show()\n```\n\n\n## Roadmap\n\nSee the [open issues](https://github.com/mbeale/timeseriesql/issues) for a list of proposed features (and known issues).\n\n\n\n\n## Contributing\n\nContributions are what make the open source community such an amazing place to be learn, inspire, and create. Any contributions you make are **greatly appreciated**.\n\n1. Fork the Project\n2. Create your Feature Branch (`git checkout -b feature/AmazingFeature`)\n3. Commit your Changes (`git commit -m 'Add some AmazingFeature'`)\n4. Push to the Branch (`git push origin feature/AmazingFeature`)\n5. Open a Pull Request\n\n\n\n\n## License\n\nDistributed under the MIT License. See `LICENSE` for more information.\n\n\n\n\n## Contact\n\nMichael Beale - michael.beale@gmail.com\n\nProject Link: [https://github.com/mbeale/timeseriesql](https://github.com/mbeale/timeseriesql)\n\n\n\n\n\n[contributors-shield]: https://img.shields.io/github/contributors/mbeale/timeseriesql.svg?style=flat-square\n[contributors-url]: https://github.com/mbeale/timeseriesql/graphs/contributors\n[forks-shield]: https://img.shields.io/github/forks/mbeale/timeseriesql.svg?style=flat-square\n[forks-url]: https://github.com/mbeale/timeseriesql/network/members\n[stars-shield]: https://img.shields.io/github/stars/mbeale/timeseriesql.svg?style=flat-square\n[stars-url]: https://github.com/mbeale/timeseriesql/stargazers\n[issues-shield]: https://img.shields.io/github/issues/mbeale/timeseriesql.svg?style=flat-square\n[issues-url]: https://github.com/mbeale/timeseriesql/issues\n[license-shield]: https://img.shields.io/github/license/mbeale/timeseriesql.svg?style=flat-square\n[license-url]: https://github.com/mbeale/timeseriesql/blob/master/LICENSE.txt\n[linkedin-shield]: https://img.shields.io/badge/-LinkedIn-black.svg?style=flat-square&logo=linkedin&colorB=555\n[linkedin-url]: https://linkedin.com/in/michael-beale-163a4670\n\n\n", "description_content_type": "text/markdown", "docs_url": null, "download_url": "", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "http://github.com/mbeale/timeseriesql", "keywords": "", "license": "MIT", "maintainer": "", "maintainer_email": "", "name": "timeseriesql", "package_url": "https://pypi.org/project/timeseriesql/", "platform": "", "project_url": "https://pypi.org/project/timeseriesql/", "project_urls": { "Homepage": "http://github.com/mbeale/timeseriesql" }, "release_url": "https://pypi.org/project/timeseriesql/0.1.1/", "requires_dist": null, "requires_python": ">=3.7", "summary": "A Pythonic query language for time series data", "version": "0.1.1" }, "last_serial": 6002559, "releases": { "0.1": [ { "comment_text": "", "digests": { "md5": "5fdfab03e639153fa3ec32d3c732416a", "sha256": "c0b1e6a9cbd74668675459961886824b6628729e64004ced8e76fc50d3dc0698" }, "downloads": -1, "filename": "timeseriesql-0.1-py3-none-any.whl", "has_sig": false, "md5_digest": "5fdfab03e639153fa3ec32d3c732416a", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": ">=3.7", "size": 17681, "upload_time": "2019-10-13T09:45:32", "url": "https://files.pythonhosted.org/packages/74/3f/d78d91176b5f6f1fdeebe45773ef1ec9826822b90c0dcb1ad14df914308c/timeseriesql-0.1-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "4210f3a8242a23ad84034910c60f8e4d", "sha256": "dcb54b497bb239ae2af4ece8c98995b93bd702099c9a9d24b6a9863c1069ed74" }, "downloads": -1, "filename": "timeseriesql-0.1.tar.gz", "has_sig": false, "md5_digest": "4210f3a8242a23ad84034910c60f8e4d", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.7", "size": 20149, "upload_time": "2019-10-13T09:45:35", "url": "https://files.pythonhosted.org/packages/a4/8c/85bca3bf1f9b931d8d2b5340ae1454c2d5900a894f1b52f462c90469d6c6/timeseriesql-0.1.tar.gz" } ], "0.1.1": [ { "comment_text": "", "digests": { "md5": "0f3df1ba6689d2db67da57787f4c8c3b", "sha256": "408c30821f76692c30a2ac5f48b7d09dad3238d422d7db281f835bb487e5030c" }, "downloads": -1, "filename": "timeseriesql-0.1.1-py3-none-any.whl", "has_sig": false, "md5_digest": "0f3df1ba6689d2db67da57787f4c8c3b", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": ">=3.7", "size": 18005, "upload_time": "2019-10-20T10:23:23", "url": "https://files.pythonhosted.org/packages/84/04/19f56664d006959d246bccc6a02ae801cd2e508a2758dce4048b6c4c83b1/timeseriesql-0.1.1-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "23150a24c41a155e9356313ce8237a6d", "sha256": "3e0022fee549b080b61811874d11fe7ed18f1fa2fbba02cc38838aa5caed0a03" }, "downloads": -1, "filename": "timeseriesql-0.1.1.tar.gz", "has_sig": false, "md5_digest": "23150a24c41a155e9356313ce8237a6d", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.7", "size": 20615, "upload_time": "2019-10-20T10:23:25", "url": "https://files.pythonhosted.org/packages/58/a8/d6e677696162904a6132d80072d8b0567bdb22263a3a266558e21826416e/timeseriesql-0.1.1.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "0f3df1ba6689d2db67da57787f4c8c3b", "sha256": "408c30821f76692c30a2ac5f48b7d09dad3238d422d7db281f835bb487e5030c" }, "downloads": -1, "filename": "timeseriesql-0.1.1-py3-none-any.whl", "has_sig": false, "md5_digest": "0f3df1ba6689d2db67da57787f4c8c3b", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": ">=3.7", "size": 18005, "upload_time": "2019-10-20T10:23:23", "url": "https://files.pythonhosted.org/packages/84/04/19f56664d006959d246bccc6a02ae801cd2e508a2758dce4048b6c4c83b1/timeseriesql-0.1.1-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "23150a24c41a155e9356313ce8237a6d", "sha256": "3e0022fee549b080b61811874d11fe7ed18f1fa2fbba02cc38838aa5caed0a03" }, "downloads": -1, "filename": "timeseriesql-0.1.1.tar.gz", "has_sig": false, "md5_digest": "23150a24c41a155e9356313ce8237a6d", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.7", "size": 20615, "upload_time": "2019-10-20T10:23:25", "url": "https://files.pythonhosted.org/packages/58/a8/d6e677696162904a6132d80072d8b0567bdb22263a3a266558e21826416e/timeseriesql-0.1.1.tar.gz" } ] }