{ "info": { "author": "Eric Frechette", "author_email": "frechetta93@gmail.com", "bugtrack_url": null, "classifiers": [ "Intended Audience :: Developers", "Programming Language :: Python" ], "description": "[![Build Status](https://travis-ci.org/Frechetta/DXQL.svg?branch=master)](https://travis-ci.org/Frechetta/DXQL) [![codecov](https://codecov.io/gh/Frechetta/DXQL/branch/master/graph/badge.svg)](https://codecov.io/gh/Frechetta/DXQL)\n\n# Data Exploration Query Language (DXQL)\n\nRequires Python 3.7\n\n## Usage\n\n1. Import dxql.search.Pipeline into your project\n2. Instantiate a Pipeline using Pipeline.create_pipeline(*query-string*)\n3. Use the new pipeline to search over an iterable of dicts using pipeline.execute(*events*)\n\nExample:\n\n```\nfrom dxql.search import Pipeline\npipeline = Pipeline.create_pipeline('search ip=192.168.1.10')\nresults = pipeline.execute(events)\n```\n\n*events* can be any iterable. To search a file, just pass the opened file to `pipeline.execute()`. Each line of the file will be considered an event.\n\nExample:\n\n```\n# myfile.json is a file where each line is a JSON dictionary\nwith open('myfile.json') as file:\n results = pipeline.execute(file)\n```\n\n## Searching\n\nSearching is inspired by Splunk's query language.\n\nThroughout the rest of this document, I will use the terms \"search\" and \"query\" interchangebly.\n\nA query can consist of multiple commands separated by a pipe (`|`). Imagine a multiple-command search as a \"pipeline\" where each command is applied to the data in turn, with the data being fed from one command to the next until the end of the pipeline.\n\nThere are four commands available:\n\n### 1. search\n\nThe `search` command allows you to filter the data using key-value pairs and modifiers like `OR` and `NOT`. It must be the first command in the query.\n\n#### Usage:\n\nsearch \\...\n\n**\\**\n\n\\ | NOT \\ | \\ OR \\\n\n**\\**\n\n\\\\\\\n\n**\\**\n\n= | != | < | <= | > | >=\n\n#### Examples\n\n##### Retrieving data from an index\n\nThis search will return all data from the `geoip` index.\n\n`search index=geoip`\n\n##### Retrieving GeoIP data for specific IPs\n\nUse the `OR` modifier to specify multiple values for a field.\n\n`search index=geoip ip=192.168.1.10 OR ip=192.168.1.11`\n\n##### Retrieving GeoIP data for all IPs except one\n\n`search index=geoip ip!=192.168.1.15`\n\nor\n\n`search index=geoip NOT ip=192.268.1.15`\n\n#### Retrieving data for a specific IP from multiple indices\n\nIt is not required to search by index.\n\n`search ip=192.168.1.15`\n\nThe above search will return data with `ip=192.168.1.15` from all indices (in this case, data from indices `geoip` and `ip_rdap` will be returned; events in `rdap` do not contain an `ip` field).\n\n### 2. fields\n\nThe `fields` command allows you to display only the fields you want to see.\n\n#### Usage\n\nfields \\...\n\n#### Example\n\nRemove all fields from the results except for `ip` and `continent_name`:\n\n`search index=geoip | fields ip continent_name`\n\n### 3. join\n\nThe `join` command allows you to join data together by a field (the \"by-field\"). Each event that shares the same value for the by-field is joined together under one event. This allows you to join data from two disparate data sources.\n\n#### Usage\n\njoin BY \\\n\n#### Example\n\nJoin an IP with its associated RDAP data using the `ip_rdap` and `rdap` indices:\n\n`search index=ip_rdap OR index=rdap | join BY handle`\n\n`handle` is the 'by-field', the field that is shared by the different kinds of data.\n\n### 4. prettyprint\n\nThe `prettyprint` command may only be used as the last command in the search. It allows you to print the result set in a prettier fashion than plain JSON blobs.\n\n#### Usage\n\nprettyprint format=\\\n\n**\\**\n\njson | table\n\n#### Examples\n\n##### Print results as pretty JSON\n\nUsing `format=json` still prints each result as JSON but with newlines and indentation.\n\n`search index=rdap | prettyprint format=json`\n\n##### Print results as a table\n\nUsing `format=table` prints the results as a formatted table.\n\n`search index=rdap | prettyprint format=table`\n\nIf there are a lot of fields in the result set, the results will overflow onto the next line(s); therefore, it is recommended to pare down unwanted fields using `fields` before using `prettyprint format=table`. This happens expecially when joining `ip_rdap` and `rdap` data together. Many IPs share the same `rdap` data, so the IP values will become very long. I recommend specifying the IP(s) you are interested in before doing the `join`.\n\n\n", "description_content_type": "text/markdown", "docs_url": null, "download_url": "", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "https://github.com/Frechetta/DXQL", "keywords": "data search query language", "license": "", "maintainer": "", "maintainer_email": "", "name": "dxql", "package_url": "https://pypi.org/project/dxql/", "platform": "", "project_url": "https://pypi.org/project/dxql/", "project_urls": { "Homepage": "https://github.com/Frechetta/DXQL" }, "release_url": "https://pypi.org/project/dxql/0.0.6/", "requires_dist": [ "lark-parser (==0.7.1)", "pytest (>=5.0.1) ; extra == 'test'", "pytest-cov (>=2.7.1) ; extra == 'test'", "codecov (>=2.0.15) ; extra == 'test'" ], "requires_python": ">= 3.7", "summary": "Data eXploration Query Language (DXQL)", "version": "0.0.6" }, "last_serial": 5669765, "releases": { "0.0.1": [ { "comment_text": "", "digests": { "md5": "b0d1649e32c424307ba887671bc67c24", "sha256": "87af8b17e31bce8ae5046a88e7e9ce58ea6416becc8ea3f3f5aefeed23b79380" }, "downloads": -1, "filename": "dxql-0.0.1-py3-none-any.whl", "has_sig": false, "md5_digest": "b0d1649e32c424307ba887671bc67c24", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": ">= 3.7", "size": 12020, "upload_time": "2019-08-04T05:16:34", "url": "https://files.pythonhosted.org/packages/1e/3e/0348ed90f8eccd5d10c1cebf57ab4cdc29636fec64b9378505584bd3f175/dxql-0.0.1-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "caa4cf7da4496563dce25250085732a9", "sha256": "b748c65d8166aaabf5d04472f9e4b8e0de7a5b5a40af38fedbf0be1ea0818ff5" }, "downloads": -1, "filename": "dxql-0.0.1.tar.gz", "has_sig": false, "md5_digest": "caa4cf7da4496563dce25250085732a9", "packagetype": "sdist", "python_version": "source", "requires_python": ">= 3.7", "size": 7591, "upload_time": "2019-08-04T05:16:36", "url": "https://files.pythonhosted.org/packages/26/48/ff0f42c65cbb2416f99ed0a75803899b2fd8eeb0ad699ea6c57918a89258/dxql-0.0.1.tar.gz" } ], "0.0.2": [ { "comment_text": "", "digests": { "md5": "060cfd5790b6f19e6d8fe4c642b446e5", "sha256": "e4aae74e4965befdfccfb29e5dfff45f9c34f83531f5e51c62f9be8a2cf3cf13" }, "downloads": -1, "filename": "dxql-0.0.2-py3-none-any.whl", "has_sig": false, "md5_digest": "060cfd5790b6f19e6d8fe4c642b446e5", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": ">= 3.7", "size": 12587, "upload_time": "2019-08-05T02:24:34", "url": "https://files.pythonhosted.org/packages/89/cf/24f81dfe4afb90800b00968b556d437fb2b7cc8de3d226f5b9c089527eec/dxql-0.0.2-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "7d9da5117af04c4fabb3f2301f2624ed", "sha256": "a7c6dfcfdd195be1adafebb91a4b689b0b26f26e1dbcc3118b722a82f7b7448e" }, "downloads": -1, "filename": "dxql-0.0.2.tar.gz", "has_sig": false, "md5_digest": "7d9da5117af04c4fabb3f2301f2624ed", "packagetype": "sdist", "python_version": "source", "requires_python": ">= 3.7", "size": 8042, "upload_time": "2019-08-05T02:24:36", "url": "https://files.pythonhosted.org/packages/ca/63/53324bebc2e0e4a696825aec335202d13ea6fec1197ffae325e9bcb7e199/dxql-0.0.2.tar.gz" } ], "0.0.3": [ { "comment_text": "", "digests": { "md5": "9be0f5406915ec02ba8ac6de8df6eb26", "sha256": "2533f60641755efb23d705bf4be2c08ed51c7470b24017f43a431b6cf4d20654" }, "downloads": -1, "filename": "dxql-0.0.3-py3-none-any.whl", "has_sig": false, "md5_digest": "9be0f5406915ec02ba8ac6de8df6eb26", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": ">= 3.7", "size": 12637, "upload_time": "2019-08-09T03:07:21", "url": "https://files.pythonhosted.org/packages/32/b8/7b8fb06e0c8715192217b14788a584088c213ae5cb48a79d94c5daefe545/dxql-0.0.3-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "6a160b279fc52b8953a5cbfc3562c113", "sha256": "abef5f3c1d96292b5025cb182cc399c203cf19346c98a629ddb3dad10c765d08" }, "downloads": -1, "filename": "dxql-0.0.3.tar.gz", "has_sig": false, "md5_digest": "6a160b279fc52b8953a5cbfc3562c113", "packagetype": "sdist", "python_version": "source", "requires_python": ">= 3.7", "size": 8096, "upload_time": "2019-08-09T03:07:23", "url": "https://files.pythonhosted.org/packages/05/03/63869531f868a70a93c6c1e7c8b4a29eddbc2bc48d8fe14cac679a1549a7/dxql-0.0.3.tar.gz" } ], "0.0.5": [ { "comment_text": "", "digests": { "md5": "2b028cb3740cf080576e5c4564491136", "sha256": "c82680907a5b03f1226ad5518ca2c08428361463ec1689f4164bcdde61aba412" }, "downloads": -1, "filename": "dxql-0.0.5-py3-none-any.whl", "has_sig": false, "md5_digest": "2b028cb3740cf080576e5c4564491136", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": ">= 3.7", "size": 12617, "upload_time": "2019-08-12T03:20:37", "url": "https://files.pythonhosted.org/packages/67/60/2fb33f6be534272bafce24c7571a63c4f57345c368e6cd368eaaaee7959b/dxql-0.0.5-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "35af2636dd6c0c9d0581e3201bb692a2", "sha256": "1a55ed697d84bb6ea023728476e5e7b2b53fb0c474e2f7e41991280787fef682" }, "downloads": -1, "filename": "dxql-0.0.5.tar.gz", "has_sig": false, "md5_digest": "35af2636dd6c0c9d0581e3201bb692a2", "packagetype": "sdist", "python_version": "source", "requires_python": ">= 3.7", "size": 8007, "upload_time": "2019-08-12T03:20:38", "url": "https://files.pythonhosted.org/packages/be/f0/529df769295b1b6f3250038876cb66df28e930e33a7bce15b96961d739ee/dxql-0.0.5.tar.gz" } ], "0.0.6": [ { "comment_text": "", "digests": { "md5": "bc7c831737a2d359acce7fa02980ab1b", "sha256": "dec3d812911515490026a7ce77abefefc466fae169db593822b512bbee33a84d" }, "downloads": -1, "filename": "dxql-0.0.6-py3-none-any.whl", "has_sig": false, "md5_digest": "bc7c831737a2d359acce7fa02980ab1b", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": ">= 3.7", "size": 12617, "upload_time": "2019-08-13T03:31:22", "url": "https://files.pythonhosted.org/packages/25/87/63b5729f0939bf40600b5ef9d6556b31e8fba5064a99d9b6ad4c57ecd8da/dxql-0.0.6-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "42c4c9ee417dffc47b0306160a2e9c14", "sha256": "e15487173d9e6fda05665c1c8603d7c5fedb1101a6fa960a12473874fc83ed72" }, "downloads": -1, "filename": "dxql-0.0.6.tar.gz", "has_sig": false, "md5_digest": "42c4c9ee417dffc47b0306160a2e9c14", "packagetype": "sdist", "python_version": "source", "requires_python": ">= 3.7", "size": 8001, "upload_time": "2019-08-13T03:31:24", "url": "https://files.pythonhosted.org/packages/1b/dd/211ebe2dd4521f9ba899b153c96b598ce84ccd00b8e0d6139f07eedc77b3/dxql-0.0.6.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "bc7c831737a2d359acce7fa02980ab1b", "sha256": "dec3d812911515490026a7ce77abefefc466fae169db593822b512bbee33a84d" }, "downloads": -1, "filename": "dxql-0.0.6-py3-none-any.whl", "has_sig": false, "md5_digest": "bc7c831737a2d359acce7fa02980ab1b", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": ">= 3.7", "size": 12617, "upload_time": "2019-08-13T03:31:22", "url": "https://files.pythonhosted.org/packages/25/87/63b5729f0939bf40600b5ef9d6556b31e8fba5064a99d9b6ad4c57ecd8da/dxql-0.0.6-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "42c4c9ee417dffc47b0306160a2e9c14", "sha256": "e15487173d9e6fda05665c1c8603d7c5fedb1101a6fa960a12473874fc83ed72" }, "downloads": -1, "filename": "dxql-0.0.6.tar.gz", "has_sig": false, "md5_digest": "42c4c9ee417dffc47b0306160a2e9c14", "packagetype": "sdist", "python_version": "source", "requires_python": ">= 3.7", "size": 8001, "upload_time": "2019-08-13T03:31:24", "url": "https://files.pythonhosted.org/packages/1b/dd/211ebe2dd4521f9ba899b153c96b598ce84ccd00b8e0d6139f07eedc77b3/dxql-0.0.6.tar.gz" } ] }