{
    "info": {
        "author": "Ori Hoch",
        "author_email": "ori@uumpa.com",
        "bugtrack_url": null,
        "classifiers": [],
        "description": "# DataFlows Shell\n\nDataFlows is a *\"novel and intuitive way of building data processing flows.\"*\n\nDataFlows Shell leverage DataFlows to use the same intuitive data processing flows for shell automation.\n\n## Introduction\n\nA lot of the work on the shell, especially for \"DevOps\" / automation type work, deals with data processing.\nThe first command a shell user learns is `ls` - which returns a set of data.\nThe second might by `grep` or `cp` - which filters and performs actions based on this data set.\n\nDataFlows Shell acts as a very minimal and intuitive layer between the shell and [the DataFlows framework](https://github.com/datahq/dataflows).\n\n## Demo\n\nThe following example demonstrates importing some processors to the local shell, using them to run a processor chain and printing the output. It uses the `kubectl` processor to get a list of pods from a Kubernetes cluster and filter based on a condition defined using a Python lambda function.\n\n```\n$ source <(dfs import printer filter_rows kubectl)\n\n$ kubectl get pods -c -q \\\n    | dfs 'lambda row: row.update(is_ckan=\"ckan\" in str(row[\"volumes\"]))' --fields=+is_ckan:boolean -q\n    | filter_rows --args='[[{\"is_ckan\":true}]]' -q\n\n{'count_of_rows': 12, 'bytes': 57584, 'hash': '5febe0c3cfe75d174e242f290f00c289', 'dataset_name': None}\ncheckpoint:1\n{'count_of_rows': 12, 'bytes': 57876, 'hash': '17f446a8f562f10cccc1de1a33c48d91', 'dataset_name': None}\ncheckpoint:2\n{'count_of_rows': 6, 'bytes': 40797, 'hash': '6ab4290efd82478b1677d1f226c4199a', 'dataset_name': None}\ncheckpoint:3\n\n$ printer --kwargs='{\"fields\":[\"kind\",\"name\",\"namespace\"]}'\n\nsaving checkpoint to: .dfs-checkpoints/__9\nusing checkpoint data from .dfs-checkpoints/__8\nres_1:\n  #  kind        name                          namespace\n     (string)    (string)                      (string)\n---  ----------  ----------------------------  -----------\n  1  Pod         ckan-5d74747649-92z9x         odata-blue\n  2  Pod         ckan-5d74747649-fzvd6         odata-blue\n  3  Pod         ckan-jobs-5d895695cf-wgrzr    odata-blue\n  4  Pod         datastore-db-944bfbc74-2nc7b  odata-blue\n  5  Pod         db-7dd99b8547-vpf57           odata-blue\n  6  Pod         pipelines-9f4466db-vlzm8      odata-blue\ncheckpoint saved: __9\n{'count_of_rows': 6, 'bytes': 40798, 'hash': 'adc31744dfc99a0d8cbe7b081f31d78b', 'dataset_name': None}\ncheckpoint:9\n```\n\n## Install\n\nThe only required core dependencies are Bash and Python3.7+\n\nTo get a compatible Python you can use [Miniconda](https://conda.io/miniconda.html):\n\n```\n$ wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh\n$ bash Miniconda3-latest-Linux-x86_64.sh\n$ wget https://raw.githubusercontent.com/OriHoch/dataflows-shell/master/environment.yaml\n$ conda env create -f environment.yaml\n$ conda activate dataflows-shell\n```\n\nInstall the dataflows-shell package\n\n```\npython3 -m pip install -U dataflows-shell\n```\n\nStart an interactive DataFlows shell session\n\n```\n$ dfs\n\nDataFlows Shell\n\npress <CTRL + C> to exit the shell\npress <Enter> to switch between DataFlows shell and system shell\ntype '--help' for the DataFlows Shell reference\n\ndfs > \n```\n\n## Documentation\n\n* [DataFlows Shell Tutorial](TUTORIAL.md)\n* [DataFlows Shell Reference](REFERENCE.md)\n* [DataFlows Shell Processors Reference](dataflows_shell/processors/README.md)\n* [DataFlows Processors Reference](https://github.com/datahq/dataflows/blob/master/PROCESSORS.md)",
        "description_content_type": "",
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://github.com/OriHoch/dataflows-shell",
        "keywords": "",
        "license": "MIT",
        "maintainer": "",
        "maintainer_email": "",
        "name": "dataflows-shell",
        "package_url": "https://pypi.org/project/dataflows-shell/",
        "platform": "",
        "project_url": "https://pypi.org/project/dataflows-shell/",
        "project_urls": {
            "Homepage": "https://github.com/OriHoch/dataflows-shell"
        },
        "release_url": "https://pypi.org/project/dataflows-shell/0.0.8/",
        "requires_dist": null,
        "requires_python": "",
        "summary": "Integrate DataFlows with shell scripts",
        "version": "0.0.8"
    },
    "last_serial": 4527346,
    "releases": {
        "0.0.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "08759b827e97ae49fce89c8717fd954e",
                    "sha256": "93d3d04e9e38cf4e2a5d264174aacc1f29ad5ed55f9766015d604bc74f89e4b8"
                },
                "downloads": -1,
                "filename": "dataflows_shell-0.0.1.tar.gz",
                "has_sig": false,
                "md5_digest": "08759b827e97ae49fce89c8717fd954e",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 9887,
                "upload_time": "2018-11-24T21:02:35",
                "url": "https://files.pythonhosted.org/packages/c5/bf/deb4eadaca2c9d53a7be9356a1959ae1f765979a53f258454c3c35db8a3c/dataflows_shell-0.0.1.tar.gz"
            }
        ],
        "0.0.2": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "34d6c0a12aecacb1c6941f98e8970181",
                    "sha256": "aff8198e02e710dd28cad6a5e3f06f2cee58ab7cf1b603cbe0cf841f390b1cf0"
                },
                "downloads": -1,
                "filename": "dataflows_shell-0.0.2.tar.gz",
                "has_sig": false,
                "md5_digest": "34d6c0a12aecacb1c6941f98e8970181",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 7787,
                "upload_time": "2018-11-25T10:49:23",
                "url": "https://files.pythonhosted.org/packages/fc/ea/90c8211d1c7fc4bad00d4370050a803e2ba09fa5c526987f6ccf08850f86/dataflows_shell-0.0.2.tar.gz"
            }
        ],
        "0.0.3": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "859144815cdbca1f974e1404c47dbaee",
                    "sha256": "4f02dbab73e6746ed230986bf698ffa4101f2a70f350fffe9f3d132e5f1809ae"
                },
                "downloads": -1,
                "filename": "dataflows_shell-0.0.3.tar.gz",
                "has_sig": false,
                "md5_digest": "859144815cdbca1f974e1404c47dbaee",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 9511,
                "upload_time": "2018-11-25T12:18:35",
                "url": "https://files.pythonhosted.org/packages/17/f7/5d8ee3b030959ea9a10cb4f77dc866fe795918574be31f03299e254545c0/dataflows_shell-0.0.3.tar.gz"
            }
        ],
        "0.0.4": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "3295cfc30310ff6fc7cbc625ca4dc131",
                    "sha256": "ea2a0dddd2f3660004ba6bcdf5ee16b1f056585419623ccf39e98e278e2d5482"
                },
                "downloads": -1,
                "filename": "dataflows_shell-0.0.4.tar.gz",
                "has_sig": false,
                "md5_digest": "3295cfc30310ff6fc7cbc625ca4dc131",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 9389,
                "upload_time": "2018-11-25T13:56:11",
                "url": "https://files.pythonhosted.org/packages/52/d5/0b150914e5bec0dac91f949966e8a6740f0a41d4a4debd648366482ad64a/dataflows_shell-0.0.4.tar.gz"
            }
        ],
        "0.0.5": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "5193a1c60de9fc2fab358f821d13294f",
                    "sha256": "e724c7c372d910eb33a26eb5feafaf9b6d45e0cb5aa5441aa061e9c51aef5f40"
                },
                "downloads": -1,
                "filename": "dataflows_shell-0.0.5.tar.gz",
                "has_sig": false,
                "md5_digest": "5193a1c60de9fc2fab358f821d13294f",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 9453,
                "upload_time": "2018-11-25T17:30:06",
                "url": "https://files.pythonhosted.org/packages/7f/1f/b8358b0fd646d5ed79ee2b79ece49d48ac177cdc61bf57df116f336f5f94/dataflows_shell-0.0.5.tar.gz"
            }
        ],
        "0.0.6": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "737410d9a0db184ab913ecfa8cb17b9e",
                    "sha256": "3bd44602570c69de46cd1d0fa9546bfffdac81838561f851b140e7d0fb224ad6"
                },
                "downloads": -1,
                "filename": "dataflows_shell-0.0.6.tar.gz",
                "has_sig": false,
                "md5_digest": "737410d9a0db184ab913ecfa8cb17b9e",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 12017,
                "upload_time": "2018-11-25T20:17:07",
                "url": "https://files.pythonhosted.org/packages/9a/79/08e423e546df552f5375f1d21c0af4d9e4d790bf3ce54b6ba65221509293/dataflows_shell-0.0.6.tar.gz"
            }
        ],
        "0.0.7": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "eacc118b9af4996256eedb96546177f2",
                    "sha256": "7eaf4fa780a0f726c289dd75203b886709696bc6578b0047ceee55ba3f6615be"
                },
                "downloads": -1,
                "filename": "dataflows_shell-0.0.7.tar.gz",
                "has_sig": false,
                "md5_digest": "eacc118b9af4996256eedb96546177f2",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 12056,
                "upload_time": "2018-11-25T20:21:22",
                "url": "https://files.pythonhosted.org/packages/7a/9a/172ab9ef1a5430b803bcc8a8e09fd3f2e021d6bd67b04553a49deda86ad5/dataflows_shell-0.0.7.tar.gz"
            }
        ],
        "0.0.8": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "eda11f1d9127ca0dc7068d63e6eea511",
                    "sha256": "a7c530a29a943c426f9ef7370a02c154881a7801ca2e301d792cc308c48a6c16"
                },
                "downloads": -1,
                "filename": "dataflows_shell-0.0.8.tar.gz",
                "has_sig": false,
                "md5_digest": "eda11f1d9127ca0dc7068d63e6eea511",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 10894,
                "upload_time": "2018-11-25T22:58:06",
                "url": "https://files.pythonhosted.org/packages/1f/98/b718adf14c950782e6c876d6bdacad407a644d7c7f8e9a9f1fc5ca1536ce/dataflows_shell-0.0.8.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "eda11f1d9127ca0dc7068d63e6eea511",
                "sha256": "a7c530a29a943c426f9ef7370a02c154881a7801ca2e301d792cc308c48a6c16"
            },
            "downloads": -1,
            "filename": "dataflows_shell-0.0.8.tar.gz",
            "has_sig": false,
            "md5_digest": "eda11f1d9127ca0dc7068d63e6eea511",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 10894,
            "upload_time": "2018-11-25T22:58:06",
            "url": "https://files.pythonhosted.org/packages/1f/98/b718adf14c950782e6c876d6bdacad407a644d7c7f8e9a9f1fc5ca1536ce/dataflows_shell-0.0.8.tar.gz"
        }
    ]
}