{
    "info": {
        "author": "Matt Garstka",
        "author_email": "matt@garstka.net",
        "bugtrack_url": null,
        "classifiers": [
            "Development Status :: 4 - Beta",
            "Intended Audience :: Developers",
            "Intended Audience :: Education",
            "Intended Audience :: Science/Research",
            "License :: OSI Approved :: MIT License",
            "Natural Language :: English",
            "Operating System :: Microsoft :: Windows",
            "Operating System :: POSIX :: Linux",
            "Programming Language :: Python :: 3.5",
            "Programming Language :: Python :: 3.6",
            "Topic :: Scientific/Engineering",
            "Topic :: System :: Distributed Computing",
            "Topic :: Utilities"
        ],
        "description": "# Welcome to idact!\n\n[![Build Status - master](https://travis-ci.com/garstka/idact.svg?token=cvggfL1vjmB383MxWGF4&branch=master)](https://travis-ci.com/garstka/idact)\n[![Build Status - develop](https://travis-ci.com/garstka/idact.svg?token=cvggfL1vjmB383MxWGF4&branch=develop)](https://travis-ci.com/garstka/idact)\n[![Coverage Status - master](https://coveralls.io/repos/github/garstka/idact/badge.svg?branch=master)](https://coveralls.io/github/garstka/idact?branch=master)\n[![PyPI - Python Version](https://img.shields.io/pypi/pyversions/idact.svg)](https://pypi.org/project/idact/)\n[![PyPI - License](https://img.shields.io/pypi/l/idact.svg)](https://pypi.org/project/idact/)\n[![PyPI](https://img.shields.io/pypi/v/idact.svg)](https://pypi.org/project/idact/)\n\nIdact, or *Interactive Data Analysis Convenience Tools*, is a Python 3.5+ library\nthat takes care of several tedious aspects of working with big data\non an HPC cluster.\n\n## Who is it for?\n\nData scientists or big data enthusiasts, who:\n - Perform computations on [Jupyter Notebook](http://jupyter.org/),\n  using libraries such as [NumPy](http://www.numpy.org/),\n  [pandas](https://pandas.pydata.org/),\n  [Matplotlib](https://matplotlib.org/),\n  or [bokeh](https://bokeh.pydata.org/en/latest/).\n - Have access to an HPC cluster with [Slurm](https://slurm.schedmd.com/)\n  as the job scheduler.\n - Would like to parallelize their computations across many nodes using\n  [Dask.distributed](http://distributed.dask.org/en/latest/), a library\n  for distributed computing.\n - May find that it takes too much manual effort to deploy Jupyter Notebook\n  and Dask on the cluster each time they need it.\n\n## Requirements\n\nPython 3.5+.\n\n### Client\n\n - Operating System: Windows or Linux\n - Recommended: [Jupyter Notebook](http://jupyter.org/)\n  or [JupyterLab](https://jupyterlab.readthedocs.io/en/stable/index.html)\n\n### Cluster\n\n - Operating System: Linux\n - Job Scheduler: [Slurm Workload Manager](https://slurm.schedmd.com/)\n - SSH access to a login (head) node.\n - Shared $HOME directory between nodes.\n - [Dask.distributed](http://distributed.dask.org/en/latest/) with [bokeh](https://bokeh.pydata.org/en/latest/).\n - [Jupyter Notebook](http://jupyter.org/)\n  or [JupyterLab](https://jupyterlab.readthedocs.io/en/stable/index.html)\n\n## Installation\n\n```\npython -m pip install idact\n```\n\nIf you're using [Conda](https://conda.io/docs/), you may want to update\nyour environment first:\n\n```\nconda update --all\n```\n\n## Code samples\n\n### Accessing a cluster\n\nCluster can be accessed with a public/private key pair via SSH.\n\n```python\nfrom idact import *\ncluster = add_cluster(name=\"short-cluster-name\",\n                      user=\"user\",\n                      host=\"login-node.cluster.example.com\",\n                      port=22,\n                      auth=AuthMethod.PUBLIC_KEY,\n                      key=\"~/.ssh/id_rsa\",\n                      install_key=False)\nnode = cluster.get_access_node()\nnode.connect()\n```\n\nTutorial:\n[01. Connecting to a cluster](https://garstka.github.io/idact/develop/html/_notebooks/01-Connecting_to_a_cluster.html)\n\n### Allocating and deallocating nodes\n\nNodes are allocated as a Slurm job.\nAfterwards, they can be used for deployments.\n\n```python\nimport bitmath\nnodes = cluster.allocate_nodes(nodes=8,\n                               cores=12,\n                               memory_per_node=bitmath.GiB(120),\n                               walltime=Walltime(hours=1, minutes=30),\n                               native_args={\n                                   '--partition': 'debug',\n                                   '--account': 'data-analysis-group'\n                               })\ntry:\n    nodes.wait(timeout=120.0)\nexcept TimeoutError:\n    nodes.cancel()\n```\n\nTutorial:\n[02. Allocating nodes](https://garstka.github.io/idact/develop/html/_notebooks/02-Allocating_nodes.html)\n\n### Deploying Jupyter Notebook\n\nJupyter Notebook is deployed on a cluster node,\nand made accessible through an SSH tunnel.\n\n```python\nnb = nodes[0].deploy_notebook()\nnb.open_in_browser()\n```\n\nTutorial:\n[03. Deploying Jupyter](https://garstka.github.io/idact/develop/html/_notebooks/03-Deploying_Jupyter.html)\n\n### Deploying Dask.distributed\n\nDask.distributed scheduler and workers are deployed\non cluster nodes, and their dashboards are made available\nthrough SSH tunnels.\n\n```python\ndd = deploy_dask(nodes[1:])\nclient = dd.get_client()\nclient.submit(...)\ndd.diagnostics.open_all()\n```\n\nTutorial:\n[04. Deploying Dask](https://garstka.github.io/idact/develop/html/_notebooks/04-Deploying_Dask.html),\n[09. Demo analysis](https://garstka.github.io/idact/develop/html/_notebooks/09a-Demo_analysis_-_local_part.html)\n\n### Managing cluster config\n\nLocal and remote cluster configuration can be saved, loaded,\nand copied to and from the cluster.\n\n```python\nsave_environment()\nload_environment()\n\npush_environment(cluster)\npull_environment(cluster)\n```\n\nTutorials:\n[01. Connecting to a cluster](https://garstka.github.io/idact/develop/html/_notebooks/01-Connecting_to_a_cluster.html),\n[05. Configuring idact on a cluster](https://garstka.github.io/idact/develop/html/_notebooks/05a-Configuring_idact_on_a_cluster_-_local_part.html)\n\n### Managing deployments\n\nDeployment objects can be serialized and copied between running program\ninstances, local or remote.\n\n```python\ncluster.push_deployment(nodes)\ncluster.push_deployment(nb)\ncluster.push_deployment(dd)\n\ncluster.pull_deployments()\n```\n\nTutorials:\n[06. Working on a cluster](https://garstka.github.io/idact/develop/html/_notebooks/06a-Working_on_a_cluster_-_local_part.html),\n[07. Adjusting timeouts](https://garstka.github.io/idact/develop/html/_notebooks/07-Adjusting_timeouts.html)\n\n### Quick deployment app\n\nQuick deployment app allocates nodes and deploys Jupyter notebook\nfrom command line:\n\n```\nidact-notebook short-cluster-name --nodes 3 --walltime 0:20:00\n```\n\nTutorial:\n[08. Using the quick deployment app](https://garstka.github.io/idact/develop/html/_notebooks/08a-Using_the_quick_deployment_app_-_local_part.html)\n\n## Documentation\n\nThe documentation contains detailed API description, tutorial notebooks,\nand other helpful information.\n\n - [Documentation - master](https://garstka.github.io/idact/master/html/index.html)\n - [Documentation - develop](https://garstka.github.io/idact/develop/html/index.html)\n - [All versions](https://garstka.github.io/idact/develop/html/docs_by_version.html)\n\n## Source code\n\nThe source code is available on [GitHub](https://github.com/garstka/idact).\n\n## License\n\nMIT License.\n\nThis library was developed under the supervision of Leszek Grzanka, PhD\nas a final project of the BEng in Computer Science program\nat the Faculty of Computer Science, Electronics and Telecommunications\nat AGH University of Science and Technology, Krakow.\n\n\n",
        "description_content_type": "text/markdown",
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://github.com/garstka/idact",
        "keywords": "interactive data analysis dask distributed jupyter notebook deployment slurm allocation cluster HPC ssh tunnel",
        "license": "MIT license",
        "maintainer": "",
        "maintainer_email": "",
        "name": "idact",
        "package_url": "https://pypi.org/project/idact/",
        "platform": "",
        "project_url": "https://pypi.org/project/idact/",
        "project_urls": {
            "Homepage": "https://github.com/garstka/idact"
        },
        "release_url": "https://pypi.org/project/idact/0.7/",
        "requires_dist": [
            "click (>=6.7)",
            "fabric3 (>=1.14)",
            "bitmath (>=1.3.1.2)",
            "python-dateutil (>=2.7.2)",
            "sshtunnel (>=0.1.4)",
            "paramiko (>=2.4.1)",
            "dask (>=0.18.2)",
            "distributed (>=1.22.0)",
            "requests (>=2.18.4)",
            "bokeh (>=0.13.0)",
            "jupyter (>=1.0.0)",
            "jupyterlab (>=0.35.4)"
        ],
        "requires_python": "",
        "summary": "A library that takes care of several tedious aspects of working with big data on an HPC cluster.",
        "version": "0.7"
    },
    "last_serial": 5109569,
    "releases": {
        "0.4.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "98e4bec02559c9ef90a5e22e127b13ea",
                    "sha256": "29010c5e28f18678ab5bdd9305e9138062158b0c832803c21979c1bbf9e7d9ba"
                },
                "downloads": -1,
                "filename": "idact-0.4.1-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "98e4bec02559c9ef90a5e22e127b13ea",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 144487,
                "upload_time": "2018-11-23T10:08:04",
                "url": "https://files.pythonhosted.org/packages/71/29/56d84c4d7a8220136378c3f811b9dfbaceb59580d0535984a2d639481f4a/idact-0.4.1-py3-none-any.whl"
            }
        ],
        "0.4.2": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "c14a8d061e6e34c65d8c765233f24525",
                    "sha256": "e90b27475236d0b38b48d5492a2e012c284a2788aedf05f0b5275e52f0e18495"
                },
                "downloads": -1,
                "filename": "idact-0.4.2-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "c14a8d061e6e34c65d8c765233f24525",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 144514,
                "upload_time": "2018-11-23T19:41:36",
                "url": "https://files.pythonhosted.org/packages/93/f6/69456ff1a884e30ab2e2ca816dbdec6c81ff9f4d739cc25936ce8f5d0ff1/idact-0.4.2-py3-none-any.whl"
            }
        ],
        "0.4.3": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "4d8c8a9fec5ed08e14a3c8d4a60fb4e0",
                    "sha256": "7c4dbe7a2eb7fbe2d4d60cc08f17aa94afed02b2c31e3e729c04a089abcc22f5"
                },
                "downloads": -1,
                "filename": "idact-0.4.3-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "4d8c8a9fec5ed08e14a3c8d4a60fb4e0",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 145046,
                "upload_time": "2018-11-26T01:17:32",
                "url": "https://files.pythonhosted.org/packages/56/e4/6d78ff85103243fb098ce4df5a178aa3f9b1d4bc669591efece023c21dc0/idact-0.4.3-py3-none-any.whl"
            }
        ],
        "0.5": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "11ebf8551321e1016b2547363b948a6e",
                    "sha256": "c01976893ee2bc10c85d75651b50d46113a815c4639c2a69dd4a9bc082832369"
                },
                "downloads": -1,
                "filename": "idact-0.5-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "11ebf8551321e1016b2547363b948a6e",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 145657,
                "upload_time": "2018-12-01T13:04:53",
                "url": "https://files.pythonhosted.org/packages/89/6d/ba6eb2865f41653113a2f80320694fcd16e20156a9129d45220eb641f8e7/idact-0.5-py3-none-any.whl"
            }
        ],
        "0.5.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "8d4622858d0da212282bfdf118f0baa9",
                    "sha256": "c95650c63cf759e2d25e556334b65f353334ccbef7168814a5b0a351817bc920"
                },
                "downloads": -1,
                "filename": "idact-0.5.1-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "8d4622858d0da212282bfdf118f0baa9",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 145750,
                "upload_time": "2018-12-07T02:53:02",
                "url": "https://files.pythonhosted.org/packages/19/5f/9e18ccf74937774f0446a2eeeef0a9ffda9ced833fc2ae80177ca3923248/idact-0.5.1-py3-none-any.whl"
            }
        ],
        "0.6": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "209deb09c5f7e8f788393ea1398409d0",
                    "sha256": "d0e2770c3a07c042fcd0d2525d4fd5d8adfca615c224a4a02521603ec78e1bd2"
                },
                "downloads": -1,
                "filename": "idact-0.6-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "209deb09c5f7e8f788393ea1398409d0",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 145955,
                "upload_time": "2019-01-02T19:44:25",
                "url": "https://files.pythonhosted.org/packages/a6/5c/a47b4250350eb312a977a70ce3d5056a02a16c671cdec49aa827f114bce2/idact-0.6-py3-none-any.whl"
            }
        ],
        "0.7": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "cf8c62b1476981d6d43bf5161380ee38",
                    "sha256": "539f63edce9d4f47dcd0492c548319a857e892b05b0ffface62941481be0b5d5"
                },
                "downloads": -1,
                "filename": "idact-0.7-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "cf8c62b1476981d6d43bf5161380ee38",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 147050,
                "upload_time": "2019-04-07T10:30:27",
                "url": "https://files.pythonhosted.org/packages/5a/db/28b0faf494fa887ded6e926817b4643e5f9be34b8373ce9e136906956c75/idact-0.7-py3-none-any.whl"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "cf8c62b1476981d6d43bf5161380ee38",
                "sha256": "539f63edce9d4f47dcd0492c548319a857e892b05b0ffface62941481be0b5d5"
            },
            "downloads": -1,
            "filename": "idact-0.7-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "cf8c62b1476981d6d43bf5161380ee38",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": null,
            "size": 147050,
            "upload_time": "2019-04-07T10:30:27",
            "url": "https://files.pythonhosted.org/packages/5a/db/28b0faf494fa887ded6e926817b4643e5f9be34b8373ce9e136906956c75/idact-0.7-py3-none-any.whl"
        }
    ]
}