{
    "info": {
        "author": "radix.ai",
        "author_email": "developers@radix.ai",
        "bugtrack_url": null,
        "classifiers": [
            "Development Status :: 4 - Beta",
            "Intended Audience :: Developers",
            "License :: OSI Approved :: MIT License",
            "Operating System :: OS Independent",
            "Programming Language :: Python :: 3",
            "Programming Language :: Python :: 3.6",
            "Topic :: Software Development :: Libraries :: Python Modules"
        ],
        "description": "[![CircleCI](https://img.shields.io/circleci/token/39b1cfd1096f95ab3c6aeb839d86763ea2a261aa/project/github/radix-ai/graphchain/master.svg)](https://circleci.com/gh/radix-ai/graphchain/tree/master) [![License](https://img.shields.io/github/license/mashape/apistatus.svg)](https://choosealicense.com/licenses/mit/) [![PyPI](https://img.shields.io/pypi/v/graphchain.svg)](https://pypi.python.org/pypi/graphchain/) [![Documentation](http://readthedocs.org/projects/graphchain/badge/?version=latest)](http://graphchain.readthedocs.io/)\n\n# Graphchain\n\n## What is graphchain?\n\nGraphchain is like [joblib.Memory](https://joblib.readthedocs.io/en/latest/memory.html#memory) for dask graphs. [Dask graph computations](http://dask.pydata.org/en/latest/spec.html) are cached to a local or remote location of your choice, specified by a [PyFilesystem FS URL](https://docs.pyfilesystem.org/en/latest/openers.html).\n\nWhen you change your dask graph (by changing a computation's implementation or its inputs), graphchain will take care to only recompute the minimum number of computations necessary to fetch the result. This allows you to iterate quickly over your graph without spending time on recomputing previously computed keys.\n\n<p align=\"center\">\n    <img src=\"https://imgs.xkcd.com/comics/is_it_worth_the_time_2x.png\" width=\"400\" /><br />\n    <span>Source: <a href=\"https://xkcd.com/1205/\">xkcd.com/1205/</a></span>\n</p>\n\nThe main difference between graphchain and joblib.Memory is that in graphchain a computation's materialised inputs are _not_ serialised and hashed (which can be very expensive when the inputs are large objects such as pandas DataFrames). Instead, a chain of hashes (hence the name graphchain) of the computation object and its dependencies (which are also computation objects) is used to identify the cache file.\n\nAdditionally, the result of a computation is only cached if it is estimated that loading that computation from cache will save time compared to simply computing the computation. The decision on whether to cache depends on the characteristics of the cache location, which are different when caching to the local filesystem compared to caching to S3 for example.\n\n## Usage by example\n\n### Basic usage\n\nInstall graphchain with pip to get started:\n\n```bash\npip install graphchain\n```\n\nTo demonstrate how graphchain can save you time, let's first create a simple dask graph that (1) creates a few pandas DataFrames, (2) runs a relatively heavy operation on these DataFrames, and (3) summarises the results.\n\n```python\nimport dask\nimport graphchain\nimport pandas as pd\n\ndef create_dataframe(num_rows, num_cols):\n    print('Creating DataFrame...')\n    return pd.DataFrame(data=[range(num_cols)]*num_rows)\n\ndef complicated_computation(df, num_quantiles):\n    print('Running complicated computation on DataFrame...')\n    return df.quantile(q=[i / num_quantiles for i in range(num_quantiles)])\n\ndef summarise_dataframes(*dfs):\n    print('Summing DataFrames...')\n    return sum(df.sum().sum() for df in dfs)\n\ndsk = {\n    'df_a': (create_dataframe, 10_000, 1000),\n    'df_b': (create_dataframe, 10_000, 1000),\n    'df_c': (complicated_computation, 'df_a', 2048),\n    'df_d': (complicated_computation, 'df_b', 2048),\n    'result': (summarise_dataframes, 'df_c', 'df_d')\n}\n```\n\nUsing `dask.get` to fetch the `'result'` key takes about 6 seconds:\n\n```python\n>>> %time dask.get(dsk, 'result')\n\nCreating DataFrame...\nRunning complicated computation on DataFrame...\nCreating DataFrame...\nRunning complicated computation on DataFrame...\nSumming DataFrames...\n\nCPU times: user 7.39 s, sys: 686 ms, total: 8.08 s\nWall time: 6.19 s\n```\n\nOn the other hand, using `graphchain.get` for the first time to fetch `'result'` takes only 4 seconds:\n\n```python\n>>> %time graphchain.get(dsk, 'result')\n\nCreating DataFrame...\nRunning complicated computation on DataFrame...\nSumming DataFrames...\n\nCPU times: user 4.7 s, sys: 519 ms, total: 5.22 s\nWall time: 4.04 s\n```\n\nThe reason `graphchain.get` is faster than `dask.get` is because it can load `df_b` and `df_d` from cache after `df_a` and `df_c` have been computed and cached. Note that graphchain will only cache the result of a computation if loading that computation from cache is estimated to be faster than simply running the computation.\n\nRunning `graphchain.get` a second time to fetch `'result'` will be almost instant since this time the result itself is also available from cache:\n\n```python\n>>> %time graphchain.get(dsk, 'result')\n\nCPU times: user 4.79 ms, sys: 1.79 ms, total: 6.58 ms\nWall time: 5.34 ms\n```\n\nNow let's say we want to change how the result is summarised from a sum to an average:\n\n```python\ndef summarise_dataframes(*dfs):\n    print('Averaging DataFrames...')\n    return sum(df.mean().mean() for df in dfs) / len(dfs)\n```\n\nIf we then ask graphchain to fetch `'result'`, it will detect that only `summarise_dataframes` has changed and therefore only recompute this function with inputs loaded from cache:\n\n```python\n>>> %time graphchain.get(dsk, 'result')\n\nAveraging DataFrames...\n\nCPU times: user 123 ms, sys: 37.2 ms, total: 160 ms\nWall time: 86.6 ms\n```\n\n### Storing the graphchain cache remotely\n\nGraphchain's cache is by default `./__graphchain_cache__`, but you can ask graphchain to use a cache at any [PyFilesystem FS URL](https://docs.pyfilesystem.org/en/latest/openers.html) such as `s3://mybucket/__graphchain_cache__`:\n\n```python\ngraphchain.get(dsk, 'result', location='s3://mybucket/__graphchain_cache__')\n```\n\n### Excluding keys from being cached\n\nIn some cases you may not want a key to be cached. To avoid writing certain keys to the graphchain cache, you can use the `skip_keys` argument:\n\n```python\ngraphchain.get(dsk, 'result', skip_keys=['result'])\n```\n\n### Using graphchain with dask.delayed\n\nAlternatively, you can use graphchain together with dask.delayed for easier dask graph creation:\n\n```python\n@dask.delayed\ndef create_dataframe(num_rows, num_cols):\n    print('Creating DataFrame...')\n    return pd.DataFrame(data=[range(num_cols)]*num_rows)\n\n@dask.delayed\ndef complicated_computation(df, num_quantiles):\n    print('Running complicated computation on DataFrame...')\n    return df.quantile(q=[i / num_quantiles for i in range(num_quantiles)])\n\n@dask.delayed\ndef summarise_dataframes(*dfs):\n    print('Summing DataFrames...')\n    return sum(df.sum().sum() for df in dfs)\n\ndf_a = create_dataframe(num_rows=50_000, num_cols=500, seed=42)\ndf_b = create_dataframe(num_rows=50_000, num_cols=500, seed=42)\ndf_c = complicated_computation(df_a, window=3)\ndf_d = complicated_computation(df_b, window=3)\nresult = summarise_dataframes(df_c, df_d)\n```\n\nAfter which you can compute `result` by setting the `delayed_optimize` method to `graphchain.optimize`:\n\n```python\nwith dask.config.set(scheduler='sync', delayed_optimize=graphchain.optimize):\n    result.compute(location='s3://mybucket/__graphchain_cache__')\n```\n\n## Developed by radix.ai\n\nAt [radix.ai](https://radix.ai/), we invent, design and develop AI-powered software.\n\nHere are some examples of what we do with Machine Learning, the technology behind AI:\n- Help job seekers find a job. On the [Belgian Public Employment Service website](https://www.vdab.be/), we serve job recommendations based on your CV.\n- Help hospitals save time. We extract diagnoses from patient discharge letters.\n- Help publishers calculate their impact, by detecting copycat articles.\n\nYou can follow our adventures on [medium](https://medium.com/radix-ai-blog).\n\n\n",
        "description_content_type": "text/markdown",
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://github.com/radix-ai/graphchain",
        "keywords": "dask graph cache distributed",
        "license": "",
        "maintainer": "",
        "maintainer_email": "",
        "name": "graphchain",
        "package_url": "https://pypi.org/project/graphchain/",
        "platform": "",
        "project_url": "https://pypi.org/project/graphchain/",
        "project_urls": {
            "Bug Reports": "https://github.com/radix-ai/graphchain/issues",
            "Homepage": "https://github.com/radix-ai/graphchain",
            "Source": "https://github.com/radix-ai/graphchain"
        },
        "release_url": "https://pypi.org/project/graphchain/1.1.0/",
        "requires_dist": [
            "cloudpickle",
            "dask",
            "fs-s3fs",
            "joblib",
            "lz4"
        ],
        "requires_python": "",
        "summary": "An efficient cache for the execution of dask graphs",
        "version": "1.1.0"
    },
    "last_serial": 5203579,
    "releases": {
        "1.0.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "cd5bff8f22ba739d89945f0954b6f9c8",
                    "sha256": "59a708bd5701ab905716d55f8e280ee94c7b201c2291f8bb21a67d4eec259e88"
                },
                "downloads": -1,
                "filename": "graphchain-1.0.0.tar.gz",
                "has_sig": false,
                "md5_digest": "cd5bff8f22ba739d89945f0954b6f9c8",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 14132,
                "upload_time": "2018-10-20T10:57:54",
                "url": "https://files.pythonhosted.org/packages/f7/a1/b0f9a1f7d5ceef14a1f377d33232733cfed1abe880d363163e0f7e090d47/graphchain-1.0.0.tar.gz"
            }
        ],
        "1.1.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "18908f059ae65876538b824cdc265c32",
                    "sha256": "e604e5326ffa6cf62d333e1ce50018c4fa380ad9598330c2aea38e8930f7b7b8"
                },
                "downloads": -1,
                "filename": "graphchain-1.1.0-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "18908f059ae65876538b824cdc265c32",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 15029,
                "upload_time": "2019-04-29T15:12:48",
                "url": "https://files.pythonhosted.org/packages/e8/4d/a23ac756f15767c3725893a18506ada073ab297039e01c2aa9fd10400165/graphchain-1.1.0-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "d270551d6000b9f5423411250bdb763d",
                    "sha256": "f34d8cf983a64ec8f2c090395e141f5f464ca335a26a9a166e8146aa7458a86e"
                },
                "downloads": -1,
                "filename": "graphchain-1.1.0.tar.gz",
                "has_sig": false,
                "md5_digest": "d270551d6000b9f5423411250bdb763d",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 15733,
                "upload_time": "2019-04-29T15:12:54",
                "url": "https://files.pythonhosted.org/packages/88/66/abd03a4d0fb929233d4851a08897173799bba9770799cd768336967f2ea9/graphchain-1.1.0.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "18908f059ae65876538b824cdc265c32",
                "sha256": "e604e5326ffa6cf62d333e1ce50018c4fa380ad9598330c2aea38e8930f7b7b8"
            },
            "downloads": -1,
            "filename": "graphchain-1.1.0-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "18908f059ae65876538b824cdc265c32",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": null,
            "size": 15029,
            "upload_time": "2019-04-29T15:12:48",
            "url": "https://files.pythonhosted.org/packages/e8/4d/a23ac756f15767c3725893a18506ada073ab297039e01c2aa9fd10400165/graphchain-1.1.0-py3-none-any.whl"
        },
        {
            "comment_text": "",
            "digests": {
                "md5": "d270551d6000b9f5423411250bdb763d",
                "sha256": "f34d8cf983a64ec8f2c090395e141f5f464ca335a26a9a166e8146aa7458a86e"
            },
            "downloads": -1,
            "filename": "graphchain-1.1.0.tar.gz",
            "has_sig": false,
            "md5_digest": "d270551d6000b9f5423411250bdb763d",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 15733,
            "upload_time": "2019-04-29T15:12:54",
            "url": "https://files.pythonhosted.org/packages/88/66/abd03a4d0fb929233d4851a08897173799bba9770799cd768336967f2ea9/graphchain-1.1.0.tar.gz"
        }
    ]
}