{ "info": { "author": "Stefano Fioravanzo", "author_email": "stefano.fioravanzo@gmail.com", "bugtrack_url": null, "classifiers": [], "description": "![Kale Logo](https://raw.githubusercontent.com/StefanoFioravanzo/kale/master/docs/imgs/kale_logo.png)\n\n---------------------------------------------------------------------\n\nKale is a Python package that aims at automatically deploy a general purpose Jupyter Notebook as a running [Kubeflow Pipelines](https://github.com/kubeflow/pipelines) instance, without requiring the use the specific KFP DSL.\n\nThe general idea of kale is to automatically arrange the cells included in a notebook, and transform them into a unified KFP-compliant pipeline. To do so, the user is only required to decide which cells correspond to which pipeline step, by the use of tags. In this way, a researcher can better focus on building and testing its code locally, and then scale it in a simple, organized and controlled way.\n\n## Tagging language\n\nJupyter provides a tagging feature out-of-the-box, that lets you associate each cells with custom defined tags. The feature is available also in JupyterLab via the [jupyterlab-celltags](https://github.com/jupyterlab/jupyterlab-celltags) extension.\n\nThe tags are used to tell Kale how to convert the notebook's code cells into an execution graph, by specifying the execution dependencies between the pipeline steps and which code cells to merge together.\n\nThis is a list of tags recognized by Kale:\n\n| Tag | Description | Example |\n| :---: | :---: | :---: |\n| `^block:(;)*$` | Assign the current cell to a (multiple) pipeline step | `block:train-model`
`block:processing-A;processing-B`| \n| `^prev:(;)*$` | Define an execution dependency of the current cell to `n` other pipeline steps | `prev:load-dataset\n| imports|functions | Tell Kale to add this code block at the beginning of every pipeline code block. Useful to add imports/function to every pipeline step | - | \n| `skip` | 'Hide' the current cell from Kale. | - |\n\nWhere `` is matched against the regex: `[a-z0-9]`. So any string containing only digits and lowercase characters.\n\n## Installation\n\nKale is provided as a Python package. Just clone the repository to your local machine and install the package in a virtual environment. \n\n```bash\n# Clone the repo to your local environment\ngit clone https://github.com/kubeflow-kale/kale\ncd kale\n# Install the package in your virtualenv\npython setup.py install\n```\n\n## Getting Started\n\nFirst you need to have a running Kubeflow instance (Kubeflow [getting started guide](https://www.kubeflow.org/docs/started/getting-started/)).\n\n\nKale provides a CLI command. Run `kale --help` for a detailed description of the execution parameters.\n\nExample:\n\n```bash\nkale --nb examples/base_example_numpy.ipynb \\\n\t--pipeline_name numpy_example \\\n\t--pipeline_descr \"Numpy Example\" \\\n\t--docker_image stefanofioravanzo/kale-kfp-examples:0.1\n```\nThis will produce a python script `examples/kfp_numpy_examples.kfp.py` containing all the definitions of the KFP stand-alone functions and the necessary code to define and deploy a pipeline. If run with the `--deploy` flag, Kale will try to automatically deploy the generated pipeline (default KFP url is `localhost:8080`, see `kale` CLI parameters for customization).\n\nSome example Notebooks can be found in the [examples](https://github.com/kubeflow-kale/examples) repository.\n\n## Architecture\n\n`core.py` contains the `Kale` class, main entry-point of the application. The function `run` does the conversion from jupyter notebook to kfp pipeline by calling all the other sub-modules.\n\nKale was developed with a modular design. There are 4 main modules, each with a specific functionality.\n\n#### 1. nbparser\n\nThis module both defines the Kale's Jupyter tagging language and provides the machinery to convert a tagged notebook to a NetworkX graph representing the resulting KFP pipeline. The main function is `parse_notebook()`, which takes as input a tagged notebook and returns the graph. The pipeline steps Python code is contained inside a special attributed of the graph's nodes.\n\n#### 2. static_analysis\n\nThe purpose of this module is to detect the data dependencies between the code snippets in the execution graph. This is achieved by running PyFlakes that is able to detect all the potential missing variables errors, plus some custom routines that iterate over the AST. `dep_analysis.variables_dependencies_detection()` is the main function called by Kale to start the static analysis process.\n\n#### 3. marshal\n\nMarshal is a module that implements two dispater classes. `PatternDispatcher` can register functions based a regex pattern, and then call one of these functions based on a user provided string. `TypeDispatcher` can register functions based on a pattern as well, but dispatches a function based on the *type* of the input objects instead of a string.\n\nExample:\n\n```python\nfrom kale.marshal import resource_save, resource_load\n\n# register functions to load and save numpy objects\n@resource_load.register('.*\\.npy') # match anything ending in '.npy'\ndef resource_numpy_load(uri, **kwargs):\n return np.load(uri)\n\n@resource_save.register('numpy\\..*') # match anything starting with 'numpy'\ndef resource_numpy_save(obj, path, **kwargs):\n return np.save(path+\".npy\", obj)\n\na = np.random.random((10,10))\nresource_save(a, 'marshal_dir/')\nb = resource_load('marshal_dir/a.npy') # a == b\n```\n\nCommon backends to support multiple data types are registered in `marshal/backends.py`. The idea is to provide backends for the most used data types and have a common fallback to standard `dill` serialization in case the data type is not recognized.\n\n**NB:** The code in this module is not used directly by Kale during the conversion process from notebook to pipeline, but is instead injected at the beginning and at the end of each generated pipeline step to serialize and de-serialize objects that need to be passed between pipeline steps. Have a look at `templates/function_template.txt` and at the generated python script (e.g. `kfp_numpy_example.kfp.py`).\n\n#### 4. codegen\n\nThis module provides a single function `gen_kfp_code` that exploits the Jinja2 templating engine to generate a fully functional python script. The templates used are under the `templates/` folder.\n\n## Flask Server\n\nThe package provides the `kale_server` CLI command that runs a Flask server. The Flask server accepts requests at `localhost:5000/kale`. The API accepts POST requests containing a Jupyter Notebook in raw format (JSON) and will call the main Kale module to deploy the notebook to a KFP instance.\n\n`kale/` API available parameters:\n\n- `nb`: Raw JSON Jupyter Notebook\n- `deploy`: Boolean Flag. `True` to deploy the Notebook automatically\n- `kfp_port`: Port of the running KFP instance\n- `pipeline_name`: Name of the resulting KFP pipeline\n- `pipeline_descr`: Description of the resulting KFP pipeline\n- `docker_image`: Docker image to use for the pipeline steps\n\n\n\n", "description_content_type": "text/markdown", "docs_url": null, "download_url": "", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "https://github.com/kubeflow-kale/kale", "keywords": "", "license": "MIT", "maintainer": "", "maintainer_email": "", "name": "kubeflow-kale", "package_url": "https://pypi.org/project/kubeflow-kale/", "platform": "", "project_url": "https://pypi.org/project/kubeflow-kale/", "project_urls": { "Homepage": "https://github.com/kubeflow-kale/kale" }, "release_url": "https://pypi.org/project/kubeflow-kale/0.3.4/", "requires_dist": [ "kfp (<0.2,>=0.1.31)", "autopep8 (<1.5,>=1.4)", "nbformat (<5.0,>=4.4)", "networkx (<3.0,>=2.3)", "jinja2 (<3.0,>=2.10)", "graphviz (<1.0,>=0.13)", "pyflakes (>=2.1.1)", "papermill (<3.0,>=1.2)" ], "requires_python": ">=3.6.0", "summary": "Convert JupyterNotebooks to Kubeflow Pipelines deployments", "version": "0.3.4" }, "last_serial": 5945047, "releases": { "0.1.0": [ { "comment_text": "", "digests": { "md5": "e1290034a9be1db8675ba6e2fbd1c73f", "sha256": "8f1fcc6378ae02fe54a4f41f0f60a3d04e88ece308e39444b74926d1db5481f9" }, "downloads": -1, "filename": "kubeflow_kale-0.1.0-py3-none-any.whl", "has_sig": false, "md5_digest": "e1290034a9be1db8675ba6e2fbd1c73f", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": ">=3.6.0", "size": 29191, "upload_time": "2019-09-02T16:47:38", "url": "https://files.pythonhosted.org/packages/c2/a3/bb8198387a5c44b05d9658cc396b82d9c3cfcaf39907a64b018569b3a3a3/kubeflow_kale-0.1.0-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "a6e7d1f1b4ac6d3b30301f3b6f1e2d31", "sha256": "efe07eeefa2d2675c8173e8e3e7a10ee1344bc744d3e83a52eb9b37ed3356af7" }, "downloads": -1, "filename": "kubeflow-kale-0.1.0.tar.gz", "has_sig": false, "md5_digest": "a6e7d1f1b4ac6d3b30301f3b6f1e2d31", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.6.0", "size": 238487, "upload_time": "2019-09-02T16:47:45", "url": "https://files.pythonhosted.org/packages/86/d5/d2a1eef8eed073edefdad8f8711ef4c424c72ef57c6ac5c4dd3b1cc90036/kubeflow-kale-0.1.0.tar.gz" } ], "0.2.0": [ { "comment_text": "", "digests": { "md5": "722ed218b7c4d94a0a9277bb65c1523f", "sha256": "291f04243a3c1b6072fc95c9f12d8b2e1f5628e86b1eb68b7096b6d4872fe07f" }, "downloads": -1, "filename": "kubeflow_kale-0.2.0-py3-none-any.whl", "has_sig": false, "md5_digest": "722ed218b7c4d94a0a9277bb65c1523f", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": ">=3.6.0", "size": 30221, "upload_time": "2019-09-22T11:09:24", "url": "https://files.pythonhosted.org/packages/a0/a1/08ccf3c97d8e2fb10ed45fd5648353b9c675ccb5629814cc607e97412a69/kubeflow_kale-0.2.0-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "b4ecc312ca40fd84191da1ac8f8b6a09", "sha256": "96e4d58b24e98a8f75024f01766222fd91421e8b1da47de94b629f1e545dc1ca" }, "downloads": -1, "filename": "kubeflow-kale-0.2.0.tar.gz", "has_sig": false, "md5_digest": "b4ecc312ca40fd84191da1ac8f8b6a09", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.6.0", "size": 239460, "upload_time": "2019-09-22T11:09:31", "url": "https://files.pythonhosted.org/packages/31/48/c59013f3a3e46d88a3159fcbcb9e751f63e33c9a4d362b3f03d961c5353c/kubeflow-kale-0.2.0.tar.gz" } ], "0.2.1": [ { "comment_text": "", "digests": { "md5": "ce3ff96be7b29850b45191be6f82693a", "sha256": "8416d81c796f53303eb148b92ec7ab5608c7376a42ac2a55fd09bcbb9a7c8e9f" }, "downloads": -1, "filename": "kubeflow_kale-0.2.1-py3-none-any.whl", "has_sig": false, "md5_digest": "ce3ff96be7b29850b45191be6f82693a", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": ">=3.6.0", "size": 30217, "upload_time": "2019-09-22T12:27:22", "url": "https://files.pythonhosted.org/packages/ec/d0/f6c6ff4c9c636253d8a3e9b5c79159ee7a070d13f794802d1955ce9ff972/kubeflow_kale-0.2.1-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "b8ed8e106431cc2d04a3cec5b41495c0", "sha256": "c138c3d346a604d885ed6e7991f91d65234cf4c47a438c27e7422429da5b7342" }, "downloads": -1, "filename": "kubeflow-kale-0.2.1.tar.gz", "has_sig": false, "md5_digest": "b8ed8e106431cc2d04a3cec5b41495c0", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.6.0", "size": 239461, "upload_time": "2019-09-22T12:27:33", "url": "https://files.pythonhosted.org/packages/2f/5d/65abefa8a84ba91621f7314965e0b27cd9cef5d77e9883fb375a67a604cf/kubeflow-kale-0.2.1.tar.gz" } ], "0.2.2": [ { "comment_text": "", "digests": { "md5": "03f45ba2288111e9d24fafdb6bf09552", "sha256": "8ca31656d73c2cb1b4a387aab930c5c36590da02622698ca2ded8d58fea084ec" }, "downloads": -1, "filename": "kubeflow_kale-0.2.2-py3-none-any.whl", "has_sig": false, "md5_digest": "03f45ba2288111e9d24fafdb6bf09552", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": ">=3.6.0", "size": 30196, "upload_time": "2019-09-22T15:03:36", "url": "https://files.pythonhosted.org/packages/37/07/d1096911fd240b9eb723106a8963b4a03c6e294ef85c6f66cc2bd8bb9548/kubeflow_kale-0.2.2-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "a1da2faece33ce198d4124561f1aa747", "sha256": "d2e21eed68af9325f401ebf6685791f4fde968e99dbbab3274edb2ff3ed889d3" }, "downloads": -1, "filename": "kubeflow-kale-0.2.2.tar.gz", "has_sig": false, "md5_digest": "a1da2faece33ce198d4124561f1aa747", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.6.0", "size": 239465, "upload_time": "2019-09-22T15:03:39", "url": "https://files.pythonhosted.org/packages/eb/37/0f1453f41e8ed0ad7e08f2e69d214ef94e017798dc4dc2071e99a5c35803/kubeflow-kale-0.2.2.tar.gz" } ], "0.3.0": [ { "comment_text": "", "digests": { "md5": "ab8f7ac19f97fdaa7855213aa7cd9497", "sha256": "cd9f4ff276432ea7579e78d58e928db33c263c796b52815aaabde9ed432981b3" }, "downloads": -1, "filename": "kubeflow_kale-0.3.0-py3-none-any.whl", "has_sig": false, "md5_digest": "ab8f7ac19f97fdaa7855213aa7cd9497", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": ">=3.6.0", "size": 30794, "upload_time": "2019-09-29T16:29:05", "url": "https://files.pythonhosted.org/packages/7e/5e/421658fd2e98fa2ef5ae2c373c3eea12fa0d057f8382bb04b9648f35f619/kubeflow_kale-0.3.0-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "97f4565b276184b47b37613b52572a8b", "sha256": "e879377f86cb667c9a80d47ae01b4562ca335b68b32fd5da0336042687b97668" }, "downloads": -1, "filename": "kubeflow-kale-0.3.0.tar.gz", "has_sig": false, "md5_digest": "97f4565b276184b47b37613b52572a8b", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.6.0", "size": 240038, "upload_time": "2019-09-29T16:29:07", "url": "https://files.pythonhosted.org/packages/ec/92/5a4e234c6af891d05d4c909d427d4516e8024c23b410933be8bb7c7cd2f9/kubeflow-kale-0.3.0.tar.gz" } ], "0.3.1": [ { "comment_text": "", "digests": { "md5": "c9b0e0f425ddfc98a42cf1c9faeacd0e", "sha256": "bfb4fbcb7e37cfccf246218fa22698c3f0ebaf9a91f6e8cd3657b859bf14acd4" }, "downloads": -1, "filename": "kubeflow_kale-0.3.1-py3-none-any.whl", "has_sig": false, "md5_digest": "c9b0e0f425ddfc98a42cf1c9faeacd0e", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": ">=3.6.0", "size": 30774, "upload_time": "2019-09-29T16:53:50", "url": "https://files.pythonhosted.org/packages/13/37/154729b86f1ae3519f192ca6d11c83217c614daf5e86643b82bcf54cecef/kubeflow_kale-0.3.1-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "bea2334ea60666d55b10f3067b6dd4df", "sha256": "5663777b725d3bca1ea46797cc0bb30cfa995fec7a2a3e55c7b07c3bc4fd2c1f" }, "downloads": -1, "filename": "kubeflow-kale-0.3.1.tar.gz", "has_sig": false, "md5_digest": "bea2334ea60666d55b10f3067b6dd4df", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.6.0", "size": 240020, "upload_time": "2019-09-29T16:53:58", "url": "https://files.pythonhosted.org/packages/84/09/805f5a5faa614e47f861ec30131f5fb38dc3aa7d7cc266ae620b24ff7954/kubeflow-kale-0.3.1.tar.gz" } ], "0.3.2": [ { "comment_text": "", "digests": { "md5": "56027f793e12a8a07c3fe08bc06ce873", "sha256": "24e96ad9be95a0fb02d580ea6dd4c6feaf96aeed7e1c776f7796774149d4dc06" }, "downloads": -1, "filename": "kubeflow_kale-0.3.2-py3-none-any.whl", "has_sig": false, "md5_digest": "56027f793e12a8a07c3fe08bc06ce873", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": ">=3.6.0", "size": 30961, "upload_time": "2019-10-07T10:20:54", "url": "https://files.pythonhosted.org/packages/39/3a/16f89f5aaf4c2655a20d50009b79a37a421c3bc69a3906837c295cb2fa6a/kubeflow_kale-0.3.2-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "98074e5a5d8d09f4a809db34a43cd5cf", "sha256": "279c1f264e649589cb7bfbd634521b60f91dd4ac6fe5a67040e686a141f09e5e" }, "downloads": -1, "filename": "kubeflow-kale-0.3.2.tar.gz", "has_sig": false, "md5_digest": "98074e5a5d8d09f4a809db34a43cd5cf", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.6.0", "size": 240234, "upload_time": "2019-10-07T10:21:05", "url": "https://files.pythonhosted.org/packages/39/a3/304d9ba930c497694dfb083d527a530d10ce2e0c858ef76e9ee8a60f9918/kubeflow-kale-0.3.2.tar.gz" } ], "0.3.3": [ { "comment_text": "", "digests": { "md5": "7d99d0fa2da1cec544a25b780f520f18", "sha256": "4b83761869b3d1c9ce7205e2a39838207fa320027d92c8c095dd31e1a9177d53" }, "downloads": -1, "filename": "kubeflow_kale-0.3.3-py3-none-any.whl", "has_sig": false, "md5_digest": "7d99d0fa2da1cec544a25b780f520f18", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": ">=3.6.0", "size": 30947, "upload_time": "2019-10-07T14:48:01", "url": "https://files.pythonhosted.org/packages/64/fe/e8fe8a33d7f910c6324e4089ca6597b3d1cd19e826ca7a7eb7f444d09e23/kubeflow_kale-0.3.3-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "dbdcade7f14a8e4e83e73aef83c058d4", "sha256": "ed45115003bf8afd09c56fc7dcb9610eb0fa4695246b9ee81d99e414f39caa49" }, "downloads": -1, "filename": "kubeflow-kale-0.3.3.tar.gz", "has_sig": false, "md5_digest": "dbdcade7f14a8e4e83e73aef83c058d4", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.6.0", "size": 240249, "upload_time": "2019-10-07T14:48:07", "url": "https://files.pythonhosted.org/packages/d3/92/df020762ed75a305af2cfa885014dc2c0b7d17132299b1e5167f5ba652cc/kubeflow-kale-0.3.3.tar.gz" } ], "0.3.4": [ { "comment_text": "", "digests": { "md5": "70e6d01de3710bfb8883a95d83dfe1bf", "sha256": "122f5fe9acf623895a0be2f8c3de3158d0d46a592e69b7b3c7766f311e08062d" }, "downloads": -1, "filename": "kubeflow_kale-0.3.4-py3-none-any.whl", "has_sig": false, "md5_digest": "70e6d01de3710bfb8883a95d83dfe1bf", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": ">=3.6.0", "size": 30926, "upload_time": "2019-10-08T14:23:42", "url": "https://files.pythonhosted.org/packages/5c/b3/0d23579993ca812cb76ffac364e62ecf28192910b848bc1045263f38791f/kubeflow_kale-0.3.4-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "ae72a8558992eeb1a9886d6805825ea0", "sha256": "bffc6681f2184827dbca9542463fe6ffd29b39d242c2829f7913658f6b4ed506" }, "downloads": -1, "filename": "kubeflow-kale-0.3.4.tar.gz", "has_sig": false, "md5_digest": "ae72a8558992eeb1a9886d6805825ea0", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.6.0", "size": 239442, "upload_time": "2019-10-08T14:23:44", "url": "https://files.pythonhosted.org/packages/a1/0b/911246e44e7940ae700f7bc54781e6926e3954e1c23657f7580b7b042336/kubeflow-kale-0.3.4.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "70e6d01de3710bfb8883a95d83dfe1bf", "sha256": "122f5fe9acf623895a0be2f8c3de3158d0d46a592e69b7b3c7766f311e08062d" }, "downloads": -1, "filename": "kubeflow_kale-0.3.4-py3-none-any.whl", "has_sig": false, "md5_digest": "70e6d01de3710bfb8883a95d83dfe1bf", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": ">=3.6.0", "size": 30926, "upload_time": "2019-10-08T14:23:42", "url": "https://files.pythonhosted.org/packages/5c/b3/0d23579993ca812cb76ffac364e62ecf28192910b848bc1045263f38791f/kubeflow_kale-0.3.4-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "ae72a8558992eeb1a9886d6805825ea0", "sha256": "bffc6681f2184827dbca9542463fe6ffd29b39d242c2829f7913658f6b4ed506" }, "downloads": -1, "filename": "kubeflow-kale-0.3.4.tar.gz", "has_sig": false, "md5_digest": "ae72a8558992eeb1a9886d6805825ea0", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.6.0", "size": 239442, "upload_time": "2019-10-08T14:23:44", "url": "https://files.pythonhosted.org/packages/a1/0b/911246e44e7940ae700f7bc54781e6926e3954e1c23657f7580b7b042336/kubeflow-kale-0.3.4.tar.gz" } ] }