{
    "info": {
        "author": "",
        "author_email": "",
        "bugtrack_url": null,
        "classifiers": [],
        "description": "# Airflow's DataDriver plugin\n\n## from Pandas' dataframes to Airflow pipelines\n\n#### WHY : \n\nIn a machine learning project, there is a recurring problem \nwith the difference between local interactive modeling source code \nand production pipelines source code. \nIt is very error prone and, as a consequence, time consuming because we \nswitch constantly between experimentation and production.\n\nThe Datadriver project aims to solve this issue by making the glue code **based on Pandas and sklearn**\nfor modelization, **and on Airflow** for automation, scheduling, and monitoring of training \nand predicting pipelines.\n\n#### Plugin description\n\n**Datadriver UI (ddui)** is the Airflow's plugin we developed to track our models. \nCombined with the Datadriver's API (pyddapi), it offers a DAG view to track machine learning workflow (or dataflow).\n\nMore specifically, it shows the **Output** of any Airflow's Task with a lot of metrics and\ncharts : \n\n - choose a DAG to track\n![img/ddui_titan1.png](img/ddui_titan1.png)\n - select a task to see charts and describe metrics on the output_table\n![img/ddui_titan3.png](img/ddui_titan3.png) \n - look at histograms to verify if columns are correct (distributions, number of NAs,\n  unique values, etc...)\n![img/ddui_titan2.png](img/ddui_titan2.png) \n\n## Getting started\n\n    git clone git_url_of_this_project && cd this_project\n\nlocal install :\n\n    pip install -e .\n    ddui install\n\ndocker install :\n\n    ./run_docker.sh\n\n\n## Package modules\n\n    ddui/\n        dash_app -> the application defined like a Dash application, with callbacks and event handeling. It is imported in plugin.py later\n        dash_components -> html custom components like a Panel or an Alert Div\n        orm -> function to access the Airflow metastore and retrieve DAGs list and infos\n        plot -> functions using plotly, they return a Graph object\n        plugin -> defines the DataDriverUI plugin that implements Airflow's Plugin interface https://airflow.apache.org/plugins.html#interface\n        views -> a FlaskAdminView that implements Dash too, to have the ability to include plotly charts in Airflow\n\n\n###### dependencies graph\n\n![pydeps ddui](img/dependencies_analysis.png)       \n\n## Developer setup\n\nThere is an existing DAG in tests/dags that mocks the behavior of Datadriver's API, but\nwithout any dependency to pyddapi.\n\nYou can use it to develop the User Interface, using the script located in tests/dev_tools.\n\n    cd tests/dev_tools\n    python run_webserver.py\n\nIt runs the Airflow's webserver, and it overrides the AIRFLOW__CORE__DAGS_FOLDER to look into tests/dags.\n\n### Setup your virtual env\n\n    virtualenv venv\n    source venv/bin/activate\n    pip install -e .\n    pip install -r ci/tests_requirements.txt\n    ddui install\n\n",
        "description_content_type": "",
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "",
        "keywords": "",
        "license": "",
        "maintainer": "",
        "maintainer_email": "",
        "name": "ddui",
        "package_url": "https://pypi.org/project/ddui/",
        "platform": "",
        "project_url": "https://pypi.org/project/ddui/",
        "project_urls": null,
        "release_url": "https://pypi.org/project/ddui/3.0.4/",
        "requires_dist": [
            "pandas",
            "jinja2 (<2.9.0,>=2.7.3)",
            "flask (<0.13,>=0.12.4)",
            "apache-airflow (==1.10.*)",
            "click",
            "dash (==0.37.*)",
            "dash-html-components (==0.13.*)",
            "dash-core-components (==0.43.*)",
            "dash-dagre (>=0.0.2)",
            "dash-table (==3.4.*)"
        ],
        "requires_python": "",
        "summary": "",
        "version": "3.0.4"
    },
    "last_serial": 4951213,
    "releases": {
        "3.0.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "bee069ab85fb1770ef032d4f78e33e26",
                    "sha256": "4d0ffb76a8afd580300c1b78975091c6c71787ab0fd1f813a68e03cb0e6b067c"
                },
                "downloads": -1,
                "filename": "ddui-3.0.0-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "bee069ab85fb1770ef032d4f78e33e26",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 13804,
                "upload_time": "2019-02-08T16:17:42",
                "url": "https://files.pythonhosted.org/packages/ef/69/80fb8c8c9a53176aa4ec83a0bdc2120fa62e080d25095b4834074c5bec5c/ddui-3.0.0-py2.py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "3ab394c3981624d1103761f821e2b123",
                    "sha256": "70f4f39593736a16e3efe935a8ec639ffc9e6348c5a48324ee96710c07552eba"
                },
                "downloads": -1,
                "filename": "ddui-3.0.0.tar.gz",
                "has_sig": false,
                "md5_digest": "3ab394c3981624d1103761f821e2b123",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 11921,
                "upload_time": "2019-02-08T16:17:43",
                "url": "https://files.pythonhosted.org/packages/ad/b3/664a43057b78b1c71faa7f9c795b4e47608d8c9e0ed992f94748358e64ca/ddui-3.0.0.tar.gz"
            }
        ],
        "3.0.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "17b1a30a9d482175fb5e96ff0c80838e",
                    "sha256": "790dbb41d93310d11fb0ecd91354b0aa8f17ee78b10f3db03f6464c458dd735f"
                },
                "downloads": -1,
                "filename": "ddui-3.0.1-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "17b1a30a9d482175fb5e96ff0c80838e",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 11640,
                "upload_time": "2019-02-15T10:35:42",
                "url": "https://files.pythonhosted.org/packages/2c/b3/093ddf7eb6a93ec5d33cf2bf50d345ceefbf82ffe29ad6777c4b253183ac/ddui-3.0.1-py2.py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "511e51419a2539e635a3bce2e51a8513",
                    "sha256": "6d36ae33762ab5ea95da9c6d8afc218de2fc3300d5de7626e326e687f1ab5296"
                },
                "downloads": -1,
                "filename": "ddui-3.0.1.tar.gz",
                "has_sig": false,
                "md5_digest": "511e51419a2539e635a3bce2e51a8513",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 10653,
                "upload_time": "2019-02-15T10:35:44",
                "url": "https://files.pythonhosted.org/packages/d1/3d/ab1c7cd253f4c5146abce3818e836cc1a79c7cd5b53300bd09417051e9ae/ddui-3.0.1.tar.gz"
            }
        ],
        "3.0.2": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "a4aafc79416bed44d38f09fcf9ccbc4b",
                    "sha256": "26bcf04835a4344efa0d4f23b291553debd3ab8c7395ca3d466c264346e30f39"
                },
                "downloads": -1,
                "filename": "ddui-3.0.2-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "a4aafc79416bed44d38f09fcf9ccbc4b",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 15781,
                "upload_time": "2019-02-19T16:03:55",
                "url": "https://files.pythonhosted.org/packages/c6/7e/c33dd7a0e6c7b1ff73a30223d2c43edb9019faf3054686104651d15b4921/ddui-3.0.2-py2.py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "7275ef175510ada073d999c8f715f6b9",
                    "sha256": "371ccbd86d2d385667d0e521ff824734d5004128185b873e65afb1cf2f360c6c"
                },
                "downloads": -1,
                "filename": "ddui-3.0.2.tar.gz",
                "has_sig": false,
                "md5_digest": "7275ef175510ada073d999c8f715f6b9",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 14610,
                "upload_time": "2019-02-19T16:03:59",
                "url": "https://files.pythonhosted.org/packages/be/63/b8f57e66250c4a805d30679b10dd66a05ba1dc37729b0c1f6e69a62b418f/ddui-3.0.2.tar.gz"
            }
        ],
        "3.0.3": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "25a4ee7bdbd10070feae9d9853007f10",
                    "sha256": "94811172068fa423f951cc003c578647cf486138a1acd97ad70fcf0e5d7798b0"
                },
                "downloads": -1,
                "filename": "ddui-3.0.3-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "25a4ee7bdbd10070feae9d9853007f10",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 15869,
                "upload_time": "2019-03-06T21:54:20",
                "url": "https://files.pythonhosted.org/packages/c7/45/7428b37f5aa681059c5f620721d75ad652e121a5e033b519b6be952c0070/ddui-3.0.3-py2.py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "210a9f5bc3b020e0999436fdf3709638",
                    "sha256": "75704d544e8f8409582a108a0d0a9bae548964c59a2f6fdd3b2493fbb4c3ace0"
                },
                "downloads": -1,
                "filename": "ddui-3.0.3.tar.gz",
                "has_sig": false,
                "md5_digest": "210a9f5bc3b020e0999436fdf3709638",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 14697,
                "upload_time": "2019-03-06T21:54:22",
                "url": "https://files.pythonhosted.org/packages/f4/a9/d2a0721c8952a0cf90a09085bfc07342f9343f7f2d751897c02962e68338/ddui-3.0.3.tar.gz"
            }
        ],
        "3.0.4": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "75e33edc591eaabbcdcb45ac7d63fd9c",
                    "sha256": "b8947a24358c21fc1b2c058a924061355fb0203a07aff38facf2b05d9a9c7688"
                },
                "downloads": -1,
                "filename": "ddui-3.0.4-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "75e33edc591eaabbcdcb45ac7d63fd9c",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 15886,
                "upload_time": "2019-03-17T19:38:07",
                "url": "https://files.pythonhosted.org/packages/5a/f5/aa5fe6ad9ebe91f5949ebad13a6bda793169ee4199248d68f5d9ebe69e84/ddui-3.0.4-py2.py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "40baa54d9501d4059949127acc7b2e71",
                    "sha256": "186a6a30c1f2497a3d52b5ee3435145fd48e0eccdecf2f345791c7831fac4633"
                },
                "downloads": -1,
                "filename": "ddui-3.0.4.tar.gz",
                "has_sig": false,
                "md5_digest": "40baa54d9501d4059949127acc7b2e71",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 14730,
                "upload_time": "2019-03-17T19:38:09",
                "url": "https://files.pythonhosted.org/packages/0d/d6/b9e73aa626afad788661d0ed10ffffc74bf2356951cf5b10e4ebdd997c22/ddui-3.0.4.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "75e33edc591eaabbcdcb45ac7d63fd9c",
                "sha256": "b8947a24358c21fc1b2c058a924061355fb0203a07aff38facf2b05d9a9c7688"
            },
            "downloads": -1,
            "filename": "ddui-3.0.4-py2.py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "75e33edc591eaabbcdcb45ac7d63fd9c",
            "packagetype": "bdist_wheel",
            "python_version": "py2.py3",
            "requires_python": null,
            "size": 15886,
            "upload_time": "2019-03-17T19:38:07",
            "url": "https://files.pythonhosted.org/packages/5a/f5/aa5fe6ad9ebe91f5949ebad13a6bda793169ee4199248d68f5d9ebe69e84/ddui-3.0.4-py2.py3-none-any.whl"
        },
        {
            "comment_text": "",
            "digests": {
                "md5": "40baa54d9501d4059949127acc7b2e71",
                "sha256": "186a6a30c1f2497a3d52b5ee3435145fd48e0eccdecf2f345791c7831fac4633"
            },
            "downloads": -1,
            "filename": "ddui-3.0.4.tar.gz",
            "has_sig": false,
            "md5_digest": "40baa54d9501d4059949127acc7b2e71",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 14730,
            "upload_time": "2019-03-17T19:38:09",
            "url": "https://files.pythonhosted.org/packages/0d/d6/b9e73aa626afad788661d0ed10ffffc74bf2356951cf5b10e4ebdd997c22/ddui-3.0.4.tar.gz"
        }
    ]
}