{
    "info": {
        "author": "ConfusionFlow Contributors",
        "author_email": "gfrogat@gmail.com",
        "bugtrack_url": null,
        "classifiers": [],
        "description": "![ConfusionFlow Banner](https://github.com/confusionflow/confusionflow/blob/master/docs/source/_static/img/confusionflow-banner.png)\n\nConfusionFlow is a visualization tool distributed as Python package that enables more nuanced monitoring of a neural network's training process.\n\n-   track and visualize the model performance of different timepoints (i.e., epochs) and dataset folds and compare different runs (e.g., different hyperparameter configurations).\n-   we provide wrappers for exporting performance logs in the required format.\n\nConfusionFlow is in an early-preview alpha. Expect some bugs and rough edges.\n\n![ConfusionFlow Teaser](https://github.com/confusionflow/confusionflow/blob/master/docs/source/_static/img/confusionflow-teaser.png)\n\n## Demo\n\nTry the latest stable release of ConfusionFlow with several [example logs](http://data.caleydo.org/app_data/confusionflow-example-data.zip) for the CIFAR-10 dataset: http://confusionflow.caleydoapp.org\n\n## Additional Information\n\nConfusionFlow was developed as a visualization tool to provide users with more feedback while developing or tuning neural network based classifiers. Performance monitoring often only utilizes simple line charts (one would for example plot the model loss and accuracy) which might easily miss many details and changes in the model's error structure.\n\nWhile the errors for a specific model state (e.g., at a certain epoch) can be represented as a confusion matrix, it is difficult to compare multiple confusion matrices or track changes over time.\nConfusionFlow visualizes model confusion over multiple model states and lets users compare different folds (e.g., train vs test set) or different hyperparameter configurations.\n\nTo the best of our knowledge there are still no other tools with similar functionality.\n\n### Limitations\n\n#### Number of classes\n\nDue to screenspace limitations the system is currently limited to around 10 classes. We are aware that we will not be able to handle datasets at _ImageNet_ scale BUT those datesets are not very common, as they are usually very expensive to obtain. A large percentage of classification problems range around 10 or less classes where ConfusionFlow can provide additional feedback.\n\n#### Runtime overhead\n\nLogging the performance for multiple folds every epoch might severely slowdown the time to convergence (especially when creating logs for the complete train and test sets). While logging on a minibatch level might provide additional information, it also severely slows down the training even further. It is possible to alleviate this problem by persisting model checkpoints and creating the performance logs on different machines. However, this also involves a lot of engineering effort and will not be supported by ConfusionFlow in the near future.\n\n## Installation\n\nConfusionflow can be either downloaded directly from PyPI via `pip install confusionflow` or by cloning and installing the repository directly from source:\n\n### From Source\n\nFirst make sure you have [node](https://nodejs.org/en/) installed (required for building the UI component).\n\nClone and install the repository:\n\n```\ngit clone https://github.com/confusionflow/confusionflow\ncd confusionflow\npython setup.py install\n```\n\n#### Development\n\nIf you want develop ConfusionFlow locally run:\n\n```\npython setup.py build_ui develop\n```\n\n## Getting Started\n\nAs a first step you need to create some logs before you can start analyzing. Have a look at the `examples` folder and run one of the examples.\nThe each example will create a new subdirectory `logs` where the performance logs will be stored.\n\nYou then can start the ConfusionFlow UI via:\n\n```\nconfusionflow --logdir `<path_to_logdir>`\n```\n\n### Usage Examples\n\n#### tf.keras\n\n-   [fashion-mnist](./examples/tf.keras/fashion-mnist)\n\n#### torch\n\n-   [mnist](./examples/torch/mnist)\n\nIf you are using your own datasets you have to create a [dataset-configuration](https://docs.confusionflow.org/notes/dataset-config) first. We provide some example configurations for some popular datasets in `examples/dataset-templates` which should help you getting started.\n\n### Docker\n\nYou can also run ConfusionFlow via Docker. For more information please see the [documentation](https://docs.confusionflow.org/notes/how-to-commandline.html#docker).\n\n## Logging\n\nWe provide simple wrappers for [Tensorflow Keras API](https://www.tensorflow.org/guide/keras) as well as [Pytorch](https://pytorch.org) for logging confusion matrices and exporting them in the format required by ConfusionFlow. For more information please consult the [logging documentation](https://docs.confusionflow.org/notes/how-to-logging.html).\n\n### Log Directory Layout\n\n```\n<logdir>\n\u251c\u2500\u2500 datasets                    <--- dataset config files\n\u2502   \u251c\u2500\u2500 mnist.json\n\u2502   \u2514\u2500\u2500 index.json\n\u251c\u2500\u2500 foldlogdata                 <--- foldlog data\n\u2502   \u251c\u2500\u2500 example_log_mnist_train_data.json\n\u2502   \u2514\u2500\u2500 example_log_mnist_test_data.json\n\u251c\u2500\u2500 foldlog                     <--- foldlog specifications\n\u2502   \u251c\u2500\u2500 example_log_mnist_train.json\n\u2502   \u2514\u2500\u2500 example_log_mnist_test.json\n\u251c\u2500\u2500 runs                        <--- run specifications\n\u2502   \u251c\u2500\u2500 example_log.json\n\u2502   \u2514\u2500\u2500 index.json\n\u2514\u2500\u2500 views                       <--- view specifications (currently unsused)\n```\n\n## Documentation\n\nThe latest documentation can be viewed on [docs.confusionflow.org](https://docs.confusionflow.org)\n\n### API\n\nThe Python package includes a simple Flask based server that implements the current API.\nThe current API definition can be found in `tools/swagger/api.yml` and can be viewed using the [Swagger Editor](https://editor.swagger.io/).\n\n## ConfusionFlow UI\n\nThe ConfusionFlow UI is currently developed at [Caleydo/confusionflow-ui](https://github.com/Caleydo/confusionflow-ui).\n\n## The Team\n\nConfusionFlow is a research project of the [Institute of Computer Graphics](https://www.jku.at/cg) at [Johannes Kepler University Linz](https://www.jku.at/) in collaboration with the [IBM Visual AI Lab](https://researcher.watson.ibm.com/researcher/view_group.php?id=5948).\nConfusionFlow is currently maintained by [Peter Ruch](https://github.com/gfrogat) and [Holger Stitz](https://github.com/thinkh).\n\n## Feedback\n\nWe would be really grateful for any [feedback](https://github.com/confusionflow/confusionflow/issues/new?template=feedback.md) via the repository's issues section.\n\n## Getting Involved\n\n-   You can ask questions on our mailing list [confusionflow-dev@googlegroups.com](https://groups.google.com/forum/#!forum/confusionflow-dev).\n-   Please report bugs by submitting a [GitHub issue](https://github.com/ConfusionFlow/confusionflow/issues/new?template=bug_report.md).\n\n\n",
        "description_content_type": "text/markdown",
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://github.com/confusionflow/confusionflow",
        "keywords": "",
        "license": "BSD-3-Clause",
        "maintainer": "",
        "maintainer_email": "",
        "name": "confusionflow",
        "package_url": "https://pypi.org/project/confusionflow/",
        "platform": "",
        "project_url": "https://pypi.org/project/confusionflow/",
        "project_urls": {
            "Documentation": "https://docs.confusionflow.org",
            "Homepage": "https://github.com/confusionflow/confusionflow",
            "Source": "https://github.com/confusionflow/confusionflow/",
            "Tracker": "https://github.com/confusionflow/confusionflow/issues"
        },
        "release_url": "https://pypi.org/project/confusionflow/0.1.3/",
        "requires_dist": [
            "flask",
            "gevent",
            "pyyaml"
        ],
        "requires_python": "",
        "summary": "ConfusionFlow is a visualization tool that enables more nuanced monitoring of a neural network's training process.",
        "version": "0.1.3"
    },
    "last_serial": 4567928,
    "releases": {
        "0.1.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "a91618678e17e2bf6b80a9526a2119ad",
                    "sha256": "1a466e8738d44719f1111acba42993b79fedda9586668a62a2be6a0307699610"
                },
                "downloads": -1,
                "filename": "confusionflow-0.1.0-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "a91618678e17e2bf6b80a9526a2119ad",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 1064546,
                "upload_time": "2018-10-01T16:07:16",
                "url": "https://files.pythonhosted.org/packages/84/6c/cf6f86b3e21d8042b641bcc6a5cc47857e76f1becfd455ce08225f51fc6a/confusionflow-0.1.0-py2.py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "ee294fa486848221c08c3a16d97aac1c",
                    "sha256": "f73bff8f166419e819beb75fe2841350764f95871d80921a905aa4adc4b3a79d"
                },
                "downloads": -1,
                "filename": "confusionflow-0.1.0.tar.gz",
                "has_sig": false,
                "md5_digest": "ee294fa486848221c08c3a16d97aac1c",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 1058766,
                "upload_time": "2018-10-01T16:07:18",
                "url": "https://files.pythonhosted.org/packages/07/aa/3f868091fd3f1f393495c6b98862446bed2c7c558bb434bdf92442ddf78f/confusionflow-0.1.0.tar.gz"
            }
        ],
        "0.1.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "3aed2ce7bc661c6897e60fff6bd2a614",
                    "sha256": "bcfd59cae362436ce5a6c3abcfc85d74496044a0f89710fa0eb05cc009e94fd7"
                },
                "downloads": -1,
                "filename": "confusionflow-0.1.1-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "3aed2ce7bc661c6897e60fff6bd2a614",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 1064559,
                "upload_time": "2018-10-02T08:45:05",
                "url": "https://files.pythonhosted.org/packages/e0/b9/22fb6106102f86e3190c66c22dfa46b2b2926e1329a5125759b9c31773ae/confusionflow-0.1.1-py2.py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "20141f8369d9ac68dfda80f4612e37ad",
                    "sha256": "3b9dffb7c507c1620e4f5f8b552e9a98861779ae667bc134e0b6c1405b927c04"
                },
                "downloads": -1,
                "filename": "confusionflow-0.1.1.tar.gz",
                "has_sig": false,
                "md5_digest": "20141f8369d9ac68dfda80f4612e37ad",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 1058828,
                "upload_time": "2018-10-02T08:45:06",
                "url": "https://files.pythonhosted.org/packages/fe/cd/1df024b2331c813d79779171834920b7c8d97f5fb95528bf06e2ec6ae7d3/confusionflow-0.1.1.tar.gz"
            }
        ],
        "0.1.2": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "73f7c70f2871c60929f71e858bf2099b",
                    "sha256": "a0938d6292c2a521eb25a74f249ddb63783645af0dd5fdb262c0190ce7c7483e"
                },
                "downloads": -1,
                "filename": "confusionflow-0.1.2-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "73f7c70f2871c60929f71e858bf2099b",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 1064664,
                "upload_time": "2018-10-10T07:39:02",
                "url": "https://files.pythonhosted.org/packages/b8/3f/814415c4922794f20911ef8375a395549c3d01cbbf252aa1a149970a54dd/confusionflow-0.1.2-py2.py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "c06257411465ad06f1fa2bba0ec20d7f",
                    "sha256": "d1a0f5e1895c61ba5dd95a51fb192074b95283489a8f35a114eef86001949c8f"
                },
                "downloads": -1,
                "filename": "confusionflow-0.1.2.tar.gz",
                "has_sig": false,
                "md5_digest": "c06257411465ad06f1fa2bba0ec20d7f",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 1057699,
                "upload_time": "2018-10-10T07:39:04",
                "url": "https://files.pythonhosted.org/packages/5a/4c/187d5a9ea62f4f5224637958e96454ab0e6f35e4b8d0e7b15dbfa1d0483a/confusionflow-0.1.2.tar.gz"
            }
        ],
        "0.1.3": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "89af1186701af761a93b5b678634ecdb",
                    "sha256": "fb5f80fe98a9cb724e9cfaa4a661c09a2e7e74d245114741fd09fd3605089e13"
                },
                "downloads": -1,
                "filename": "confusionflow-0.1.3-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "89af1186701af761a93b5b678634ecdb",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 2114191,
                "upload_time": "2018-12-06T13:26:00",
                "url": "https://files.pythonhosted.org/packages/33/e5/c9755d551274296d75a26294894de8e3002dfaeb086767a193db364d55b0/confusionflow-0.1.3-py2.py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "4754f78012f8ce3fd6fdcc066ff31510",
                    "sha256": "4572731dbb9b721e5034e32e1bacf0c9792ff17132a3c728b18cda834effe58c"
                },
                "downloads": -1,
                "filename": "confusionflow-0.1.3.tar.gz",
                "has_sig": false,
                "md5_digest": "4754f78012f8ce3fd6fdcc066ff31510",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 2108625,
                "upload_time": "2018-12-06T13:26:04",
                "url": "https://files.pythonhosted.org/packages/e4/0c/268a114769882e4ee61243d94a921a7efa5cd77275a8feb4aa97faa3db0a/confusionflow-0.1.3.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "89af1186701af761a93b5b678634ecdb",
                "sha256": "fb5f80fe98a9cb724e9cfaa4a661c09a2e7e74d245114741fd09fd3605089e13"
            },
            "downloads": -1,
            "filename": "confusionflow-0.1.3-py2.py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "89af1186701af761a93b5b678634ecdb",
            "packagetype": "bdist_wheel",
            "python_version": "py2.py3",
            "requires_python": null,
            "size": 2114191,
            "upload_time": "2018-12-06T13:26:00",
            "url": "https://files.pythonhosted.org/packages/33/e5/c9755d551274296d75a26294894de8e3002dfaeb086767a193db364d55b0/confusionflow-0.1.3-py2.py3-none-any.whl"
        },
        {
            "comment_text": "",
            "digests": {
                "md5": "4754f78012f8ce3fd6fdcc066ff31510",
                "sha256": "4572731dbb9b721e5034e32e1bacf0c9792ff17132a3c728b18cda834effe58c"
            },
            "downloads": -1,
            "filename": "confusionflow-0.1.3.tar.gz",
            "has_sig": false,
            "md5_digest": "4754f78012f8ce3fd6fdcc066ff31510",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 2108625,
            "upload_time": "2018-12-06T13:26:04",
            "url": "https://files.pythonhosted.org/packages/e4/0c/268a114769882e4ee61243d94a921a7efa5cd77275a8feb4aa97faa3db0a/confusionflow-0.1.3.tar.gz"
        }
    ]
}