{
    "info": {
        "author": "Roman Kh at al",
        "author_email": "rhudor@gmail.com",
        "bugtrack_url": null,
        "classifiers": [
            "Development Status :: 4 - Beta",
            "Intended Audience :: Developers",
            "Intended Audience :: Science/Research",
            "License :: OSI Approved :: Apache Software License",
            "Operating System :: OS Independent",
            "Programming Language :: Python",
            "Programming Language :: Python :: 3",
            "Programming Language :: Python :: 3.5",
            "Programming Language :: Python :: 3.6",
            "Programming Language :: Python :: 3.7",
            "Topic :: Scientific/Engineering"
        ],
        "description": "[![License](https://img.shields.io/github/license/analysiscenter/batchflow.svg)](https://www.apache.org/licenses/LICENSE-2.0)\n[![Python](https://img.shields.io/badge/python-3.5-blue.svg)](https://python.org)\n[![TensorFlow](https://img.shields.io/badge/TensorFlow-1.12-orange.svg)](https://tensorflow.org)\n[![PyTorch](https://img.shields.io/badge/PyTorch-0.4-orange.svg)](https://pytorch.org)\n[![Run Status](https://api.shippable.com/projects/58c6ada92e042a0600297f61/badge?branch=master)](https://app.shippable.com/github/analysiscenter/batchflow)\n[![codecov](https://codecov.io/gh/analysiscenter/batchflow/branch/master/graph/badge.svg)](https://codecov.io/gh/analysiscenter/batchflow)\n\n# BatchFlow\n\n`BatchFlow` helps you conveniently work with random or sequential batches of your data\nand define data processing and machine learning workflows even for datasets that do not fit into memory.\n\nFor more details see [the documentation and tutorials](https://analysiscenter.github.io/batchflow/).\n\nMain features:\n- flexible batch generaton\n- deterministic and stochastic pipelines\n- datasets and pipelines joins and merges\n- data processing actions\n- flexible model configuration\n- within batch parallelism\n- batch prefetching\n- ready to use ML models and proven NN architectures\n- convenient layers and helper functions to build custom models\n- a powerful research engine with parallel model training and extended experiment logging.\n\n## Basic usage\n\n```python\nmy_workflow = my_dataset.pipeline()\n              .load('/some/path')\n              .do_something()\n              .do_something_else()\n              .some_additional_action()\n              .save('/to/other/path')\n```\nThe trick here is that all the processing actions are lazy. They are not executed until their results are needed, e.g. when you request a preprocessed batch:\n```python\nmy_workflow.run(BATCH_SIZE, shuffle=True, n_epochs=5)\n```\nor\n```python\nfor batch in my_workflow.gen_batch(BATCH_SIZE, shuffle=True, n_epochs=5):\n    # only now the actions are fired and data is being changed with the workflow defined earlier\n    # actions are executed one by one and here you get a fully processed batch\n```\nor\n```python\nNUM_ITERS = 1000\nfor i in range(NUM_ITERS):\n    processed_batch = my_workflow.next_batch(BATCH_SIZE, shuffle=True, n_epochs=None)\n    # only now the actions are fired and data is changed with the workflow defined earlier\n    # actions are executed one by one and here you get a fully processed batch\n```\n\n\n## Train a neural network\n`BatchFlow` includes ready-to-use proven architectures like VGG, Inception, ResNet and many others.\nTo apply them to your data just choose a model, specify the inputs (like the number of classes or images shape)\nand call `train_model`. Of course, you can also choose a loss function, an optimizer and many other parameters, if you want.\n```python\nfrom batchflow.models.tf import ResNet34\n\nmy_workflow = my_dataset.pipeline()\n              .init_model('dynamic', ResNet34, config={\n                          'inputs/images/shape': B('image_shape'),\n                          'labels/classes': 10,\n                          'initial_block/inputs': 'images'})\n              .load('/some/path')\n              .some_transform()\n              .another_transform()\n              .train_model('ResNet34', images=B('images'), labels=B('labels'))\n              .run(BATCH_SIZE, shuffle=True)\n```\n\nFor more advanced cases and detailed API see [the documentation](https://analysiscenter.github.io/batchflow/).\n\n\n## Installation\n\n> `BatchFlow` module is in the beta stage. Your suggestions and improvements are very welcome.\n\n> `BatchFlow` supports python 3.5 or higher.\n\n### Stable python package\n\nWith modern [pipenv](https://docs.pipenv.org/)\n```\npipenv install batchflow\n```\n\nWith old-fashioned [pip](https://pip.pypa.io/en/stable/)\n```\npip3 install batchflow\n```\n\n### Development vesrion\n\nWith modern [pipenv](https://docs.pipenv.org/)\n```\npipenv install git+https://github.com/analysiscenter/batchflow.git#egg=batchflow\n```\n\nWith old-fashioned [pip](https://pip.pypa.io/en/stable/)\n```\npip3 install git+https://github.com/analysiscenter/batchflow.git\n```\n\nAfter that just import `batchflow`:\n```python\nimport batchflow as bf\n```\n\n### Git submodule\nIn many cases it might be more convenient to install `batchflow` as a submodule in your project repository than as a python package.\n```\ngit submodule add https://github.com/analysiscenter/batchflow.git\ngit submodule init\ngit submodule update\n```\n\nIf your python file is located in another directory, you might need to add a path to `batchflow`:\n```python\nimport sys\nsys.path.insert(0, \"/path/to/batchflow\")\nimport batchflow as bf\n```\n\nWhat is great about using a submodule that every commit in your project can be linked to its own commit of a submodule.\nThis is extremely convenient in a fast paced research environment.\n\nRelative import is also possible:\n```python\nfrom .batchflow import Dataset\n```\n\n\n## Citing BatchFlow\nPlease cite BatchFlow in your publications if it helps your research.\n\n[![DOI](https://zenodo.org/badge/DOI/10.5281/zenodo.1041203.svg)](https://doi.org/10.5281/zenodo.1041203)\n\n```\nRoman Khudorozhkov et al. BatchFlow library for fast ML workflows. 2017. doi:10.5281/zenodo.1041203\n```\n\n```\n@misc{roman_kh_2017_1041203,\n  author       = {Khudorozhkov, Roman and others},\n  title        = {BatchFlow library for fast ML workflows},\n  year         = 2017,\n  doi          = {10.5281/zenodo.1041203},\n  url          = {https://doi.org/10.5281/zenodo.1041203}\n}\n```\n\n\n",
        "description_content_type": "text/markdown",
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://github.com/analysiscenter/batchflow",
        "keywords": "",
        "license": "Apache License 2.0",
        "maintainer": "",
        "maintainer_email": "",
        "name": "batchflow",
        "package_url": "https://pypi.org/project/batchflow/",
        "platform": "any",
        "project_url": "https://pypi.org/project/batchflow/",
        "project_urls": {
            "Homepage": "https://github.com/analysiscenter/batchflow"
        },
        "release_url": "https://pypi.org/project/batchflow/0.3.0/",
        "requires_dist": [
            "numpy (>=1.10)",
            "dill (>=0.2.7)",
            "tqdm (>=4.19.7)",
            "scipy (>=0.19.1)",
            "scikit-image (>=0.13.1)"
        ],
        "requires_python": "",
        "summary": "A framework for fast data processing and ML models training",
        "version": "0.3.0"
    },
    "last_serial": 5720510,
    "releases": {
        "0.3.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "947021f6b4244b12050240707323b106",
                    "sha256": "2aa19b45e1274667268fe3db81377430789696379f25da1ba72696092bb908c3"
                },
                "downloads": -1,
                "filename": "batchflow-0.3.0-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "947021f6b4244b12050240707323b106",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 257469,
                "upload_time": "2019-08-23T12:21:50",
                "url": "https://files.pythonhosted.org/packages/2f/57/f58c35525ad303dc793aad1adca5f98086530ebcbbcc9dcc96285a9209dc/batchflow-0.3.0-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "0e9588c48ddc9e27121b5d714dfd9d74",
                    "sha256": "e21aa210ed0d7b59fde3ae6799a3057bfb3d74e50c78710b22bca44d58774454"
                },
                "downloads": -1,
                "filename": "batchflow-0.3.0.tar.gz",
                "has_sig": false,
                "md5_digest": "0e9588c48ddc9e27121b5d714dfd9d74",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 1429825,
                "upload_time": "2019-08-23T12:21:55",
                "url": "https://files.pythonhosted.org/packages/02/73/ae65a4b952f84195043ab481ae589149382ff93f89c26da1572b1dac1933/batchflow-0.3.0.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "947021f6b4244b12050240707323b106",
                "sha256": "2aa19b45e1274667268fe3db81377430789696379f25da1ba72696092bb908c3"
            },
            "downloads": -1,
            "filename": "batchflow-0.3.0-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "947021f6b4244b12050240707323b106",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": null,
            "size": 257469,
            "upload_time": "2019-08-23T12:21:50",
            "url": "https://files.pythonhosted.org/packages/2f/57/f58c35525ad303dc793aad1adca5f98086530ebcbbcc9dcc96285a9209dc/batchflow-0.3.0-py3-none-any.whl"
        },
        {
            "comment_text": "",
            "digests": {
                "md5": "0e9588c48ddc9e27121b5d714dfd9d74",
                "sha256": "e21aa210ed0d7b59fde3ae6799a3057bfb3d74e50c78710b22bca44d58774454"
            },
            "downloads": -1,
            "filename": "batchflow-0.3.0.tar.gz",
            "has_sig": false,
            "md5_digest": "0e9588c48ddc9e27121b5d714dfd9d74",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 1429825,
            "upload_time": "2019-08-23T12:21:55",
            "url": "https://files.pythonhosted.org/packages/02/73/ae65a4b952f84195043ab481ae589149382ff93f89c26da1572b1dac1933/batchflow-0.3.0.tar.gz"
        }
    ]
}