{
    "info": {
        "author": "Arthur Brazinskas",
        "author_email": "bulletdll@gmail.com",
        "bugtrack_url": null,
        "classifiers": [],
        "description": "# Machine Learning Data Pipeline (MLDP) #\n\nThis repository contains a module for **parallel**, **real-time data processing** for machine learning purposes.\nEssentially, it provides an infrastructure for efficient parallel data processing for models **development (training, testing)**, and **production**. A data pipeline can be understood as a sequence of computational steps that are applied to data-chunks that are read progressively (opposed to reading all data to the main memory at once), as shown below. The final output of the data pipeline are data-chunks (batches) in a desired format, which will depend on a model at hand.\n\n<img src=\"img/mldp.png\"/>\n\nAs opposed to a common meaning of data pipeline, the module focuses on providing data-chunks in real-time, as fast as possible, assuming that a data processing procedure might change over the course of experiments series.\n\nBecause similar computational steps can in principle be applied to different ML projects, the main problem is that it's not straightforward to apply them to data in different format.\nFortunately, the pipeline passes a light-weight intermediate data format across steps. Therefore, **computational steps can be reused** as their input will always be of the same format regardless of whether your data is in csv, xml, or some other format.\n\nFinally, a need for experiments reproducibility is considered by **documentation generation**. Such that it would be possible to rerun an older experiment and obtain the same results, or simply lookup what setup has been used in a previous model run. To facilitate that, the module allows documentation generation of pipeline setups and individual components that were used to process data.\n\n## MLDP main features ##\n\n* Highly scalable data processing on multi-core CPU architectures.\n* Automatic documentation generation of data pipelines and individual steps.\n* Code reusability of computational steps (agnostic to an input data format).\n* Ease of custom steps integration to the module.\n* Complete independence of other machine learning libraries, such as TensorFlow.\n* A set of implemented steps that can be used out-of-the-box for data processing, such as a csv reader, vocabulary mapper, padder, and token processor.\n\nThe module does not provide an exhaustive set of computational steps, and a user has a complete freedom to write own steps in a pure Python, and use any extra libraries that he might wish.\n\n## Installation ##\n\n```python\npip install machine-learning-data-pipeline\n```\n\nAlternatively, clone the repository, and from its root directory run:\n```python\npip install .\n```\n\nIt was fully tested on Python 2.7.13\n\n## Usage ##\n\n### Example ###\n```python\nfrom mldp.pipeline import Pipeline\nfrom mldp.steps.readers import CsvReader\nfrom mldp.steps.transformers import TokenProcessor, FieldsSelector, Padder\n\ndata_path = \"mldp/tests/data/tweets.csv\"\n\n# creating steps\ncsv_reader = CsvReader(sep='\\t', chunk_size=30)\nfields_selector = FieldsSelector(field_names=[\"tweets\", \"labels\"])\ntoken_processor = TokenProcessor(field_names=\"tweets\",\n                                 tokenization_func=lambda x: x.split(),\n                                 lower_case=True)\npadder = Padder(field_names=\"tweets\", pad_symbol=\"<PAD>\")\n\n# creating the pipeline\npipeline = Pipeline(reader=csv_reader, worker_processes_num=1)\npipeline.add_step(fields_selector)\npipeline.add_step(token_processor)\npipeline.add_step(padder)\n\n# iterate over data chunks\nfor data_chunk in pipeline.iter(data_path=data_path):\n    pass\n\n# generate documentation and print it\nprint(pipeline)\n```\n\n### Implementation of custom steps ###\n\nIn order to implement custom steps, the user has to extend base classes of a desired step. For example, to write a custom reader, the user needs to extend **BaseReader** from **mldp/steps/readers/base_reader.py**.\n\nPlease remember that all steps are required to output data-chunks in the **intermediate format** as explained below.\n\n### Tutorials ###\n\nFor a more detailed presentation of the module's features and applications, please refer to the <a href=\"/tutorials\">tutorials folder</a>.\n\n## Main principles ##\n\n### Computational step types ###\n\n1. **Readers** - fetch data from a remote or local storage and convert it to the intermediate data-chunk format (dict of numpy arrays). E.g., CSV files or JSON readers.\n2. **Transformers** - transform field values of data-chunks or create new data-chunk fields. E.g. a Padder that makes all sequences to be of the same length.\n3. **Formatters** - convert the intermediate data-chunk format to a project specific data format. E.g. steps that convert to pandas data-frames or tuples (features, labels).\n4. **General**  - a special case steps, at the moment only the data chunk size adjuster that allows to change the size of data-chunks in the middle of a pipeline.\n5. **Preprocessing** - while preprocessing is not the main focus of the module, steps of this kind allow to perform data alteration and caching for multiple subsequent re-use during processing steps execution. For example, a step of this kind could pre-load remote data, or perform some data cleaning. Operations in this step are usually necessary to execute only once, then save the result, and use it for processing runs without re-execution of the step.\n\n\n### Intermediate format ###\n\nThe MLDP restricts the format of data-chunks that travel along the pipeline to be **dictionaries of numpy arrays** of arbitrary types and shapes, with the following rational:\n\n1. Potential to apply fast vectorized operations on numpy arrays.\n2. Code re-usability of steps. Namely, if a step is implemented once, it can be re-used in different projects regardless of input data format (under mild assumptions).\n\nData-chunks have **field_names** (dict keys) corresponding to data attributes (e.g. \"text_ids\", \"texts\", \"labels\"). And **field_values** (dict values) corresponding to data-unit values, for example the actual *text ids* or *text* strings. While *field_values* are restricted to be numpy arrays, their actual elements can be of arbitrary types, and thus any data can be frames easily into the format.\n\nAn example data-chunk for machine translation is shown below.\n\n<img src='img/interm_format.png' width='70%'/>\n\nThe shaded rectangles correspond to numpy arrays of different types but with the same first dimension (i.e the same length). Note that numpy arrays can contain arbitrary types besides standard floats and ints.\n\n## License ##\n\nAttribution-NonCommercial 4.0 International\n\n\n",
        "description_content_type": "",
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://github.com/ixlan/machine-learning-data-pipeline",
        "keywords": "",
        "license": "",
        "maintainer": "",
        "maintainer_email": "",
        "name": "machine-learning-data-pipeline",
        "package_url": "https://pypi.org/project/machine-learning-data-pipeline/",
        "platform": "",
        "project_url": "https://pypi.org/project/machine-learning-data-pipeline/",
        "project_urls": {
            "Homepage": "https://github.com/ixlan/machine-learning-data-pipeline"
        },
        "release_url": "https://pypi.org/project/machine-learning-data-pipeline/1.0.3/",
        "requires_dist": [
            "numpy (==1.15.2)",
            "pandas (==0.21.1)",
            "plumbum (==1.6.7)",
            "s3fs (==0.1.6)",
            "urllib3 (==1.23)",
            "mock (==2.0.0)"
        ],
        "requires_python": "",
        "summary": "Pipeline module for parallel real-time data processing for machine learning models development and production purposes.",
        "version": "1.0.3"
    },
    "last_serial": 4400136,
    "releases": {
        "1.0.2": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "fd3893f3bb1febf43674c13213c93877",
                    "sha256": "11e2fda606c13b8c8661d1f54f7489f864f7f226a8ca74bab94759e45555ab9c"
                },
                "downloads": -1,
                "filename": "machine_learning_data_pipeline-1.0.2-py2-none-any.whl",
                "has_sig": false,
                "md5_digest": "fd3893f3bb1febf43674c13213c93877",
                "packagetype": "bdist_wheel",
                "python_version": "py2",
                "requires_python": null,
                "size": 55892,
                "upload_time": "2018-10-21T18:44:15",
                "url": "https://files.pythonhosted.org/packages/b1/64/b57bcbde4533da1216ac83c67412afbc04f25416ee62341eb1b131c1edf2/machine_learning_data_pipeline-1.0.2-py2-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "e1c7dc07cb34347f380873c96f15921f",
                    "sha256": "06fb7274ef5ddc4d3450a2b00af4ea203d29a2ef68d8d535540f5c9edec79793"
                },
                "downloads": -1,
                "filename": "machine_learning_data_pipeline-1.0.2.tar.gz",
                "has_sig": false,
                "md5_digest": "e1c7dc07cb34347f380873c96f15921f",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 38986,
                "upload_time": "2018-10-21T18:44:17",
                "url": "https://files.pythonhosted.org/packages/d1/84/0f2e8d955556f49447ed6d2118c5895859c07374ffb7e7a4714cafa1a1d4/machine_learning_data_pipeline-1.0.2.tar.gz"
            }
        ],
        "1.0.3": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "8ab78c58115de55628f5f0fee4df4e1f",
                    "sha256": "2995dbaffaedc4c7c443dde3efb06328e6893877efa19dece27ff1fd5be4211e"
                },
                "downloads": -1,
                "filename": "machine_learning_data_pipeline-1.0.3-py2-none-any.whl",
                "has_sig": false,
                "md5_digest": "8ab78c58115de55628f5f0fee4df4e1f",
                "packagetype": "bdist_wheel",
                "python_version": "py2",
                "requires_python": null,
                "size": 55961,
                "upload_time": "2018-10-21T18:49:56",
                "url": "https://files.pythonhosted.org/packages/6a/e2/2ca088302b928a9c99b18978562a3bc96b2409e959d29d82d2341b080c2c/machine_learning_data_pipeline-1.0.3-py2-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "fe189efed4df65e8ed5cd30390f90544",
                    "sha256": "003c0e3ee90005df94e0a7066035c3378f2fe08b773e9ecdfd116fddcccb4b9c"
                },
                "downloads": -1,
                "filename": "machine_learning_data_pipeline-1.0.3.tar.gz",
                "has_sig": false,
                "md5_digest": "fe189efed4df65e8ed5cd30390f90544",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 39175,
                "upload_time": "2018-10-21T18:49:58",
                "url": "https://files.pythonhosted.org/packages/ee/8a/39d60dae649150e571c6f1aca603bef21fe4ab8eb712a01eaa7253c3566a/machine_learning_data_pipeline-1.0.3.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "8ab78c58115de55628f5f0fee4df4e1f",
                "sha256": "2995dbaffaedc4c7c443dde3efb06328e6893877efa19dece27ff1fd5be4211e"
            },
            "downloads": -1,
            "filename": "machine_learning_data_pipeline-1.0.3-py2-none-any.whl",
            "has_sig": false,
            "md5_digest": "8ab78c58115de55628f5f0fee4df4e1f",
            "packagetype": "bdist_wheel",
            "python_version": "py2",
            "requires_python": null,
            "size": 55961,
            "upload_time": "2018-10-21T18:49:56",
            "url": "https://files.pythonhosted.org/packages/6a/e2/2ca088302b928a9c99b18978562a3bc96b2409e959d29d82d2341b080c2c/machine_learning_data_pipeline-1.0.3-py2-none-any.whl"
        },
        {
            "comment_text": "",
            "digests": {
                "md5": "fe189efed4df65e8ed5cd30390f90544",
                "sha256": "003c0e3ee90005df94e0a7066035c3378f2fe08b773e9ecdfd116fddcccb4b9c"
            },
            "downloads": -1,
            "filename": "machine_learning_data_pipeline-1.0.3.tar.gz",
            "has_sig": false,
            "md5_digest": "fe189efed4df65e8ed5cd30390f90544",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 39175,
            "upload_time": "2018-10-21T18:49:58",
            "url": "https://files.pythonhosted.org/packages/ee/8a/39d60dae649150e571c6f1aca603bef21fe4ab8eb712a01eaa7253c3566a/machine_learning_data_pipeline-1.0.3.tar.gz"
        }
    ]
}