{
    "info": {
        "author": "Yasufumi Taniguchi",
        "author_email": "yasufumi.taniguchi@gmail.com",
        "bugtrack_url": null,
        "classifiers": [
            "Programming Language :: Python :: 3.6",
            "Programming Language :: Python :: 3.7",
            "Programming Language :: Python :: 3.8"
        ],
        "description": "# Torchpipe: Simple ETL Pipeline for PyTorch\n\nTorchpipe is a simple ETL framework, especially for PyTorch.\nIt is an alternative to [tf.data](https://www.tensorflow.org/api_docs/python/tf/data) in TensorFlow\n\n\n# Requirements\n\n- Python 3.6+\n- PyTorch 1.2+\n\n\n# Installation\n\nTo install Torchpipe:\n\n```bash\npip install torchpipe\n```\n\n\n# Basic Usage\n\n```py\nimport torchpipe as tp\n\nd = tp.Dataset(range(1_000))\nd.shuffle(buffer_size=100).batch(batch_size=10).first() # [4, 44, 71, 92, 97, 86, 43, 57, 60, 62]\n```\n\n# Usage with PyTorch\n\n```py\nfrom torch.utils.data import DataLoader\nimport torchpipe as tp\n\n\nd = tp.Dataset(range(1_000)).shuffle(100).batch(10)\nloader = DataLoader(d, num_workers=4, collate_fn=lambda x: x)\nfor x in loader:\n    ...\n```\n\n# Usage with LineFlow\n\nYou can use Torchpipe with pre-defined datasets in [LineFlow](https://github.com/tofunlp/lineflow):\n\n```py\nfrom torch.utils.data import DataLoader\nfrom lineflow.datasets.wikitext import cached_get_wikitext\nimport torchpipe as tp\n\ndataset = cached_get_wikitext('wikitext-2')\n# Preprocessing dataset\ntrain_data = tp.Dataset(dataset['train']) \\\n    .flat_map(lambda x: x.split() + ['<eos>']) \\\n    .window(35) \\\n    .parallel() \\\n    .shuffle(64 * 100) \\\n    .batch(64)\n\n# Iterating dataset\nloader = DataLoader(train_data, num_workers=4, collate_fn=lambda x: x)\nfor x in loader:\n    ...\n```",
        "description_content_type": "text/markdown",
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://github.com/yasufumy/torchpipe",
        "keywords": "",
        "license": "MIT",
        "maintainer": "",
        "maintainer_email": "",
        "name": "torchpipe",
        "package_url": "https://pypi.org/project/torchpipe/",
        "platform": "",
        "project_url": "https://pypi.org/project/torchpipe/",
        "project_urls": {
            "Homepage": "https://github.com/yasufumy/torchpipe"
        },
        "release_url": "https://pypi.org/project/torchpipe/0.1.0/",
        "requires_dist": null,
        "requires_python": "",
        "summary": "Simple ETL Pipeline for PyTorch",
        "version": "0.1.0"
    },
    "last_serial": 6000097,
    "releases": {
        "0.0.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "69422e6195159f079ddee8f22c9f9042",
                    "sha256": "e259506188d7ba6d0a451110552efcc99ba5a8ab7dadef1dfbe628a04d45ea29"
                },
                "downloads": -1,
                "filename": "torchpipe-0.0.1.tar.gz",
                "has_sig": false,
                "md5_digest": "69422e6195159f079ddee8f22c9f9042",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 13904,
                "upload_time": "2019-06-28T07:54:21",
                "url": "https://files.pythonhosted.org/packages/e9/f2/cd4a9be6e3ecd15c858a064cae39f5b908ba348087dc2db7296d64c61bcd/torchpipe-0.0.1.tar.gz"
            }
        ],
        "0.0.2": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "3c5f57d1525da866da22ce0744cd693e",
                    "sha256": "2c88117a6403520d74105bb4b37c586283ba74413af9b6d6c1667d92fdf8e29a"
                },
                "downloads": -1,
                "filename": "torchpipe-0.0.2.tar.gz",
                "has_sig": false,
                "md5_digest": "3c5f57d1525da866da22ce0744cd693e",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 13906,
                "upload_time": "2019-08-09T05:59:22",
                "url": "https://files.pythonhosted.org/packages/31/5d/cc9fdca59c88905e899df159c118babdf89545c462a7eaf28c5c57e2f43f/torchpipe-0.0.2.tar.gz"
            }
        ],
        "0.0.3": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "015762621479380497cb9a596f04357c",
                    "sha256": "6bb61dd6c3e870f63ecba2862c978244a10433b31753b7c3a58664fd8549a20d"
                },
                "downloads": -1,
                "filename": "torchpipe-0.0.3.tar.gz",
                "has_sig": false,
                "md5_digest": "015762621479380497cb9a596f04357c",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 14965,
                "upload_time": "2019-08-21T00:49:57",
                "url": "https://files.pythonhosted.org/packages/d7/1d/10abb797580ed194554f7297416a8dd30309d4528d2102951320e0c083a4/torchpipe-0.0.3.tar.gz"
            }
        ],
        "0.1.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "13311d1bea46920e204fe3a9e4de612e",
                    "sha256": "6c49012012811347e1bad68eee1a0a112c50e713dc2a101a47747ffc187dc5d0"
                },
                "downloads": -1,
                "filename": "torchpipe-0.1.0.tar.gz",
                "has_sig": false,
                "md5_digest": "13311d1bea46920e204fe3a9e4de612e",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 14621,
                "upload_time": "2019-10-19T15:31:06",
                "url": "https://files.pythonhosted.org/packages/94/e2/2da8c9557dd92b76e589f66ca2491f6993fddb8f9b167e4ef16232a25e20/torchpipe-0.1.0.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "13311d1bea46920e204fe3a9e4de612e",
                "sha256": "6c49012012811347e1bad68eee1a0a112c50e713dc2a101a47747ffc187dc5d0"
            },
            "downloads": -1,
            "filename": "torchpipe-0.1.0.tar.gz",
            "has_sig": false,
            "md5_digest": "13311d1bea46920e204fe3a9e4de612e",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 14621,
            "upload_time": "2019-10-19T15:31:06",
            "url": "https://files.pythonhosted.org/packages/94/e2/2da8c9557dd92b76e589f66ca2491f6993fddb8f9b167e4ef16232a25e20/torchpipe-0.1.0.tar.gz"
        }
    ]
}