{
    "info": {
        "author": "Hugo Hromic",
        "author_email": "hhromic@gmail.com",
        "bugtrack_url": null,
        "classifiers": [
            "Intended Audience :: Developers",
            "Topic :: System :: Archiving"
        ],
        "description": "# python-inpout\n\nSimple library for input/output using MessagePack and LZ4 compression in Python.\n\n## Installation\n\nYou can use `pip` (or any PyPI-compatible package manager) for installation:\n\n    pip install inpout\n\nor, if you prefer a local user installation:\n\n    pip install --user inpout\n\nFor Microsoft Windows users, you might need to run `pip` through the Python interpreter:\n\n    python -m pip install inpout\n\n**Note:** Visual C++ 14.0 is required for Windows installation. Get it with *Microsoft Visual C++ Build Tools*: <https://visualstudio.microsoft.com/downloads/>\n\n## Usage\n\nTo use the functionality of this library, simply import it in your Python programs:\n\n    import inpout\n\n\n### High-Level Functions\n\nFor saving/loading data using MessagePack and LZ4 compression, the following high-level convenience functions are provided in the root namespace:\n\n* `load_obj(path, compression=None, **kwargs)`: return a single object loaded from a file on disk.\n\n  See `data_unpacker()` for details on the arguments.\n\n* `load_iter(path, compression=None, **kwargs)`: return an iterator of objects loaded from a file on disk.\n\n  See `data_unpacker()` for details on the arguments.\n\n* `save_obj(obj, path, compression=None, level=None, **kwargs)`: save a single object `obj` to a file on disk.\n\n  See `data_pack()` for details on the arguments.\n\n* `save_iter(iterable, path, compression=None, level=None, **kwargs)`: save an interable of objects `iterable` to a file on disk.\n\n  See `data_pack()` for details on the arguments.\n\n\n### Context Manager Functions\n\nFor more flexibility, the following context manager functions are provided in the root namespace:\n\n* `data_unpacker(path, compression=None, **kwargs)`: create a data unpacker (MessagePack) context manager with optional compression (LZ4) support to be used as an iterable unpacker.\n  - `path`: path to the file on disk containing the data to read.\n  - `compression`: boolean flag for using LZ4 compression. If `None`, defaults to `True`.\n  - `kwargs`: keyword arguments passed directly to the MessagePack unpacker. See below.\n\n* `data_pack(path, compression=None, level=None, **kwargs)`: create a data pack (MessagePack) context manager with optional compression (LZ4) support to be used as a packing function.\n  - `path`: path to the file on disk that will contain the written data.\n  - `compression`: boolean flag for using LZ4 compression. If `None`, defaults to `True`.\n  - `level`: the compression level for the LZ4 compressor. See `compressor()` for details.\n  - `kwargs`: keyword arguments passed directly to the MessagePack packer. See below.\n\n### Packing Functions\n\nFor packing/unpacking data with MessagePack directly without compression, the following functions are provided in `inpout.packing`:\n\n* `pack(obj, stream, **kwargs)`: pack a single object using MessagePack (with extended types support) to a stream of bytes.\n  - `obj`: the object to pack.\n  - `stream`: the bytes stream to use for writing data.\n  - `kwargs`: keyword arguments passed directly to the MessagePack packer. See below.\n\n* `packb(obj, **kwargs)`: pack a single object using MessagePack (with extended types support) and return packed bytes.\n  - `obj`: the object to pack.\n  - `kwargs`: keyword arguments passed directly to the MessagePack packer. See below.\n\n* `unpack(stream, **kwargs)`: unpack a stream of packed bytes using MessagePack (with extended types support) and return a single unpacked object.\n  - `stream`: the bytes stream to use for reading data.\n  - `kwargs`: keyword arguments passed directly to the MessagePack unpacker. See below.\n\n* `unpackb(packed, **kwargs)`: unpack packed bytes using MessagePack (with extended types support) and return a single unpacked object.\n  - `packed`: the packed bytes to unpack.\n  - `kwargs`: keyword arguments passed directly to the MessagePack unpacker. See below.\n\n### Compressing Functions\n\nFor compressing/decompressing arbitrary data with LZ4 directly without packing, the following context manager functions are provided in `inpout.compression`:\n\n* `decompressor(path)`: create a data decompressing context manager to be used as reader.\n  - `path`: path to the file on disk containing the compressed data.\n\n* `compressor(path, level=None)`: create a data compressing context manager to be used as a writer.\n  - `path`: path to the file on disk that will contain the compressed data.\n  - `level`: compression level to use. Defaults to `LZ4F_COMPRESSION_MAX` if `None`. Values lower than `3` (including negative ones) use fast compression. Recommended range for hc-type compression is between `4` and `9`. More information can be [found here](https://github.com/lz4/lz4/blob/dev/README.md).\n\n### Keyword Arguments for MessagePack\n\nFunctions involving data packing with MessagePack support optional keyword arguments `kwargs` to be passed directly to MessagePack packer and unpacker. Useful options are described below:\n\n* `use_list`: can be `True` (default) or `False`.\n\n  List is the default sequence/array type for Python. But tuples are lighter than lists. You can use `use_list=False` while unpacking when performance is important for your program. Python objects that require hashable elements such as `dict` or `set` can't use lists as key, therefore `use_list=False` is required for unpacking data containing tuples as keys.\n\n## Examples\n\nBelow is example code of how to use the main convenience functions of this library.\n\n```python\nfrom datetime import datetime\nimport inpout\n\n# create some Python objects to test, set and datetime are supported out of the box\nobj1 = [1,2,3,4,5]\nobj2 = (\"test\", 1234)\nobj3 = {\"test\": 1234, \"test2\": 5678}\nobj4 = {\"a\", \"b\", \"c\", 5, 6, 7, 8}\nobj5 = datetime.now()\nobj6 = {(1,2): \"tuple_key\"}\n\n# save all the above objects as a single tuple to disk\ninpout.save_obj((obj1, obj2, obj3, obj4, obj5), \"test1.mp.lz4\")\n\n# save all the above objects in order to disk one by one (iterator)\niterator = (o for o in (obj1, obj2, obj3, obj4, obj5))\ninpout.save_iter(iterator, \"test2.mp.lz4\")\n\n# save an object with a tuple as key to demonstrate 'use_list=False'\ninpout.save_obj(obj6, \"test3.mp.lz4\")\n\n# load the first test file\ndata = inpout.load_obj(\"test1.mp.lz4\")\nprint(\"DATA=%r\" % (data,))\n\n# load the second test file (iterator)\nfor obj in inpout.load_iter(\"test2.mp.lz4\"):\n    print(\"OBJ=%r\" % (obj,))\n\n# load the third test file using tuple types, otherwise it fails\ndata = inpout.load_obj(\"test3.mp.lz4\", use_list=False)\nprint(\"DATA=%r\" % (data,))\n\n# demonstrate the data pack function\nwith inpout.data_pack(\"test4.mp.lz4\") as pack:\n    for obj in (obj1, obj2, obj3, obj4, obj5, obj6):\n        pack(obj)\n\n# demonstrate the data unpacker function\nwith inpout.data_unpacker(\"test4.mp.lz4\", use_list=False) as unpacker:\n    for obj in unpacker:\n        print(\"OBJ=%r\" % (obj,))\n\n# demonstrate the data pack function (no compression)\nwith inpout.data_pack(\"test4.mp\", compression=False) as pack:\n    for obj in (obj1, obj2, obj3, obj4, obj5, obj6):\n        pack(obj)\n\n# demonstrate the data unpacker function (no compression)\nwith inpout.data_unpacker(\"test4.mp\", compression=False, use_list=False) as unpacker:\n    for obj in unpacker:\n        print(\"OBJ=%r\" % (obj,))\n```\n\n## MessagePack Extended Types\n\nThis library supports MessagePack extended types and includes encoders/decoders for two standard Python objects: `set` (typecode `127`) and `datetime` (typecode `126`). These are automatically registered upon importing the library.\n\n* `set` objects are serialised as tuples containing their elements and reconstructed from these stored tuples.\n* `datetime` objects are serialised as a tuple of two integers `(seconds, microseconds)` representing the number of seconds and microseconds since the UNIX epoch (00:00:00 Thursday, 1 January 1970). Timezone information is used for the conversion but not stored, therefore `datetime` objects are reconstructed as naive, i.e. without timezone.\n\nYou can also easily create your own encoders/decoders for Python objects and register them for this library to be used during serialisation/deserialisation:\n\n```python\nimport inpout\n\nclass MyType(object):\n    def __init__(self, data1, data2):\n        self.data1 = data1\n        self.data2 = data2\n\n# define a representation for your type (encoder)\n# we will assign '50' as the typecode for this type\ndef encode_mytype(obj, packb, ext_type):\n    return ext_type(50, packb((obj.data1, obj.data2)))\n\n# define how to create your type from your representation (decoder)\ndef decode_mytype(data, unpackb):\n    data1, data2 = unpackb(data)\n    return MyType(data1, data2)\n\n# register custom encoder/decoders for your type\ninpout.packing.register_ext_type_encoder(MyType, encode_mytype)\ninpout.packing.register_ext_type_decoder(50, decode_mytype)\n\n# test saving/loading your type\nobj = MyType(\"test\", 1234)\ninpout.save_obj(obj, \"test.mp.lz4\")\nobj2 = inpout.load_obj(\"test.mp.lz4\")\nprint(obj2.data1, obj2.data2)\n```\n\nYou can use any typecode for your own extended types, however it must be between `0` and `125` (inclusive).\n\nMore information about MessagePack extended types can be [found here](https://github.com/msgpack/msgpack/blob/master/spec.md#extension-types).\n\n## Command-line Tools\n\nThe library includes the following command-line tools that are installed automatically by `pip`:\n\n* `inpout-pprint`: iterate and pretty-print data files generated by this library.\n\n  This tool is based on the `load_iter` function with the `use_list=False` keyword argument. Compression is activated if filenames end with `.lz4` (case insensitive). Usage:\n\n      $ inpout-pprint <FILE> <FILE> ...\n\n## License\n\nThis software is under the **Apache License 2.0**.\n\n    Licensed under the Apache License, Version 2.0 (the \"License\");\n    you may not use this file except in compliance with the License.\n    You may obtain a copy of the License at\n\n        http://www.apache.org/licenses/LICENSE-2.0\n\n    Unless required by applicable law or agreed to in writing, software\n    distributed under the License is distributed on an \"AS IS\" BASIS,\n    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n    See the License for the specific language governing permissions and\n    limitations under the License.\n\n\n\n",
        "description_content_type": "text/markdown",
        "docs_url": null,
        "download_url": "https://github.com/hhromic/python-inpout/tarball/1.0.5",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://github.com/hhromic/python-inpout",
        "keywords": "messagepack,lz4,input,output,library",
        "license": "Apache-2.0",
        "maintainer": "Hugo Hromic",
        "maintainer_email": "hhromic@gmail.com",
        "name": "inpout",
        "package_url": "https://pypi.org/project/inpout/",
        "platform": "all",
        "project_url": "https://pypi.org/project/inpout/",
        "project_urls": {
            "Download": "https://github.com/hhromic/python-inpout/tarball/1.0.5",
            "Homepage": "https://github.com/hhromic/python-inpout"
        },
        "release_url": "https://pypi.org/project/inpout/1.0.5/",
        "requires_dist": [
            "msgpack",
            "py-lz4framed"
        ],
        "requires_python": "",
        "summary": "Simple input/output using MessagePack and LZ4 for Python",
        "version": "1.0.5"
    },
    "last_serial": 5097269,
    "releases": {
        "1.0.5": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "9b50239f4a77da84dce7fa5301d8aa5d",
                    "sha256": "43115fc20625379c9641d03f8861780cdfac733abdcccdd4bd66369bd8d48699"
                },
                "downloads": -1,
                "filename": "inpout-1.0.5-py2.py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "9b50239f4a77da84dce7fa5301d8aa5d",
                "packagetype": "bdist_wheel",
                "python_version": "py2.py3",
                "requires_python": null,
                "size": 14690,
                "upload_time": "2019-04-04T13:20:41",
                "url": "https://files.pythonhosted.org/packages/ff/aa/bf4425edf0f967fab8be375afb35378e5ab23f10ff25e10276f7f595ac53/inpout-1.0.5-py2.py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "e921898ff051981123e9c873f6574020",
                    "sha256": "87e9081a3ce547e2f194e97c44e98adafd089e5eb6bdc42a9a6474e2166775d7"
                },
                "downloads": -1,
                "filename": "inpout-1.0.5.tar.gz",
                "has_sig": false,
                "md5_digest": "e921898ff051981123e9c873f6574020",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 8511,
                "upload_time": "2019-04-04T13:20:43",
                "url": "https://files.pythonhosted.org/packages/fe/53/d0a08e2b980dd67bb3e5bae29e2a25f93694d5b1272d43e53935c13d56b8/inpout-1.0.5.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "9b50239f4a77da84dce7fa5301d8aa5d",
                "sha256": "43115fc20625379c9641d03f8861780cdfac733abdcccdd4bd66369bd8d48699"
            },
            "downloads": -1,
            "filename": "inpout-1.0.5-py2.py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "9b50239f4a77da84dce7fa5301d8aa5d",
            "packagetype": "bdist_wheel",
            "python_version": "py2.py3",
            "requires_python": null,
            "size": 14690,
            "upload_time": "2019-04-04T13:20:41",
            "url": "https://files.pythonhosted.org/packages/ff/aa/bf4425edf0f967fab8be375afb35378e5ab23f10ff25e10276f7f595ac53/inpout-1.0.5-py2.py3-none-any.whl"
        },
        {
            "comment_text": "",
            "digests": {
                "md5": "e921898ff051981123e9c873f6574020",
                "sha256": "87e9081a3ce547e2f194e97c44e98adafd089e5eb6bdc42a9a6474e2166775d7"
            },
            "downloads": -1,
            "filename": "inpout-1.0.5.tar.gz",
            "has_sig": false,
            "md5_digest": "e921898ff051981123e9c873f6574020",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 8511,
            "upload_time": "2019-04-04T13:20:43",
            "url": "https://files.pythonhosted.org/packages/fe/53/d0a08e2b980dd67bb3e5bae29e2a25f93694d5b1272d43e53935c13d56b8/inpout-1.0.5.tar.gz"
        }
    ]
}