{ "info": { "author": "eterna2", "author_email": "eterna2@hotmail.com", "bugtrack_url": null, "classifiers": [ "Development Status :: 2 - Pre-Alpha", "Intended Audience :: Developers", "Natural Language :: English", "Programming Language :: Python :: 3", "Programming Language :: Python :: 3.6", "Programming Language :: Python :: 3.7" ], "description": "# e2fyi-utils\n\n[![PyPI version](https://badge.fury.io/py/e2fyi-utils.svg)](https://badge.fury.io/py/e2fyi-utils)\n[![Build Status](https://travis-ci.org/e2fyi/py-utils.svg?branch=master)](https://travis-ci.org/e2fyi/py-utils)\n[![Coverage Status](https://coveralls.io/repos/github/e2fyi/py-utils/badge.svg?branch=master)](https://coveralls.io/github/e2fyi/py-utils?branch=master)\n[![Documentation Status](https://readthedocs.org/projects/e2fyi-utils/badge/?version=latest)](https://e2fyi-utils.readthedocs.io/en/latest/?badge=latest)\n[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)\n[![Downloads](https://pepy.tech/badge/e2fyi-utils/month)](https://pepy.tech/project/e2fyi-utils/month)\n\n`e2fyi-utils` is an `e2fyi` namespaced python package with `utils` subpackage\n(i.e. `e2fyi.utils`) which holds a collections of useful helper classes and\nfunctions to interact with various cloud resources.\n\nAPI documentation can be found at [https://e2fyi-utils.readthedocs.io/en/latest/](https://e2fyi-utils.readthedocs.io/en/latest/).\n\nChange logs are available in [CHANGELOG.md](./CHANGELOG.md).\n\n> - Python 3.6 and above\n> - Licensed under [Apache-2.0](./LICENSE).\n\n\n## Quickstart\n\n```bash\npip install e2fyi-utils>=0.2\n```\n\n### S3Stream\n\n`S3Stream` represents the data stream of a S3 resource, and provides static\nmethods to convert any python objects into a stream. This is generally used with\n`S3Resource` to upload or download resource from S3 buckets.\n\n> NOTE:\n> - `str`, `float`, `int`, and `bool` will be saved as txt files with mime type \"text/plain\".\n> - `pydantic` models, `dict` or `list` will be saved as json files with mime type \"application/json\" (fallback to pickle if unable to serialize into json string).\n> - `pandas` dataframe or series can be saved as either a csv (\"application/csv\") or json format (\"application/json\").\n> - path to files will be read with `open` and mime type will be inferred (fallback to \"application/octet-stream\").\n> - all other python objects will be pickled with `joblib`.\n\n```py\nimport io\n\nimport pandas as pd\n\nfrom e2fyi.utils.aws import S3Stream\nfrom pydantic import BaseModel\n\n# create a s3 stream\nstream = S3Stream(io.StringIO(\"random text\"), \"text/plain\")\nprint(stream.read()) # prints \"random text\"\nprint(stream.content_type) # prints \"text/plain\"\n\n# string\nstream = S3Stream.from_any(\"hello world\")\nprint(stream.read()) # prints \"hello world\"\nprint(stream.content_type) # prints \"text/plain\"\n\n# dict\nstream = S3Stream.from_any({\"foo\": \"bar\"})\nprint(stream.read()) # prints \"{\"foo\": \"bar\"}\"\nprint(stream.content_type) # prints \"application/json\"\n\n# pandas dataframe as csv\ndf = pd.DataFrame([{\"key\": \"a\", \"value\": 1}, {\"key\": \"b\", \"value\": 2}])\nstream = S3Stream.from_any(df, index=False) # do not include index column\nprint(stream.read()) # prints string as csv format for the dataframe\nprint(stream.content_type) # prints \"application/csv\"\n\n# pandas dataframe as json\nstream = S3Stream.from_any(df, orient=\"records\") # orient dataframe as records\nprint(stream.read()) # prints string as json list for the dataframe\nprint(stream.content_type) # prints \"application/json\"\n\n\n# pydantic model\nclass Person(BaseModel):\n name: str\n age: int\nstream = S3Stream.from_any(Person(name=\"william\", age=21))\nprint(stream.read()) # prints \"{\"name\": \"william\", \"age\": 21}\"\nprint(stream.content_type) # prints \"application/json\"\n\n\n# any other python objects\nclass Pet:\n name: str\n age: int\nstream = S3Stream.from_any(Pet(name=\"kopi\", age=1))\nprint(stream.read()) # prints some binary bytes\nprint(stream.content_type) # prints \"application/octet-stream\"\n\n```\n\n### S3Resource\n\n`S3Resource` represents a resource in S3 currently or a local resource that will\nbe uploaded to S3. `S3Resource` constructor will automatically attempts to convert\nany inputs into a `S3Stream`, but for more granular control `S3Stream.from_any`\nshould be used instead to create the `S3Stream`.\n\n`S3Resource` is a readable stream - i.e. it has `read`, `seek`, and `close`.\n\n> NOTE:\n>\n> See https://boto3.amazonaws.com/v1/documentation/api/latest/reference/customizations/s3.html\n> for additional keyword arguments that can be passed to S3Resource.\n\n#### Example: Creating S3Resource from local python object or file.\n```py\nimport boto3\n\nfrom e2fyi.utils.aws import S3Resource, S3Stream\n\n# create custom s3 client\ns3client = boto3.client(\n 's3',\n aws_access_key_id=ACCESS_KEY,\n aws_secret_access_key=SECRET_KEY\n)\n\n# creates a local copy of s3 resource with S3Stream from a local file\nobj = S3Resource(\n # full path shld be \"prefix/some_file.json\"\n filename=\"some_file.json\",\n prefix=\"prefix/\",\n # bucket to download from or upload to\n bucketname=\"some_bucket\",\n # or \"s3n://\" or \"s3://\"\n protocol=\"s3a://\",\n # uses default client if not provided\n s3client=s3client,\n # attempts to convert to S3Stream if input is not a S3Stream\n stream=S3Stream.from_file(\"./some_path/some_file.json\"),\n # addition kwarg to pass to `s3.upload_fileobj` or `s3.download_fileobj` methods\n Metadata={\"label\": \"foo\"}\n)\nprint(obj.key) # prints \"prefix/some_file.json\"\nprint(obj.uri) # prints \"s3a://some_bucket/prefix/some_file.json\"\n\n# will attempt to fix prefix and filename if incorrect filename is provided\nobj = S3Resource(\n filename=\"subfolder/some_file.json\",\n prefix=\"prefix/\"\n)\nprint(obj.filename) # prints \"some_file.json\"\nprint(obj.prefix) # prints \"prefix/subfolder/\"\n```\n\n#### Example: Upload S3Resource to S3\n```py\n# creates a local copy of s3 resource with some python object\nobj = S3Resource(\n filename=\"some_file.txt\",\n prefix=\"prefix/\",\n bucketname=\"some_bucket\",\n stream={\"some\": \"dict\"},\n)\n\n# upload obj to s3 bucket \"some_bucket\" with the key \"prefix/some_file.json\"\n# with the json string content.\nobj.save()\n\n# upload to s3 bucket \"another_bucket\" instead with a metadata tag.\nobj.save(\"another_bucket\", MetaData={\"label\": \"foo\"})\n```\n\n#### Example: Read S3Resource from S3\n```py\nfrom pydantic import BaseModel\n\n# do not provide a stream input to the S3Resource constructor\nobj = S3Resource(\n filename=\"some_file.json\",\n prefix=\"prefix/\",\n bucketname=\"some_bucket\",\n content_type=\"application/json\"\n)\n\n# read the resource like a normal file object from S3\ndata = obj.read()\nprint(type(data)) # prints \n\n# read and load json string into a dict or list\n# for content_type == \"application/json\" only\ndata_obj = obj.load()\nprint(type(data_obj)) # prints or \n\n\n# read and convert into a pydantic model\nclass Person(BaseModel):\n name: str\n age: int\n\n# automatically unpack the dict\ndata_obj = obj.load(lambda name, age: Person(name=name, age=age))\n# alternatively, do not unpack\ndata_obj = obj.load(lambda data: Person(**data), unpack=False)\nprint(type(data_obj)) # prints \n```\n\n### S3Bucket\n\n`S3Bucket` is an abstraction of the actual S3 bucket with methods to interact\nwith the actual S3 bucket (e.g. list objects inside the bucket), and some utility\nmethods.\n\nPrefix rules can also be set during the creation of the `S3Bucket` object - i.e.\nenforce a particular prefix rules for a particular bucket.\n\n#### Quickstart\n```py\nfrom e2fyi.utils.aws import S3Bucket\n\n# prints key for all resources with prefix \"some_folder/\"\nfor resource in S3Bucket(\"some_bucket\").list(\"some_folder/\"):\n print(resource.key)\n\n# prints key for the first 2,000 resources with prefix \"some_folder/\"\nfor resource in S3Bucket(\"some_bucket\").list(\"some_folder/\", max_objects=2000):\n print(resource.key)\n\n# creates a s3 bucket with prefix rule\nprj_bucket = S3Bucket(\"some_bucket\", get_prefix=lambda prefix: \"prj-a/%s\" % prefix)\nfor resource in prj_bucket.list(\"some_folder/\"):\n print(resource.key) # prints \"prj-a/some_folder/\"\n print(resource.stats) # prints metadata for the resource (e.g. size, etag)\n\n# get obj key in the bucket\nprint(prj_bucket.create_resource_key(\"foo.json\")) # prints \"prj-a/foo.json\"\n\n# get obj uri in the bucket\n# prints \"s3a://some_bucket/prj-a/foo.json\"\nprint(prj_bucket.create_resource_uri(\"foo.json\", \"s3a://\"))\n\n# create S3Resource in bucket to read in\nfoo = prj_bucket.create_resource(\"foo.json\", \"application/json\")\n# read \"s3a://some_bucket/prj-a/foo.json\" and load as a dict (or list)\nfoo_dict = foo.load()\n\n# create S3Resource in bucket and save to \"s3a://some_bucket/prj-a/foo.json\"\nprj_bucket.create_resource(\"foo.json\", obj={\"foo\": \"bar\"}).save()\n```\n\n### e2fyi.utils.core.Maybe\n\n`Maybe` represents an uncertain object (exception might be raised so no value\nwill be returned). This is generally used inside a function where all exceptions\nwill be caught.\n\n> NOTE:\n> - `Maybe.value` is the actual returned value.\n> - `Maybe.exception` is the exception caught (if any).\n> - `Maybe.with_default` method can be used to provide a default value if no value\nis returned.\n> - `Maybe.is_ok` method can be used to check if any value is returned.\n\n```py\nimport logging\n\nfrom e2fyi.utils.core import Maybe\n\n\ndef load_from_file(filepath: str) -> Maybe[string]:\n \"\"\"loads the content of a file.\"\"\"\n try:\n with open(filepath, \"r\") as fp:\n return Maybe(fp.read())\n except IOError as err:\n return Maybe(exception=err)\n\ndata = load_from_file(\"some_file.json\")\n\n# print with a default value fallback\nprint(data.with_default(\"default value\"))\n\n# print data if ok, else log exception\nif data.is_ok:\n print(data)\nelse:\n logging.exception(data.exception)\n\n```", "description_content_type": "text/markdown", "docs_url": null, "download_url": "", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "https://github.com/e2fyi/py-utils", "keywords": "util aws s3 boto3 upload", "license": "", "maintainer": "", "maintainer_email": "", "name": "e2fyi-utils", "package_url": "https://pypi.org/project/e2fyi-utils/", "platform": "", "project_url": "https://pypi.org/project/e2fyi-utils/", "project_urls": { "Homepage": "https://github.com/e2fyi/py-utils" }, "release_url": "https://pypi.org/project/e2fyi-utils/0.2.2/", "requires_dist": null, "requires_python": ">=3.6", "summary": "General utils for interacting with aws resources.", "version": "0.2.2", "yanked": false, "yanked_reason": null }, "last_serial": 8317589, "releases": { "0.1.0": [ { "comment_text": "", "digests": { "md5": "c84e8943f59ede77d6b68bd452599622", "sha256": "f158ab4b9eccd829e5b99dc0cb03e7eef8a4ec710fb63711690f0a78e5c74b58" }, "downloads": -1, "filename": "e2fyi-utils-0.1.0.tar.gz", "has_sig": false, "md5_digest": "c84e8943f59ede77d6b68bd452599622", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.6", "size": 15699, "upload_time": "2019-12-04T09:45:35", "upload_time_iso_8601": "2019-12-04T09:45:35.987282Z", "url": "https://files.pythonhosted.org/packages/1a/64/ebaf7c7283ee064fdac2f95ffd578958265fc986bd48cb9bd7718ad282b5/e2fyi-utils-0.1.0.tar.gz", "yanked": false, "yanked_reason": null } ], "0.1.0a1": [ { "comment_text": "", "digests": { "md5": "15407b5f5f531e566fc5a1990da0260a", "sha256": "271616d0a72f711b187b18f2a6d96aaec90c294a69599582e311ff4bfdefe86f" }, "downloads": -1, "filename": "e2fyi-utils-0.1.0a1.tar.gz", "has_sig": false, "md5_digest": "15407b5f5f531e566fc5a1990da0260a", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.6", "size": 7268, "upload_time": "2019-10-31T06:24:30", "upload_time_iso_8601": "2019-10-31T06:24:30.142324Z", "url": "https://files.pythonhosted.org/packages/be/3e/cc4110810f2f997a19a4fa256b93f2fc00e4e4016098152528a7bc89a5b3/e2fyi-utils-0.1.0a1.tar.gz", "yanked": false, "yanked_reason": null } ], "0.1.0a1.post1": [ { "comment_text": "", "digests": { "md5": "29451494722eaed1e0c67d2a43ac7a99", "sha256": "d3e1e2be006f9c5901cc5bcf128459204a134eda1077d8294730a4016acadf86" }, "downloads": -1, "filename": "e2fyi-utils-0.1.0a1.post1.tar.gz", "has_sig": false, "md5_digest": "29451494722eaed1e0c67d2a43ac7a99", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.6", "size": 7866, "upload_time": "2019-10-31T06:45:51", "upload_time_iso_8601": "2019-10-31T06:45:51.570091Z", "url": "https://files.pythonhosted.org/packages/72/65/5990aa9e2af343bfe2330f695dd92aa8bec3cb51c48d87febaf02abe70a7/e2fyi-utils-0.1.0a1.post1.tar.gz", "yanked": false, "yanked_reason": null } ], "0.1.0a2": [ { "comment_text": "", "digests": { "md5": "8ea28bb44db1319004d32c156d2fc22d", "sha256": "2489ee31b0ccf3e642ca1e0e20e760544303c18f99a79ba045a28c8d988ba87a" }, "downloads": -1, "filename": "e2fyi-utils-0.1.0a2.tar.gz", "has_sig": false, "md5_digest": "8ea28bb44db1319004d32c156d2fc22d", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.6", "size": 8198, "upload_time": "2019-10-31T07:02:48", "upload_time_iso_8601": "2019-10-31T07:02:48.005620Z", "url": "https://files.pythonhosted.org/packages/9d/8e/f5de54922e20b326d9b5026763f2cc6a83fbcc1d66afdd0e3c59cfa938a2/e2fyi-utils-0.1.0a2.tar.gz", "yanked": false, "yanked_reason": null } ], "0.1.0a3": [ { "comment_text": "", "digests": { "md5": "e1678b4c5bf150c1439779e142932a1b", "sha256": "e765dd5c79d7cd87c44739ef520d700e91914f506c30317f7f69bcea63f537ce" }, "downloads": -1, "filename": "e2fyi-utils-0.1.0a3.tar.gz", "has_sig": false, "md5_digest": "e1678b4c5bf150c1439779e142932a1b", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.6", "size": 14660, "upload_time": "2019-11-11T08:58:27", "upload_time_iso_8601": "2019-11-11T08:58:27.233026Z", "url": "https://files.pythonhosted.org/packages/c8/65/d2839bcb7c5801a044a5740796096bcde276537951384e9c4e496efee84b/e2fyi-utils-0.1.0a3.tar.gz", "yanked": false, "yanked_reason": null } ], "0.1.0a4": [ { "comment_text": "", "digests": { "md5": "0263ec87a9f2b4c4b069888e93b3907e", "sha256": "536149f77e86af7f73e2062bf72163507a2ceeaad975bf2251369780874ecc96" }, "downloads": -1, "filename": "e2fyi-utils-0.1.0a4.tar.gz", "has_sig": false, "md5_digest": "0263ec87a9f2b4c4b069888e93b3907e", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.6", "size": 15866, "upload_time": "2019-11-12T11:21:16", "upload_time_iso_8601": "2019-11-12T11:21:16.583329Z", "url": "https://files.pythonhosted.org/packages/3d/65/0d5f69f13fee5d0a30e2cf505ea91281cfa5b61c110d097b97f3cd752de0/e2fyi-utils-0.1.0a4.tar.gz", "yanked": false, "yanked_reason": null } ], "0.2.0": [ { "comment_text": "", "digests": { "md5": "4449fdd65e46f71e9fa1c9f3d5aee63d", "sha256": "d10c5cb9b81a54f97b4d5d75175bc4f5b9ef9684704b884d33d7cb0259b458ef" }, "downloads": -1, "filename": "e2fyi-utils-0.2.0.tar.gz", "has_sig": false, "md5_digest": "4449fdd65e46f71e9fa1c9f3d5aee63d", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.6", "size": 22628, "upload_time": "2019-12-12T11:16:13", "upload_time_iso_8601": "2019-12-12T11:16:13.087680Z", "url": "https://files.pythonhosted.org/packages/84/e2/e79739f6c8fd2d15f2bb32efb799c58c189f2f5687bb343bb6b25833927d/e2fyi-utils-0.2.0.tar.gz", "yanked": false, "yanked_reason": null } ], "0.2.0.post1": [ { "comment_text": "", "digests": { "md5": "622447b2821685cb491988076257533e", "sha256": "ba3eda779e5a0d59b1fd05f19a6d58077dc6791734ba95cc55dffb9f6fc8c27b" }, "downloads": -1, "filename": "e2fyi-utils-0.2.0.post1.tar.gz", "has_sig": false, "md5_digest": "622447b2821685cb491988076257533e", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.6", "size": 22663, "upload_time": "2019-12-12T11:59:34", "upload_time_iso_8601": "2019-12-12T11:59:34.091955Z", "url": "https://files.pythonhosted.org/packages/4e/c3/6641897c8f8f9d7e3a813592d23ed43e34e8130e62421384171e139feb10/e2fyi-utils-0.2.0.post1.tar.gz", "yanked": false, "yanked_reason": null } ], "0.2.1": [ { "comment_text": "", "digests": { "md5": "ca2a0eacb0a4a54cfdf2395bd7443476", "sha256": "6fcba91e471d1ed64a9ddd1d4c8ca94e2f645467cd00f60d75a5071c1d6fdc79" }, "downloads": -1, "filename": "e2fyi-utils-0.2.1.tar.gz", "has_sig": false, "md5_digest": "ca2a0eacb0a4a54cfdf2395bd7443476", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.6", "size": 22662, "upload_time": "2019-12-27T03:03:26", "upload_time_iso_8601": "2019-12-27T03:03:26.494570Z", "url": "https://files.pythonhosted.org/packages/16/1e/26a45c17c3029a661412e693c770dae8855af02172c2a785e23238f6180a/e2fyi-utils-0.2.1.tar.gz", "yanked": false, "yanked_reason": null } ], "0.2.2": [ { "comment_text": "", "digests": { "md5": "35588cf499713b30a288b96841365ec8", "sha256": "66dc0f4774e43a2c92331a54e850a2ae0a0aa16460c2848dc82d69a3bd8e202d" }, "downloads": -1, "filename": "e2fyi-utils-0.2.2.tar.gz", "has_sig": false, "md5_digest": "35588cf499713b30a288b96841365ec8", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.6", "size": 22887, "upload_time": "2019-12-27T08:58:46", "upload_time_iso_8601": "2019-12-27T08:58:46.095020Z", "url": "https://files.pythonhosted.org/packages/04/4c/9b4acc0f2f3ab54c009ac334319cbc7a720b53cd7d45b05d6073199fd081/e2fyi-utils-0.2.2.tar.gz", "yanked": false, "yanked_reason": null } ], "0.3.0rc1": [ { "comment_text": "", "digests": { "md5": "904a08eba045a938d3693f5ae8e5b0bb", "sha256": "8857bba8e0619fc67fe487e3ac6ea0ba3f6f5d6f451ffbf1efcfa6b3764a14aa" }, "downloads": -1, "filename": "e2fyi_utils-0.3.0rc1-py3-none-any.whl", "has_sig": false, "md5_digest": "904a08eba045a938d3693f5ae8e5b0bb", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": ">=3.6.1,<4", "size": 25270, "upload_time": "2020-10-01T17:18:17", "upload_time_iso_8601": "2020-10-01T17:18:17.680574Z", "url": "https://files.pythonhosted.org/packages/c4/c4/0f3c5b86eacb85a1fedb973ecdb5691ae4a4e5be60788e52859609784b5c/e2fyi_utils-0.3.0rc1-py3-none-any.whl", "yanked": false, "yanked_reason": null }, { "comment_text": "", "digests": { "md5": "dd5b8e6eef88693e2c2d7821d76129ae", "sha256": "d61fbb0a51c15961a06004e684b0829508944b2b6087cd6fa5139b1c423cc5fa" }, "downloads": -1, "filename": "e2fyi-utils-0.3.0rc1.tar.gz", "has_sig": false, "md5_digest": "dd5b8e6eef88693e2c2d7821d76129ae", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.6.1,<4", "size": 22591, "upload_time": "2020-10-01T17:18:16", "upload_time_iso_8601": "2020-10-01T17:18:16.141026Z", "url": "https://files.pythonhosted.org/packages/91/28/cb7439e61b0e2e00877c4a9f06d43b53e5493736311a7f97e0d4360ccdc4/e2fyi-utils-0.3.0rc1.tar.gz", "yanked": false, "yanked_reason": null } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "35588cf499713b30a288b96841365ec8", "sha256": "66dc0f4774e43a2c92331a54e850a2ae0a0aa16460c2848dc82d69a3bd8e202d" }, "downloads": -1, "filename": "e2fyi-utils-0.2.2.tar.gz", "has_sig": false, "md5_digest": "35588cf499713b30a288b96841365ec8", "packagetype": "sdist", "python_version": "source", "requires_python": ">=3.6", "size": 22887, "upload_time": "2019-12-27T08:58:46", "upload_time_iso_8601": "2019-12-27T08:58:46.095020Z", "url": "https://files.pythonhosted.org/packages/04/4c/9b4acc0f2f3ab54c009ac334319cbc7a720b53cd7d45b05d6073199fd081/e2fyi-utils-0.2.2.tar.gz", "yanked": false, "yanked_reason": null } ], "vulnerabilities": [] }