{ "info": { "author": "hipages Data Science Team", "author_email": "datascience@hipagesgroup.com.au", "bugtrack_url": null, "classifiers": [], "description": "# hip-data-tools\n\u00a9 Hipages Group Pty Ltd 2019\n\n[![PyPI version](https://badge.fury.io/py/hip-data-tools.svg)](https://pypi.org/project/hip-data-tools/#history) \n[![CircleCI](https://circleci.com/gh/hipagesgroup/data-tools/tree/master.svg?style=svg)](https://circleci.com/gh/hipagesgroup/data-tools/tree/master)\n\nCommon Python tools and utilities for data engineering, ETL, Exploration, etc. \nThe package is uploaded to PyPi for easy drop and use in various environmnets, such as (but not limited to):\n\n1. Running production workloads\n1. ML Training in Jupyter like notebooks\n1. Local machine for dev and exploration\n\n\n## Installation\nInstall from PyPi repo:\n```bash\npip3 install hip-data-tools\n```\n\nInstall from source\n```bash\npip3 install .\n```\n\n## Connect to aws \n\nYou will need to instantiate an AWS Connection:\n```python\nfrom hip_data_tools.authenticate import AwsConnection\n\nconn = AwsConnection(mode=\"assume_role\", settings={\"profile_name\": \"default\"})\n\n# OR if you want to connect using Env Vars:\nconn = AwsConnection(mode=\"standard_env_var\", settings={})\n\n# OR if you want custom set of env vars to connect\nconn = AwsConnection(mode=\"custom_env_var\", settings={\n \"aws_access_key_id_env_var\": \"aws_access_key_id\",\n \"aws_secret_access_key_env_var\": \"aws_secret_access_key\"\n })\n\n```\n\nUsing this connection to object you can use the aws utilities, for example aws Athena:\n```python\nfrom hip_data_tools.aws.athena import AthenaUtil\n\nau = AthenaUtil(database=\"default\", conn=conn, output_bucket=\"example\", output_key=\"tmp/scratch/\")\nresult = au.run_query(\"SELECT * FROM temp limit 10\", return_result=True)\nprint(result)\n```\n\n\n", "description_content_type": "text/markdown", "docs_url": null, "download_url": "", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "https://github.com/hipagesgroup/data-tools", "keywords": "", "license": "osl-3.0", "maintainer": "", "maintainer_email": "", "name": "hip-data-tools", "package_url": "https://pypi.org/project/hip-data-tools/", "platform": "", "project_url": "https://pypi.org/project/hip-data-tools/", "project_urls": { "Homepage": "https://github.com/hipagesgroup/data-tools" }, "release_url": "https://pypi.org/project/hip-data-tools/1.7.1/", "requires_dist": [ "boto3 (==1.9.216)", "joblib (==0.13.2)", "pandas (==0.25.1)", "GitPython (==3.0.0)", "confluent-kafka (==1.1.0)" ], "requires_python": "~=3.6", "summary": "Common utility functions for data engineering usecases", "version": "1.7.1" }, "last_serial": 5980913, "releases": { "0.0.0": [ { "comment_text": "", "digests": { "md5": "12b5abe88af585fae3bc73da0af51839", "sha256": "acc7d27a2a5f39a26049509cbe983442d25c604538226683727d14914d57a25f" }, "downloads": -1, "filename": "hip_data_tools-0.0.0-py3-none-any.whl", "has_sig": false, "md5_digest": "12b5abe88af585fae3bc73da0af51839", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": "~=3.6", "size": 10309, "upload_time": "2019-08-21T05:11:05", "url": "https://files.pythonhosted.org/packages/20/fa/b567c352df51d84a53cc08b0d463dc3734fc4704ece4f5d1507780f6f8a1/hip_data_tools-0.0.0-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "f8f94040a7229a2c3f8a8c6951e9b8b7", "sha256": "0b8775b3b7ebbdab8daf370dd11abb2b1231782cf44cb254099c29ae2a2a1c3c" }, "downloads": -1, "filename": "hip_data_tools-0.0.0.tar.gz", "has_sig": false, "md5_digest": "f8f94040a7229a2c3f8a8c6951e9b8b7", "packagetype": "sdist", "python_version": "source", "requires_python": "~=3.6", "size": 5871, "upload_time": "2019-08-21T05:11:07", "url": "https://files.pythonhosted.org/packages/d2/ae/7a824f50a37e0453d5f945fb7fb6ef11c0d096d36bcbc82ccf059c4270e5/hip_data_tools-0.0.0.tar.gz" } ], "1.1.0": [ { "comment_text": "", "digests": { "md5": "a0ac03b31d8208b3d963ffd12cbad85f", "sha256": "730950c6cd1f9abb92c8151a59005a80ee0036b67eb0f0b3e5868d9ee510d2c5" }, "downloads": -1, "filename": "hip_data_tools-1.1.0-py3-none-any.whl", "has_sig": false, "md5_digest": "a0ac03b31d8208b3d963ffd12cbad85f", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": "~=3.6", "size": 10463, "upload_time": "2019-08-19T23:55:27", "url": "https://files.pythonhosted.org/packages/e3/4a/3f9b66bda5a25f8dec16cf4af98a27d71b96572fc30261928027c9ae5680/hip_data_tools-1.1.0-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "4028885dfa63b79c25ca21ea500e856e", "sha256": "0137bf9b2c1867a6f8d257a55c05f8e1c055e3fcc4dc2a66a25f3eb0174b8882" }, "downloads": -1, "filename": "hip_data_tools-1.1.0.tar.gz", "has_sig": false, "md5_digest": "4028885dfa63b79c25ca21ea500e856e", "packagetype": "sdist", "python_version": "source", "requires_python": "~=3.6", "size": 5955, "upload_time": "2019-08-19T23:55:29", "url": "https://files.pythonhosted.org/packages/df/df/56d7e76e3e952e0454a84574338d5f643fcfa51ea7480c9a5d900b837a69/hip_data_tools-1.1.0.tar.gz" } ], "1.2.0": [ { "comment_text": "", "digests": { "md5": "8456cf8d53f15a712db9e34de2f87f74", "sha256": "0c49dac233807342729272db099e55f3c2cb7180630a7106d0e93606dfc9a96f" }, "downloads": -1, "filename": "hip_data_tools-1.2.0-py3-none-any.whl", "has_sig": false, "md5_digest": "8456cf8d53f15a712db9e34de2f87f74", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": "~=3.6", "size": 10536, "upload_time": "2019-08-21T06:57:01", "url": "https://files.pythonhosted.org/packages/56/16/bf0c7a97bb1cf12d464fe9814c37f437dd24c6e7aedec19ba394a6f23cb6/hip_data_tools-1.2.0-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "ed9a029a2abc0f62fee497ba26f93f42", "sha256": "7962d8f7c44fa3b299f5a735d1c785e52bd5e03cd1aa341447f9da85c08e1210" }, "downloads": -1, "filename": "hip_data_tools-1.2.0.tar.gz", "has_sig": false, "md5_digest": "ed9a029a2abc0f62fee497ba26f93f42", "packagetype": "sdist", "python_version": "source", "requires_python": "~=3.6", "size": 14029, "upload_time": "2019-08-21T06:57:03", "url": "https://files.pythonhosted.org/packages/31/5d/25a00c97cf3c291529ddc17ac57e3716977a2362e5b741398837aa12ecba/hip_data_tools-1.2.0.tar.gz" } ], "1.2.1": [ { "comment_text": "", "digests": { "md5": "f1304abe11b5aba429c1a7f6db42a052", "sha256": "681f6089d8a030b03bd137d96023ca826e8039e721be4c61c4c5be7e6c15b5dc" }, "downloads": -1, "filename": "hip_data_tools-1.2.1-py3-none-any.whl", "has_sig": false, "md5_digest": "f1304abe11b5aba429c1a7f6db42a052", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": "~=3.6", "size": 11345, "upload_time": "2019-08-21T23:41:16", "url": "https://files.pythonhosted.org/packages/8a/62/8830649481c9e58e8121222f0253caf63e277c79428f0f34f7861ac8fbf9/hip_data_tools-1.2.1-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "82e2a74e37253cff8e0b9eecc4042c50", "sha256": "b7764136aee00f7a1b66d0e983dcf307034f98351c0c614d3ee97563c3fb3189" }, "downloads": -1, "filename": "hip_data_tools-1.2.1.tar.gz", "has_sig": false, "md5_digest": "82e2a74e37253cff8e0b9eecc4042c50", "packagetype": "sdist", "python_version": "source", "requires_python": "~=3.6", "size": 10729, "upload_time": "2019-08-21T23:41:17", "url": "https://files.pythonhosted.org/packages/c9/6d/409a5a8f831e0b65a0a2b41bebd8228b87b7f15e0dcb28f3058b7dd12d2b/hip_data_tools-1.2.1.tar.gz" } ], "1.2.1.dev1": [ { "comment_text": "", "digests": { "md5": "dd8a69ef715371a8caaf7533a7c9d11c", "sha256": "aa2d19ee8b16dbda578c6f2972d14d8a8d925b7d8c18efd89f42b5ff3aabdad5" }, "downloads": -1, "filename": "hip_data_tools-1.2.1.dev1-py3-none-any.whl", "has_sig": false, "md5_digest": "dd8a69ef715371a8caaf7533a7c9d11c", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": "~=3.6", "size": 10634, "upload_time": "2019-08-21T07:33:46", "url": "https://files.pythonhosted.org/packages/42/66/8990e5e3f4b290a2979dce613a8fef68ff511901bd5c303dbdc15f6cdb87/hip_data_tools-1.2.1.dev1-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "f2a7e77a709e22d752c3d42b0b82515c", "sha256": "e5a8f645e3dfc62ee471333eb7352d575911946ffa71c46d95fd414e5ad3beab" }, "downloads": -1, "filename": "hip_data_tools-1.2.1.dev1.tar.gz", "has_sig": false, "md5_digest": "f2a7e77a709e22d752c3d42b0b82515c", "packagetype": "sdist", "python_version": "source", "requires_python": "~=3.6", "size": 13880, "upload_time": "2019-08-21T07:33:47", "url": "https://files.pythonhosted.org/packages/14/58/d5c7706cd9b7e9450ea8650d7a8405b6420622ac3a2198e3e01afc62f056/hip_data_tools-1.2.1.dev1.tar.gz" } ], "1.3.0": [ { "comment_text": "", "digests": { "md5": "d183f65b04e5af12e1b08c5c5c661f7f", "sha256": "bf0c572cb70fe4baf4f6509c3fe5ec5c471aaed78baafd8860426eaf3371d602" }, "downloads": -1, "filename": "hip_data_tools-1.3.0-py3-none-any.whl", "has_sig": false, "md5_digest": "d183f65b04e5af12e1b08c5c5c661f7f", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": "~=3.6", "size": 12385, "upload_time": "2019-08-27T06:35:36", "url": "https://files.pythonhosted.org/packages/39/01/23721f8057c0db92099b5a7bc6e9b4aad67660a27054914daf6bf3bb7b9d/hip_data_tools-1.3.0-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "96eb4bf435bd98dc1f31c9378a963913", "sha256": "5ac877ad52130703861835aee1ec382419f04e5bf3b944b126b76731c487f50e" }, "downloads": -1, "filename": "hip_data_tools-1.3.0.tar.gz", "has_sig": false, "md5_digest": "96eb4bf435bd98dc1f31c9378a963913", "packagetype": "sdist", "python_version": "source", "requires_python": "~=3.6", "size": 11251, "upload_time": "2019-08-27T06:35:37", "url": "https://files.pythonhosted.org/packages/c4/f9/37b07a0b97c747217a0c62c1f650f6f8bbeeb919e730bf973df9d0ae8e20/hip_data_tools-1.3.0.tar.gz" } ], "1.3.1": [ { "comment_text": "", "digests": { "md5": "08c4ce4415a86fe38e4826a929def77b", "sha256": "2fdd7e1dba71e2a1ea67f8be28bec53598c65886840bfcec1bc5317202ca8f5e" }, "downloads": -1, "filename": "hip_data_tools-1.3.1-py3-none-any.whl", "has_sig": false, "md5_digest": "08c4ce4415a86fe38e4826a929def77b", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": "~=3.6", "size": 12503, "upload_time": "2019-08-30T04:11:43", "url": "https://files.pythonhosted.org/packages/a1/19/c74441eb07fc25de99bd8fed2e1ebd5784dd678486d832e357a54a8fae1b/hip_data_tools-1.3.1-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "3b24618631b2925b7d9d183877260368", "sha256": "a725ddec3830a8d073a7464c7641c4e396e062764820f9c00b91e6896b2369fc" }, "downloads": -1, "filename": "hip_data_tools-1.3.1.tar.gz", "has_sig": false, "md5_digest": "3b24618631b2925b7d9d183877260368", "packagetype": "sdist", "python_version": "source", "requires_python": "~=3.6", "size": 10904, "upload_time": "2019-08-30T04:11:45", "url": "https://files.pythonhosted.org/packages/db/91/cdf6307baea5890d40afcae6ebcf41b6eee283b106af39cf6136ed6f60bd/hip_data_tools-1.3.1.tar.gz" } ], "1.4.0": [ { "comment_text": "", "digests": { "md5": "25b08e96088d2e79fc4b3e0824bac65e", "sha256": "33ec39af3e3f96c2b29d6fc87c565ae352b74a080902ef09b667c3e530798ce6" }, "downloads": -1, "filename": "hip_data_tools-1.4.0-py3-none-any.whl", "has_sig": false, "md5_digest": "25b08e96088d2e79fc4b3e0824bac65e", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": "~=3.6", "size": 13551, "upload_time": "2019-09-25T03:34:50", "url": "https://files.pythonhosted.org/packages/7e/ad/0bd9ee7506bb511e517a3d0ece92afda148b66704faf8d4a65bacddf9301/hip_data_tools-1.4.0-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "8b49114e865b7cacc58748b8e0be3c20", "sha256": "edcafe79cfdfb10647fd08cd59c18780b33ce20884cfcf6686bbe5f88c27860d" }, "downloads": -1, "filename": "hip_data_tools-1.4.0.tar.gz", "has_sig": false, "md5_digest": "8b49114e865b7cacc58748b8e0be3c20", "packagetype": "sdist", "python_version": "source", "requires_python": "~=3.6", "size": 11892, "upload_time": "2019-09-25T03:34:51", "url": "https://files.pythonhosted.org/packages/34/31/89c9372f18ed9db052db392bbc3fee7a2b283d7e05e580ab65fa81c2d980/hip_data_tools-1.4.0.tar.gz" } ], "1.5.0": [ { "comment_text": "", "digests": { "md5": "e9cf13fcc069eac16423ded951136390", "sha256": "647bba4dc58f71ec4e3a87fb80158951ec9f451b52927cb432f4b64a6d02e465" }, "downloads": -1, "filename": "hip_data_tools-1.5.0-py3-none-any.whl", "has_sig": false, "md5_digest": "e9cf13fcc069eac16423ded951136390", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": "~=3.6", "size": 19760, "upload_time": "2019-09-26T06:43:05", "url": "https://files.pythonhosted.org/packages/ea/47/c73d78b4eb29cc4b5e43de1335a3c993cee69258c51dbde6aa438de8caa0/hip_data_tools-1.5.0-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "453f985542eb9940fd9c910d48e39985", "sha256": "ef4814fe26599405bff47cd0e661df20968313dfc7c2cfdccd9ef9a6594532b7" }, "downloads": -1, "filename": "hip_data_tools-1.5.0.tar.gz", "has_sig": false, "md5_digest": "453f985542eb9940fd9c910d48e39985", "packagetype": "sdist", "python_version": "source", "requires_python": "~=3.6", "size": 18628, "upload_time": "2019-09-26T06:43:07", "url": "https://files.pythonhosted.org/packages/80/48/b521d07e036b3c898cf9a266bc54f0d1c3af56565347c6d5041aeae39a59/hip_data_tools-1.5.0.tar.gz" } ], "1.6.0": [ { "comment_text": "", "digests": { "md5": "3d23304d8a8dd412058b44cb7568cb9e", "sha256": "c71cf944138a465e8aac7f01a94b953bd34d9bab699ae19289cd909259e01d48" }, "downloads": -1, "filename": "hip_data_tools-1.6.0-py3-none-any.whl", "has_sig": false, "md5_digest": "3d23304d8a8dd412058b44cb7568cb9e", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": "~=3.6", "size": 23888, "upload_time": "2019-10-11T00:51:35", "url": "https://files.pythonhosted.org/packages/82/2a/e0f61360e4bf086ab53c549e54ecd7f901b09048c4a29af4a99b4413cdf5/hip_data_tools-1.6.0-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "3f8fa1148ac4ee3f9fc2e150d7497790", "sha256": "122dc476e75141c3a5b4730137d682d3f25fa4de252030a04619c71879fc4931" }, "downloads": -1, "filename": "hip_data_tools-1.6.0.tar.gz", "has_sig": false, "md5_digest": "3f8fa1148ac4ee3f9fc2e150d7497790", "packagetype": "sdist", "python_version": "source", "requires_python": "~=3.6", "size": 22082, "upload_time": "2019-10-11T00:51:36", "url": "https://files.pythonhosted.org/packages/b6/e3/b4beceb12f4febd9f67b5a837a56ee36b0732444067c7acf49779481e12c/hip_data_tools-1.6.0.tar.gz" } ], "1.7.0": [ { "comment_text": "", "digests": { "md5": "7f3e5b0d60ef7dc76f7ce86f85cb7db4", "sha256": "09d63820eb13e4368be4b10ee617b78f1cb8404163da9f23ec8806ac6115399c" }, "downloads": -1, "filename": "hip_data_tools-1.7.0-py3-none-any.whl", "has_sig": false, "md5_digest": "7f3e5b0d60ef7dc76f7ce86f85cb7db4", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": "~=3.6", "size": 24954, "upload_time": "2019-10-14T00:23:12", "url": "https://files.pythonhosted.org/packages/97/f5/1097eb13cf43dfdb954aaf60f1960615078ca0aa8e4b400c3bf7e0ad4c81/hip_data_tools-1.7.0-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "f23b01e231199eab4d28943dc0ff33d9", "sha256": "d5defe5b878b2114854de54021e21980af782f9afe3ddd52250475560051a017" }, "downloads": -1, "filename": "hip_data_tools-1.7.0.tar.gz", "has_sig": false, "md5_digest": "f23b01e231199eab4d28943dc0ff33d9", "packagetype": "sdist", "python_version": "source", "requires_python": "~=3.6", "size": 23082, "upload_time": "2019-10-14T00:23:14", "url": "https://files.pythonhosted.org/packages/3d/ed/3688781a166aaed9b1a6bad9cf86d8a06bee324d10c37cf01d467d2bd615/hip_data_tools-1.7.0.tar.gz" } ], "1.7.1": [ { "comment_text": "", "digests": { "md5": "0ec9a25b9bf5be5c18decaed66ec8f03", "sha256": "58458324f974e97d1487b045177b8c32b2a83cc7152db8914190d657398de8a1" }, "downloads": -1, "filename": "hip_data_tools-1.7.1-py3-none-any.whl", "has_sig": false, "md5_digest": "0ec9a25b9bf5be5c18decaed66ec8f03", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": "~=3.6", "size": 24928, "upload_time": "2019-10-16T02:54:30", "url": "https://files.pythonhosted.org/packages/cc/38/3120f6d4ed6f71dc2917a85460a0f1dc991934e3a268ebc7cc759df08604/hip_data_tools-1.7.1-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "c75bd279a1b68865bd204a80bf5c4823", "sha256": "50f5b6d6a0b7377921d6f0f3927e0476875e30c12c58fb716220ac269343c471" }, "downloads": -1, "filename": "hip_data_tools-1.7.1.tar.gz", "has_sig": false, "md5_digest": "c75bd279a1b68865bd204a80bf5c4823", "packagetype": "sdist", "python_version": "source", "requires_python": "~=3.6", "size": 22882, "upload_time": "2019-10-16T02:54:32", "url": "https://files.pythonhosted.org/packages/ce/fc/070b3b026bcba8c43a99fa474ee8337d8700dfa8852dc1dc42fa50ab4f6f/hip_data_tools-1.7.1.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "0ec9a25b9bf5be5c18decaed66ec8f03", "sha256": "58458324f974e97d1487b045177b8c32b2a83cc7152db8914190d657398de8a1" }, "downloads": -1, "filename": "hip_data_tools-1.7.1-py3-none-any.whl", "has_sig": false, "md5_digest": "0ec9a25b9bf5be5c18decaed66ec8f03", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": "~=3.6", "size": 24928, "upload_time": "2019-10-16T02:54:30", "url": "https://files.pythonhosted.org/packages/cc/38/3120f6d4ed6f71dc2917a85460a0f1dc991934e3a268ebc7cc759df08604/hip_data_tools-1.7.1-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "c75bd279a1b68865bd204a80bf5c4823", "sha256": "50f5b6d6a0b7377921d6f0f3927e0476875e30c12c58fb716220ac269343c471" }, "downloads": -1, "filename": "hip_data_tools-1.7.1.tar.gz", "has_sig": false, "md5_digest": "c75bd279a1b68865bd204a80bf5c4823", "packagetype": "sdist", "python_version": "source", "requires_python": "~=3.6", "size": 22882, "upload_time": "2019-10-16T02:54:32", "url": "https://files.pythonhosted.org/packages/ce/fc/070b3b026bcba8c43a99fa474ee8337d8700dfa8852dc1dc42fa50ab4f6f/hip_data_tools-1.7.1.tar.gz" } ] }