{
    "info": {
        "author": "Jonathan Rahn",
        "author_email": "jonathan.rahn@42digital.de",
        "bugtrack_url": null,
        "classifiers": [
            "Development Status :: 4 - Beta",
            "License :: OSI Approved :: MIT License",
            "Operating System :: OS Independent",
            "Programming Language :: Python :: 3",
            "Topic :: Database"
        ],
        "description": "[![Build Status](https://travis-ci.org/42DIGITAL/bqtools.svg?branch=master)](https://travis-ci.org/42DIGITAL/bqtools) [![PyPI version](https://badge.fury.io/py/bqtools.svg)](https://badge.fury.io/py/bqtools)\n\n# Python Tools for BigQuery\n\n### Why?\nFor data collection and data exploration, we like to work with [BigQuery](https://cloud.google.com/bigquery/). But we have not found a python library, to easily handle recurring tasks like adding new data (of potentially inconsistent schema) and schema migrations. So we took a couple of our solutions for those tasks and put them into this library.\n\n### What?\n`bqtools` provides a light-weight solution to explicit schema management with python-native types (unlike pandas dtype) and \nsome convenient type checking, inference and conversions. Table-objects created by `bqtools` can be read from BigQuery, stored locally, read from a local file and written to BigQuery. Table schemas can be changed and data can be added or modified.\n\n### Install\n```bash\npip install --upgrade bqtools\n```\n\n## Examples:\n### Create basic tables\n```python\nfrom fourtytwo import bqtools\n\nschema = [\n    {'name': 'number', 'field_type': 'INTEGER'},\n    {'name': 'text', 'field_type': 'STRING'},\n    {'name': 'struct', 'field_type':'RECORD', 'mode':'REPEATED', \n        'fields': [\n            {'name':'integer', 'field_type':'INTEGER'},\n            {'name':'text', 'field_type':'STRING'}\n        ]\n    }\n]\n# valid BigQuery types see: \n# https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types\n# geo and array are currently not/not fully supported\n\n# data = columns of lists\ntable = bqtools.BQTable(\n    schema=schema, \n    data=[[1, 2, 3, 4], ['a', 'b', 'c', 'd']]\n)\n\n# data = rows of dicts\ntable = bqtools.BQTable(\n    schema=schema, \n    data=[\n        {'number': 1, 'text': 'a'}, \n        {'number': 2, 'text': 'b'},\n        ...\n    ]\n)\n```\n\n### View data\n```python\nprint(table.data)       # list of all columns\nprint(table.rows(n=10)) # list of first n rows\n\n# convert to pandas.DataFrame\ndf = table.to_df()               \n# warning: pandas dtypes may be inconsistent \n# with BigQuery Schema field_types\n```\n\n### Append data\n```python\nrows = [{'number': 5, 'text': 'e'}]\ntable.append(rows)\n\nrow = [[6, 'f']]\ntable.append(rows)\n```\n\n### Load table from BigQuery\n```python\n# requires environment variable GOOGLE_APPLICATION_CREDENTIALS \n# or parameter credentials='path-to-credentials.json'\ntable = bqtools.read_bq(\n    table_ref='project_id.dataset_id.new_table_id', \n    limit=10,           # limit query rows\n    schema_only=False   # set True to only add data\n)\n```\n\n### Modify table schema\n```python\n# change column order and field_type\nnew_schema = [\n    {'name': 'text', 'field_type': 'STRING'},\n    {'name': 'number', 'field_type': 'FLOAT'},\n]\ntable.schema(new_schema)\n\n# change column names\ntable.rename(columns={'number': 'decimal'})\n```\n\n### Write table to BigQuery\n```python\n# requires environment variable GOOGLE_APPLICATION_CREDENTIALS\n# or parameter credentials='path-to-credentials.json'\ntable.to_bq(table_ref, mode='append')\n```\n\n### Persist tables locally\n```python\n# write to local file (compressed binary format)\ntable.save('local_table.bqt')\n\n# load from local file\ntable = bqtools.load('local_table.bqt')\n```\n\n\n",
        "description_content_type": "text/markdown",
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://github.com/42DIGITAL/bqtools",
        "keywords": "",
        "license": "",
        "maintainer": "",
        "maintainer_email": "",
        "name": "bqtools",
        "package_url": "https://pypi.org/project/bqtools/",
        "platform": "",
        "project_url": "https://pypi.org/project/bqtools/",
        "project_urls": {
            "Homepage": "https://github.com/42DIGITAL/bqtools"
        },
        "release_url": "https://pypi.org/project/bqtools/0.6.0/",
        "requires_dist": [
            "pandas (>=0.24.0)",
            "google-cloud-bigquery (>=1.10.0)",
            "python-dateutil (>=2.8.0)",
            "pytest ; extra == 'test'"
        ],
        "requires_python": "",
        "summary": "Python Tools for BigQuery",
        "version": "0.6.0",
        "yanked": false,
        "yanked_reason": null
    },
    "last_serial": 8753129,
    "releases": {
        "0.3.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "4ad16d5adb046651f7abf49b073c45e5",
                    "sha256": "0650b80b5c4d3855060f9d921a351afbb284ef5177c5fd47c321363f7630138e"
                },
                "downloads": -1,
                "filename": "bqtools-0.3.0-py3.7.egg",
                "has_sig": false,
                "md5_digest": "4ad16d5adb046651f7abf49b073c45e5",
                "packagetype": "bdist_egg",
                "python_version": "3.7",
                "requires_python": null,
                "size": 25981,
                "upload_time": "2019-04-02T11:41:58",
                "upload_time_iso_8601": "2019-04-02T11:41:58.459428Z",
                "url": "https://files.pythonhosted.org/packages/15/96/3dcaf28dc79570f516d938d89eaca7b1d5ff9844e94f595d2c22d282149a/bqtools-0.3.0-py3.7.egg",
                "yanked": false,
                "yanked_reason": null
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "33d986ec6c73a994d9f22bb1bdb02cd3",
                    "sha256": "d1a2a80cf869e3354b09befa1e19cda928565d70bd60a885d79bd9cd89ecebc3"
                },
                "downloads": -1,
                "filename": "bqtools-0.3.0-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "33d986ec6c73a994d9f22bb1bdb02cd3",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 11658,
                "upload_time": "2019-04-02T11:41:52",
                "upload_time_iso_8601": "2019-04-02T11:41:52.697523Z",
                "url": "https://files.pythonhosted.org/packages/94/d7/37626f29b43b24a31f88868c31485ffeedba8f506a044de53fc1f3a59ad9/bqtools-0.3.0-py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "312d36304d0f46885a3358292dbd6574",
                    "sha256": "bf44e06b4b770bfcbb53b25ef7fd1182c9de4571bdd5fc8ec96142ef612f0306"
                },
                "downloads": -1,
                "filename": "bqtools-0.3.0.tar.gz",
                "has_sig": false,
                "md5_digest": "312d36304d0f46885a3358292dbd6574",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 6470,
                "upload_time": "2019-04-02T11:41:59",
                "upload_time_iso_8601": "2019-04-02T11:41:59.767654Z",
                "url": "https://files.pythonhosted.org/packages/4b/70/df2009d97376a3c85c5979b81d0eb7043499052d4a6bdb8b6aa34fc737ef/bqtools-0.3.0.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "0.3.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "901101e2074af0b7fd766310486fefbf",
                    "sha256": "013b2548e12c4a5a2c0c1ea4563cc24e3a2450bb385b55ca0ab9074a44f4490c"
                },
                "downloads": -1,
                "filename": "bqtools-0.3.1-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "901101e2074af0b7fd766310486fefbf",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 11757,
                "upload_time": "2019-04-02T12:44:41",
                "upload_time_iso_8601": "2019-04-02T12:44:41.696093Z",
                "url": "https://files.pythonhosted.org/packages/b3/35/7641d178e2fc09065c41716dbcd8fa3ad07c493eb9db57ff2663b4557f85/bqtools-0.3.1-py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "2de9dfbb549c1571ed96f64918cb79e1",
                    "sha256": "89f43d2331be48fb3e2214f3c69e7a63b516d15053c08bb16a972025a61caf04"
                },
                "downloads": -1,
                "filename": "bqtools-0.3.1.tar.gz",
                "has_sig": false,
                "md5_digest": "2de9dfbb549c1571ed96f64918cb79e1",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 6530,
                "upload_time": "2019-04-02T12:44:45",
                "upload_time_iso_8601": "2019-04-02T12:44:45.775700Z",
                "url": "https://files.pythonhosted.org/packages/76/60/fc869ab6600fb2eaf2322ab56d58c97da5a97e08fc0ee284f14c756f89a4/bqtools-0.3.1.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "0.3.2": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "b7a1846e96688d447264c47a902fd51c",
                    "sha256": "1fe104671f1672e5944962e650a376132a9cca9589e63b0bc1c35ac2a4a2d7f6"
                },
                "downloads": -1,
                "filename": "bqtools-0.3.2-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "b7a1846e96688d447264c47a902fd51c",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 8031,
                "upload_time": "2019-04-26T07:47:42",
                "upload_time_iso_8601": "2019-04-26T07:47:42.850775Z",
                "url": "https://files.pythonhosted.org/packages/75/8c/03301245b1c24c7202ea53d227b94e4b5cf10d2b546b5596279fc39b8cde/bqtools-0.3.2-py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "a430cdf9ae5ebc21754aa0475c589cb2",
                    "sha256": "02c451056d4b4d71566168ab7ff9a44a7183693c3d613d38dba51146fac0ff90"
                },
                "downloads": -1,
                "filename": "bqtools-0.3.2.tar.gz",
                "has_sig": false,
                "md5_digest": "a430cdf9ae5ebc21754aa0475c589cb2",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 6541,
                "upload_time": "2019-04-26T07:42:32",
                "upload_time_iso_8601": "2019-04-26T07:42:32.670065Z",
                "url": "https://files.pythonhosted.org/packages/59/90/7273b3178fdbdb0b2ea0baa7511637a2f00c15bc0a46eb812b6afaa09656/bqtools-0.3.2.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "0.4.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "44124e4a1cefeaad4ea772d7d5e72bb8",
                    "sha256": "ed41283c33ede0708327ede6dbd5ddcf9c84f1fead78162799ee7c38baf167a8"
                },
                "downloads": -1,
                "filename": "bqtools-0.4.0-py3.7.egg",
                "has_sig": false,
                "md5_digest": "44124e4a1cefeaad4ea772d7d5e72bb8",
                "packagetype": "bdist_egg",
                "python_version": "3.7",
                "requires_python": null,
                "size": 17290,
                "upload_time": "2019-10-22T06:26:08",
                "upload_time_iso_8601": "2019-10-22T06:26:08.822780Z",
                "url": "https://files.pythonhosted.org/packages/bf/31/8b1a84d8b3770f886bf12e5886f62c924ee0b113013fcbf4ccfedaba982f/bqtools-0.4.0-py3.7.egg",
                "yanked": false,
                "yanked_reason": null
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "b5eda0b3fab5ebb53662dbc598739158",
                    "sha256": "7ac96874f4d1fb648beb48161c74fd537d2dfb58b95a9dc6d223e5430bdbb54a"
                },
                "downloads": -1,
                "filename": "bqtools-0.4.0-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "b5eda0b3fab5ebb53662dbc598739158",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 8200,
                "upload_time": "2019-07-09T09:31:18",
                "upload_time_iso_8601": "2019-07-09T09:31:18.508726Z",
                "url": "https://files.pythonhosted.org/packages/8b/17/d0c095fbe0ca7b782ba6e9126827b24e0c7252397f988473d836347b8535/bqtools-0.4.0-py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "0.5.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "57764143f5bbf9d841a4cc0caad58665",
                    "sha256": "b73005b10f638f2c783984f180ab7f258f34471a06bda9fa7ad48a280759cc07"
                },
                "downloads": -1,
                "filename": "bqtools-0.5.0-py3.7.egg",
                "has_sig": false,
                "md5_digest": "57764143f5bbf9d841a4cc0caad58665",
                "packagetype": "bdist_egg",
                "python_version": "3.7",
                "requires_python": null,
                "size": 17290,
                "upload_time": "2019-10-22T06:26:11",
                "upload_time_iso_8601": "2019-10-22T06:26:11.242788Z",
                "url": "https://files.pythonhosted.org/packages/cb/20/c427059422ea3ba8bb159697c2de185461e560c06cb00fd7fbac4b6f7164/bqtools-0.5.0-py3.7.egg",
                "yanked": false,
                "yanked_reason": null
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "9b8ff32ced7210908b24239713690edb",
                    "sha256": "1bd3d26fa0af65d806c4b5c1b6c29d5bea9b111e7b631d4a5e24c9e4d0c4ef3d"
                },
                "downloads": -1,
                "filename": "bqtools-0.5.0.tar.gz",
                "has_sig": false,
                "md5_digest": "9b8ff32ced7210908b24239713690edb",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 9883,
                "upload_time": "2019-10-22T06:26:13",
                "upload_time_iso_8601": "2019-10-22T06:26:13.962937Z",
                "url": "https://files.pythonhosted.org/packages/95/9e/a72b6b0fabeb76cb365a4bf01e5b652d40a984975b3cc0c5b00cb2a2e2b1/bqtools-0.5.0.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "0.6.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "212f7c2afb1ce5fdcff27c363587b44f",
                    "sha256": "5fa43bc8c48fb6440e05219fc8b83853dab6ab30e2be94c43984b2ae6b3e0ce2"
                },
                "downloads": -1,
                "filename": "bqtools-0.6.0-py3.7.egg",
                "has_sig": false,
                "md5_digest": "212f7c2afb1ce5fdcff27c363587b44f",
                "packagetype": "bdist_egg",
                "python_version": "3.7",
                "requires_python": null,
                "size": 17927,
                "upload_time": "2020-11-26T07:35:38",
                "upload_time_iso_8601": "2020-11-26T07:35:38.213060Z",
                "url": "https://files.pythonhosted.org/packages/d1/2a/76eb23c64a4ceb94ccb4037631a2e2b92bdff6043d4ba2bb308ba5a73ba8/bqtools-0.6.0-py3.7.egg",
                "yanked": false,
                "yanked_reason": null
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "bba4581de882df5bfaa797a35dfdead0",
                    "sha256": "2fa877a0eb58b89d7eb75be5fc4ae27b206b2cded4421d6099f330917dcf34ce"
                },
                "downloads": -1,
                "filename": "bqtools-0.6.0-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "bba4581de882df5bfaa797a35dfdead0",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 9201,
                "upload_time": "2020-11-26T07:35:37",
                "upload_time_iso_8601": "2020-11-26T07:35:37.065814Z",
                "url": "https://files.pythonhosted.org/packages/9c/d6/7cb4fa73cfbae8bb0266941cd11d88857511215a46e691bb6af21aa3071e/bqtools-0.6.0-py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "2d75793d1ee679b8a3646fbd2de80153",
                    "sha256": "8af25c798fa2c6a8338336c3d7f37bd38035fe5aa8bc61671a3c6afb57c84c6f"
                },
                "downloads": -1,
                "filename": "bqtools-0.6.0.tar.gz",
                "has_sig": false,
                "md5_digest": "2d75793d1ee679b8a3646fbd2de80153",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 10012,
                "upload_time": "2020-11-26T07:35:39",
                "upload_time_iso_8601": "2020-11-26T07:35:39.443456Z",
                "url": "https://files.pythonhosted.org/packages/78/56/53d2e7beca6d6223e5fd52dcdd095341b66ca6bb1ebe944d6502baf27c21/bqtools-0.6.0.tar.gz",
                "yanked": false,
                "yanked_reason": null
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "212f7c2afb1ce5fdcff27c363587b44f",
                "sha256": "5fa43bc8c48fb6440e05219fc8b83853dab6ab30e2be94c43984b2ae6b3e0ce2"
            },
            "downloads": -1,
            "filename": "bqtools-0.6.0-py3.7.egg",
            "has_sig": false,
            "md5_digest": "212f7c2afb1ce5fdcff27c363587b44f",
            "packagetype": "bdist_egg",
            "python_version": "3.7",
            "requires_python": null,
            "size": 17927,
            "upload_time": "2020-11-26T07:35:38",
            "upload_time_iso_8601": "2020-11-26T07:35:38.213060Z",
            "url": "https://files.pythonhosted.org/packages/d1/2a/76eb23c64a4ceb94ccb4037631a2e2b92bdff6043d4ba2bb308ba5a73ba8/bqtools-0.6.0-py3.7.egg",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": "",
            "digests": {
                "md5": "bba4581de882df5bfaa797a35dfdead0",
                "sha256": "2fa877a0eb58b89d7eb75be5fc4ae27b206b2cded4421d6099f330917dcf34ce"
            },
            "downloads": -1,
            "filename": "bqtools-0.6.0-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "bba4581de882df5bfaa797a35dfdead0",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": null,
            "size": 9201,
            "upload_time": "2020-11-26T07:35:37",
            "upload_time_iso_8601": "2020-11-26T07:35:37.065814Z",
            "url": "https://files.pythonhosted.org/packages/9c/d6/7cb4fa73cfbae8bb0266941cd11d88857511215a46e691bb6af21aa3071e/bqtools-0.6.0-py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": "",
            "digests": {
                "md5": "2d75793d1ee679b8a3646fbd2de80153",
                "sha256": "8af25c798fa2c6a8338336c3d7f37bd38035fe5aa8bc61671a3c6afb57c84c6f"
            },
            "downloads": -1,
            "filename": "bqtools-0.6.0.tar.gz",
            "has_sig": false,
            "md5_digest": "2d75793d1ee679b8a3646fbd2de80153",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 10012,
            "upload_time": "2020-11-26T07:35:39",
            "upload_time_iso_8601": "2020-11-26T07:35:39.443456Z",
            "url": "https://files.pythonhosted.org/packages/78/56/53d2e7beca6d6223e5fd52dcdd095341b66ca6bb1ebe944d6502baf27c21/bqtools-0.6.0.tar.gz",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "vulnerabilities": []
}