{
    "info": {
        "author": "Geographica",
        "author_email": "hello@geographica.com",
        "bugtrack_url": null,
        "classifiers": [
            "License :: OSI Approved :: MIT License",
            "Programming Language :: Python :: 3",
            "Programming Language :: Python :: 3.6",
            "Programming Language :: Python :: 3.7"
        ],
        "description": "# GeoLibs-Dator\nDator, a data extractor (ETL as a library), that uses Pandas' DataFrames as in memory temporal storage.\n\n### Features\n| Source | Extract | Transform | Load |\n| --- | --- | --- | --- |\n| BigQuery | Y | Y |  |\n| CARTO | Y | Y | Y* |\n| CSV | Y |  | Y |\n| Pandas |  | Y |  |\n| PostgreSQL | Y | Y | Y |\n\n_* Note:_ We are waiting for the append feature on [CARTOframes](https://github.com/CartoDB/cartoframes), because the one we are using is a _\u00f1apa_.\n\n### Configuration\nCreate a `config.yml` file using the `config.example.yml` one as guide. You can find in that one all the possible ETL cases.\n\nIf you are using BigQuery in your ETL process, you need to add a `GOOGLE_APPLICATION_CREDENTIALS` environment variable with the path to your Google Cloud's `credentials.json` file.\n\nYou can test them with the `example.py` file.\n\n#### Example\n\n*dator_config.yml*\n\n```yml\ndatastorages:\n  bigquery_input:\n    type: bigquery\n    data:\n      query: SELECT * FROM `dataset.table` WHERE updated_at >= '2019-05-04T00:00:00Z' AND updated_at < '2019-06-01T00:00:00Z';\n\n  carto_input:\n    type: carto\n    credentials:\n      url: https://domain.com/user/user/\n      api_key: api_key\n    data:\n      table: table\n\n  postgresql_input:\n    credentials:\n      ...\n    data:\n      query: SELECT * FROM somewhere;\n      types:\n        - name: timeinstant\n          type: datetime\n        - name: fillinglevel\n          type: float\n        - name: temperature\n          type: int\n        - name: category\n          type: str\n\n  carto_output:\n    type: carto\n    credentials:\n      url: https://domain.com/user/user/\n      api_key: api_key\n    data:\n      table: table\n      append: false\n\ntransformations:\n  bigquery_agg:\n    type: bigquery\n    time:\n      field: updated_at\n      start: \"2019-05-02T00:00:00Z\"  # As string or YAML will parse them as DateTimes\n      finish: \"2019-05-03T00:00:00Z\"\n      step: 5 MINUTE\n    aggregate:\n      by:\n        - container_id\n        - updated_at\n      fields:\n        field_0: avg\n        field_1: max\n\nextract: bigquery_input\ntransform: bigquery_agg\nload: carto_output\n```\n\n### How to use\n\nThis package is designed to accomplish ETL operations in three steps:\n\n#### Extract\n\nThe extract method is a default method, this means although this method can be overwritten, by default, it must work via config.\n\n(This section under construction)\n\n#### Transform\n\n(This section under construction)\n\n#### Load\n\nThe load method is a default method, this means although this method can be overwritten, by default, it must work via config. It can receive 2 parameters, the Pandas dataframe and a dictionary with extra info.\n\n#### Example\n\n*app.py*\n\n```python\nfrom dator import Dator\n\ndator = Dator('/usr/src/app/dator_config.yml')\ndf = dator.extract()\ndf = dator.transform(df)\ndator.load(df)\n```\n\n*app.py* with extra info\n\n```python\nfrom dator import Dator\n\ndef upsert_method:\n  pass\n\ndator = Dator('/usr/src/app/dator_config.yml')\ndf = dator.extract()\ndf = dator.transform(df)\ndator.load(df, {'method': upsert_method})\n```\n\n### TODOs\n- Better doc.\n- Tests.\n",
        "description_content_type": "text/markdown",
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://github.com/GeographicaGS/GeoLibs-Dator",
        "keywords": "",
        "license": "MIT",
        "maintainer": "Geographica",
        "maintainer_email": "hello@geographica.com",
        "name": "geolibs-dator",
        "package_url": "https://pypi.org/project/geolibs-dator/",
        "platform": "",
        "project_url": "https://pypi.org/project/geolibs-dator/",
        "project_urls": {
            "Homepage": "https://github.com/GeographicaGS/GeoLibs-Dator"
        },
        "release_url": "https://pypi.org/project/geolibs-dator/0.0.7/",
        "requires_dist": [
            "carto (>=1.4,<2.0)",
            "cartoframes (>=0.9.2,<0.10.0)",
            "google-cloud-bigquery (>=1.11,<2.0)",
            "marshmallow (>=2.19,<3.0)",
            "pandas (>=0.24.2,<0.25.0)",
            "psycopg2-binary (>=2.8.2,<3.0.0)",
            "PyYAML (>=5.1,<6.0)",
            "sqlalchemy (>=1.1.15,<2.0.0)",
            "numpy (>=1.17,<2.0)"
        ],
        "requires_python": ">=3.6,<4.0",
        "summary": "GeoLibs Dator - A data extractor",
        "version": "0.0.7"
    },
    "last_serial": 5953700,
    "releases": {
        "0.0.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "d3c9aaeb932f0f562ce792d3618d0bd5",
                    "sha256": "464b22ae87e4bb670914c3cb2a911feff29b5fa7122b0effb893d9a6bc2cc552"
                },
                "downloads": -1,
                "filename": "geolibs_dator-0.0.1-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "d3c9aaeb932f0f562ce792d3618d0bd5",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": ">=3.6,<4.0",
                "size": 10882,
                "upload_time": "2019-05-16T14:39:36",
                "url": "https://files.pythonhosted.org/packages/59/ba/edb8c0c71a5b0d39f7f720efb80858789514574d6463e637f887b1ad614e/geolibs_dator-0.0.1-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "6b634030f7b260272d52914f52b42f33",
                    "sha256": "be28e2b3b15fe990613a34c1044a43642e0b3385616717cb8b23033ce7df8846"
                },
                "downloads": -1,
                "filename": "geolibs-dator-0.0.1.tar.gz",
                "has_sig": false,
                "md5_digest": "6b634030f7b260272d52914f52b42f33",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=3.6,<4.0",
                "size": 7369,
                "upload_time": "2019-05-16T14:39:34",
                "url": "https://files.pythonhosted.org/packages/58/0f/c83670ca6913bf07bc8ece76a3eee3f2a20520731bee632032f95ac6d5d6/geolibs-dator-0.0.1.tar.gz"
            }
        ],
        "0.0.2": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "46c720f43d0c6bbd0877768224eb2303",
                    "sha256": "3232c5e0e9d697b2de455e3e249818c8f0ddaebe52e5d3b3aff3f1d1553fcebf"
                },
                "downloads": -1,
                "filename": "geolibs_dator-0.0.2-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "46c720f43d0c6bbd0877768224eb2303",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": ">=3.6,<4.0",
                "size": 10883,
                "upload_time": "2019-05-16T16:15:08",
                "url": "https://files.pythonhosted.org/packages/d1/73/fff5c9fb850b6ac2c8841c5d6041406de9eb48a9f469df28aa9a27ac88dc/geolibs_dator-0.0.2-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "7909e5af7aa75b9ac46f78889bb54d31",
                    "sha256": "183990cf2264db160afbd42caa763275aff6b459cea63332ae7ecdc94f612556"
                },
                "downloads": -1,
                "filename": "geolibs-dator-0.0.2.tar.gz",
                "has_sig": false,
                "md5_digest": "7909e5af7aa75b9ac46f78889bb54d31",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=3.6,<4.0",
                "size": 7383,
                "upload_time": "2019-05-16T16:15:06",
                "url": "https://files.pythonhosted.org/packages/c7/c3/52fa06ea966ea69415032c4b552d1eb7aa0cf3792fda65cd0ca1ab875abf/geolibs-dator-0.0.2.tar.gz"
            }
        ],
        "0.0.3": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "bbd8a13ad204a16e3d30cbd57021f57e",
                    "sha256": "3238491207a70aaa8a85eb73f3927faa0e56566351f8728e814cd1bbf3a3f167"
                },
                "downloads": -1,
                "filename": "geolibs_dator-0.0.3-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "bbd8a13ad204a16e3d30cbd57021f57e",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": ">=3.6,<4.0",
                "size": 10710,
                "upload_time": "2019-05-16T16:46:46",
                "url": "https://files.pythonhosted.org/packages/92/f7/564fc70bec6934fe9723a3c3c636e63406a8d845dc7f652b63134dd9e8c6/geolibs_dator-0.0.3-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "046a39c30a2c501929927c1158424adb",
                    "sha256": "d0ae123d4c858ea3ed3cef7a2598144817eacce443ac7c7370d83e8fe3808d82"
                },
                "downloads": -1,
                "filename": "geolibs-dator-0.0.3.tar.gz",
                "has_sig": false,
                "md5_digest": "046a39c30a2c501929927c1158424adb",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=3.6,<4.0",
                "size": 7473,
                "upload_time": "2019-05-16T16:46:45",
                "url": "https://files.pythonhosted.org/packages/48/cd/864b3f023c03d1c6c6e50c5f345695728c87902a4869d9480d1847a6bdcc/geolibs-dator-0.0.3.tar.gz"
            }
        ],
        "0.0.4": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "d873e2bad5055c5d5175677039ceada3",
                    "sha256": "f7db9e61e8a35f6cf25e8a4148b9352bed4fe0f247aeec4a4e18487af269d4ab"
                },
                "downloads": -1,
                "filename": "geolibs_dator-0.0.4-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "d873e2bad5055c5d5175677039ceada3",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": ">=3.6,<4.0",
                "size": 11920,
                "upload_time": "2019-07-16T15:36:20",
                "url": "https://files.pythonhosted.org/packages/6f/79/bffae70ceb06e592e4ef14c8dad0ea5fb0ab3381bb95b5dbd48e65d6c640/geolibs_dator-0.0.4-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "bde4199c56f064a6b26c51eb99e62e31",
                    "sha256": "d93cf2c51d9cf3575fd118a0529d08e9f08b3fc37cfb9c6be8f80021e82e7f94"
                },
                "downloads": -1,
                "filename": "geolibs-dator-0.0.4.tar.gz",
                "has_sig": false,
                "md5_digest": "bde4199c56f064a6b26c51eb99e62e31",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=3.6,<4.0",
                "size": 9370,
                "upload_time": "2019-07-16T15:36:18",
                "url": "https://files.pythonhosted.org/packages/25/9f/4811a3de960044c55d3cef2b4578aeb9c2fd91b9a3644fcf193a82af81e9/geolibs-dator-0.0.4.tar.gz"
            }
        ],
        "0.0.5": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "d0569c60d258fa59218aa6cd4e1d5668",
                    "sha256": "d3ec8bc65818006d2a3bb371d1154672945785e1f4bd89267ed02ed06b3590eb"
                },
                "downloads": -1,
                "filename": "geolibs_dator-0.0.5-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "d0569c60d258fa59218aa6cd4e1d5668",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": ">=3.6,<4.0",
                "size": 11924,
                "upload_time": "2019-07-17T07:06:02",
                "url": "https://files.pythonhosted.org/packages/79/22/49f33d16e8f26f5fd577e4fb27b7270d945363cc7ab8e4630f9ded910cd1/geolibs_dator-0.0.5-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "74ae56e794745539f958f9e075eafd12",
                    "sha256": "923f3092dc8f96d8c48bc290470140093ffc042fa27315e3ed9f9b459e97c5eb"
                },
                "downloads": -1,
                "filename": "geolibs-dator-0.0.5.tar.gz",
                "has_sig": false,
                "md5_digest": "74ae56e794745539f958f9e075eafd12",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=3.6,<4.0",
                "size": 9372,
                "upload_time": "2019-07-17T07:06:00",
                "url": "https://files.pythonhosted.org/packages/de/c3/46fa6cf2b5802df505fb1d62cb25cee41d5b9a187ded4150821ec0cf1bbe/geolibs-dator-0.0.5.tar.gz"
            }
        ],
        "0.0.6": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "af9e905d936ed54047aa2348a0c33cb5",
                    "sha256": "a19a08b5308cb5f5181da34c042712dd535160781cc66caca85abc0fd0098f1c"
                },
                "downloads": -1,
                "filename": "geolibs_dator-0.0.6-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "af9e905d936ed54047aa2348a0c33cb5",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": ">=3.6,<4.0",
                "size": 12329,
                "upload_time": "2019-08-26T12:29:00",
                "url": "https://files.pythonhosted.org/packages/be/b5/bbcef6f19de13fcd1afc302f5675fcad2f72c6d1879eb58aac85b10bf033/geolibs_dator-0.0.6-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "d8c06d69bc560d3c8993c074b6e543c9",
                    "sha256": "08416d9a22c2eff7b9ca161769d1d9f05549e792b76b7fc6b91de8925e516a39"
                },
                "downloads": -1,
                "filename": "geolibs-dator-0.0.6.tar.gz",
                "has_sig": false,
                "md5_digest": "d8c06d69bc560d3c8993c074b6e543c9",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=3.6,<4.0",
                "size": 9759,
                "upload_time": "2019-08-26T12:28:59",
                "url": "https://files.pythonhosted.org/packages/b4/e0/d89fdf1c75a756e2add12a155c2b3577a69691f2e2d9f078d3503d31a481/geolibs-dator-0.0.6.tar.gz"
            }
        ],
        "0.0.7": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "cc0a9232355e6c8e854620e97cca0404",
                    "sha256": "e0311502b41a5526d92243fa68dac7ef4e0577c7e7fb9d6edb1ea04ea483fbb4"
                },
                "downloads": -1,
                "filename": "geolibs_dator-0.0.7-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "cc0a9232355e6c8e854620e97cca0404",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": ">=3.6,<4.0",
                "size": 12357,
                "upload_time": "2019-10-10T08:59:40",
                "url": "https://files.pythonhosted.org/packages/70/ac/4676adc34ca9b396143bfb3c89b85696fa6f2485599dae432ee299dcd837/geolibs_dator-0.0.7-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "89518dbd0f889cf13827b8f261756e23",
                    "sha256": "3bedbea6607d6dfcff5a88a6604abf3713a27b7bab51e34ff5cad307848ec1f5"
                },
                "downloads": -1,
                "filename": "geolibs-dator-0.0.7.tar.gz",
                "has_sig": false,
                "md5_digest": "89518dbd0f889cf13827b8f261756e23",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=3.6,<4.0",
                "size": 9781,
                "upload_time": "2019-10-10T08:59:38",
                "url": "https://files.pythonhosted.org/packages/d4/bd/b69d924a4fefa9b47bbab7ed1448094c834024e478297fab1c53456e80ca/geolibs-dator-0.0.7.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "cc0a9232355e6c8e854620e97cca0404",
                "sha256": "e0311502b41a5526d92243fa68dac7ef4e0577c7e7fb9d6edb1ea04ea483fbb4"
            },
            "downloads": -1,
            "filename": "geolibs_dator-0.0.7-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "cc0a9232355e6c8e854620e97cca0404",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": ">=3.6,<4.0",
            "size": 12357,
            "upload_time": "2019-10-10T08:59:40",
            "url": "https://files.pythonhosted.org/packages/70/ac/4676adc34ca9b396143bfb3c89b85696fa6f2485599dae432ee299dcd837/geolibs_dator-0.0.7-py3-none-any.whl"
        },
        {
            "comment_text": "",
            "digests": {
                "md5": "89518dbd0f889cf13827b8f261756e23",
                "sha256": "3bedbea6607d6dfcff5a88a6604abf3713a27b7bab51e34ff5cad307848ec1f5"
            },
            "downloads": -1,
            "filename": "geolibs-dator-0.0.7.tar.gz",
            "has_sig": false,
            "md5_digest": "89518dbd0f889cf13827b8f261756e23",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": ">=3.6,<4.0",
            "size": 9781,
            "upload_time": "2019-10-10T08:59:38",
            "url": "https://files.pythonhosted.org/packages/d4/bd/b69d924a4fefa9b47bbab7ed1448094c834024e478297fab1c53456e80ca/geolibs-dator-0.0.7.tar.gz"
        }
    ]
}