{
    "info": {
        "author": "BGS Informatics",
        "author_email": "jostev@bgs.ac.uk",
        "bugtrack_url": null,
        "classifiers": [
            "Development Status :: 4 - Beta",
            "Intended Audience :: Developers",
            "Intended Audience :: Information Technology",
            "Intended Audience :: Science/Research",
            "License :: OSI Approved :: GNU Lesser General Public License v3 or later (LGPLv3+)",
            "Natural Language :: English",
            "Operating System :: OS Independent",
            "Programming Language :: Python :: 3",
            "Topic :: Database",
            "Topic :: Scientific/Engineering :: GIS"
        ],
        "description": "# etlhelper\n\n> etlhelper is a Python library to simplify data transfer into and out of databases.\n\n## Overview\n\n`etlhelper` makes it easy to run a SQL query via Python and return the results.\nIt is built upon the [DBAPI2\nspecification](https://www.python.org/dev/peps/pep-0249/) and takes care of\nimporting drivers, formatting connection strings and cursor management.\nThis reduces the amount of boilerplate code required to query a relational\ndatabase with Python.\n\n### Features\n\n+ `setup_oracle_client` script installs Oracle Instant Client on Linux systems\n+ `DbParams` objects provide consistent way to connect to different database types (currently Oracle, PostgreSQL, SQLite and MS SQL Server)\n+ `get_rows`, `iter_rows`, `fetchone` and other functions for querying database\n+ `execute`, `executemany`, and `load` functions to insert data\n+ `copy_rows` and `copy_table_rows` to transfer data from one database to another\n+ `on_error` function to process rows that fail to insert\n+ Support for parameterised queries and in-flight transformation of data\n+ Output results as namedtuple or dictionary\n+ Timestamped log messages for tracking long-running data transfers\n+ Helpful error messages display the failed query SQL\n\nThese tools can create easy-to-understand, lightweight, versionable and testable Extract-Transform-Load (ETL) workflows.\n`etlhelper` is not a tool for coordinating ETL jobs (use [Apache Airflow](https://airflow.apache.org)), for\nconverting GIS data formats (use [ogr2ogr](https://gdal.org/programs/ogr2ogr.html) or [fiona](https://pypi.org/project/Fiona/)), for translating between SQL dialects or providing Object Relation Mapping (use [SQLAlchemy](https://www.sqlalchemy.org/)).\nHowever, it can be used in conjunction with each of these.\n\n![screencast](https://github.com/BritishGeologicalSurvey/etlhelper/blob/main/docs/screencast.gif?raw=true)\n\nThe documentation below explains how the main features are used.\nSee the individual function docstrings for full details of parameters and\noptions.\n\nFor a high level introduction to `etlhelper`, see the FOSS4GUK 2019 presentation _Open Source Spatial ETL with Python and Apache Airflow_: [video](https://www.youtube.com/watch?v=12rzUW4ps74&feature=youtu.be&t=6238) (20 mins),\n[slides](https://volcan01010.github.io/FOSS4G2019-talk).\n\n\n### Documentation\n\n + [Installation](#installation)\n + [Connect to databases](#connect-to-databases)\n + [Transfer data](#transfer-data)\n + [Recipes](#recipes)\n + [Development](#development)\n + [References](#references)\n\n\n## Installation\n\n### Python packages\n\n```bash\npip install etlhelper\n```\n\nDatabase driver packages are not included by default and should be specified in\nsquare brackets.\nOptions are `oracle` (installs cx_Oracle), `mssql` (installs pyodbc) and `postgres` (installs psycopg2).\nMultiple values can be separated by commas.\n\n```\npip install etlhelper[oracle,postgres]\n```\n\nThe `sqlite3` driver is included within Python's Standard Library.\n\n\n### Database driver dependencies\n\nSome database drivers have additional dependencies.\nOn Linux, these can be installed via the system package manager.\n\ncx_Oracle (for Oracle):\n\n+ `sudo apt install libaio1` (Debian/Ubuntu) or `sudo dnf install libaio`\n  (CentOS, RHEL, Fedora)\n\npyodbc (for MS SQL Server):\n\n+ Follow instructions on [Microsoft SQL Docs website](https://docs.microsoft.com/en-us/sql/connect/odbc/linux-mac/installing-the-microsoft-odbc-driver-for-sql-server?view=sql-server-2017)\n\n\n### Oracle Instant Client\n\nOracle Instant Client libraries are required to connect to Oracle databases.\nOn Linux, `etlhelper` provides a script to download and unzip them from the [Oracle\nwebsite](https://www.oracle.com/database/technologies/instant-client/linux-x86-64-downloads.html).\nOnce the drivers are installed, their location must be added to LD_LIBRARY_PATH\nenvironment variable before they can be used.  `setup_oracle_client` writes\na file that can then be \"sourced\" to do this for the current shell.  These two steps\ncan be executed in a single command as:\n\n```bash\nsource $(setup_oracle_client)\n```\n\nThis command must be run in each new shell session.\nSee `setup_oracle_client --help` for further command line flags, including\nspecifying an alternative URL or filesystem path for the zipfile location.\n\n\n## Connect to databases\n\n### DbParams\n\nDatabase connection details are defined by `DbParams` objects.\nConnections are made via their `connect` functions (see below).\n`DbParams` objects are created as follows or from environment variables using the\n`from_environment()` function.\nThe class initialisation function checks that the correct attributes have been provided for\na given `dbtype`.\n\n```python\nfrom etlhelper import DbParams\n\nORACLEDB = DbParams(dbtype='ORACLE', host=\"localhost\", port=1521,\n                    dbname=\"mydata\", user=\"oracle_user\")\n\nPOSTGRESDB = DbParams(dbtype='PG', host=\"localhost\", port=5432,\n                      dbname=\"mydata\", user=\"postgres_user\")\n\nSQLITEDB = DbParams(dbtype='SQLITE', filename='/path/to/file.db')\n\nMSSQLDB = DbParams(dbtype='MSSQL', host=\"localhost\", port=1433,\n                   dbname=\"mydata\", user=\"mssql_user\",\n                   odbc_driver=\"ODBC Driver 17 for SQL Server\")\n```\n\nDbParams objects have a function to check if a given database can be reached\nover the network.  This does not require a username or password.\n\n```python\nif not ORACLEDB.is_reachable():\n    raise ETLHelperError(\"Network problems\")\n```\n\nOther methods/properties are `get_connection_string`,\n`get_sqlalchemy_connection_string`, `paramstyle` and `copy`.\nSee function docstrings for details.\n\n###  `connect` function\n\nThe `DbParams.connect()` function returns a DBAPI2 connection as provided by the\nunderlying driver.\nUsing context-manager syntax as below ensures that the connection is closed\nafter use.\n\n```python\nwith SQLITEDB.connect() as conn1:\n    with POSTGRESDB.connect('PGPASSWORD') as conn2:\n        do_something()\n```\n\nA standalone `connect` function provides backwards-compatibility with\nprevious releases of `etlhelper`:\n\n```python\nfrom etlhelper import connect\nconn3 = connect(ORACLEDB, 'ORACLE_PASSWORD')\n```\n\nBoth versions accept additional keyword arguments that are passed to the `connect`\nfunction of the underlying driver.  For example, the following sets the character\nencoding used by cx_Oracle to ensure that values are returned as UTF-8:\n\n```python\nconn4 = connect(ORACLEDB, 'ORACLE_PASSWORD', encoding=\"UTF-8\", nencoding=\"UTF8\")\n```\n\nThe above is a solution when special characters are scrambled in the returned data.\n\n#### Disabling fast_executemany for SQL Server and other pyODBC connections\n\nBy default an `etlhelper` pyODBC connection uses a cursor with its\n`fast_executemany` attribute set to `True`. This setting improves the\nperformance of the `executemany` when performing bulk inserts to a\nSQL Server database. However, this overides the default behaviour\nof pyODBC and there are some limitations in doing this. Importantly,\nit is only recommended for applications that use Microsoft's ODBC Driver for\nSQL Server. See [pyODBC fast_executemany](https://github.com/mkleehammer/pyodbc/wiki/Features-beyond-the-DB-API#fast_executemany).\n\nUsing `fast_executemany` may raise a `MemoryError` if query involves columns of types\n`TEXT` and `NTEXT`, which are now deprecated.\nUnder these circumstances, `etlhelper` falls back on `fast_executemany` being set to\n`False` and produces a warning output. See [Inserting into SQL server with\nfast_executemany results in MemoryError](https://github.com/mkleehammer/pyodbc/issues/547).\n\nIf required, the `fast_executemany` attribute can be set to `False` via the\n`connect` function:\n\n```python\nconn5 = connect(MSSQLDB, 'MSSQL_PASSWORD', fast_executemany=False)\n```\n\nThis keyword argument is used by `etlhelper`, any further keyword arguments are\npassed to the `connect` function of the underlying driver.\n\n### Passwords\n\nDatabase passwords must be specified via an environment variable.\nThis reduces the temptation to store them within scripts.\nThis can be done on the command line via:\n\n+ `export ORACLE_PASSWORD=some-secret-password` on Linux\n+ `set ORACLE_PASSWORD=some-secret-password` on Windows\n\nOr in a Python terminal via:\n\n```python\nimport os\nos.environ['ORACLE_PASSWORD'] = 'some-secret-password'\n```\n\nNo password is required for SQLite databases.\n\n\n## Transfer data\n\n### Get rows\n\nThe `get_rows` function returns a list of named tuples containing data as\nnative Python objects.\n\n```python\nfrom my_databases import ORACLEDB\nfrom etlhelper import get_rows\n\nsql = \"SELECT * FROM src\"\n\nwith ORACLEDB.connect(\"ORA_PASSWORD\") as conn:\n    get_rows(sql, conn)\n```\n\nreturns\n\n```\n[Row(id=1, value=1.234, simple_text='text', utf8_text='\u00d6\u00e6\u00b0\\nz',\n     day=datetime.date(2018, 12, 7),\n     date_time=datetime.datetime(2018, 12, 7, 13, 1, 59)),\n Row(id=2, value=2.234, simple_text='text', utf8_text='\u00d6\u00e6\u00b0\\nz',\n     day=datetime.date(2018, 12, 8),\n     date_time=datetime.datetime(2018, 12, 8, 13, 1, 59)),\n Row(id=3, value=2.234, simple_text='text', utf8_text='\u00d6\u00e6\u00b0\\nz',\n     day=datetime.date(2018, 12, 9),\n     date_time=datetime.datetime(2018, 12, 9, 13, 1, 59))]\n```\n\nData are accessible via index (`row[4]`) or name (`row.day`).\n\nOther functions are provided to select data.  `fetchone`, `fetchmany` and\n`fetchall` are equivalent to the cursor methods specified in the DBAPI v2.0.\n`dump_rows` passes each row to a function (default is `print`).\n\n\n#### iter_rows\n\nIt is recommended to use `iter_rows` for looping over large result sets.  It\nis a generator function that only yields data as requested.  This ensures that\nthe data are not all loaded into memory at once.\n\n```\nwith ORACLEDB.connect(\"ORA_PASSWORD\") as conn:\n    for row in iter_rows(sql, conn):\n        do_something(row)\n```\n\n\n#### Parameters\n\nVariables can be inserted into queries by passing them as parameters.\nThese \"bind variables\" are sanitised by the underlying drivers to prevent [SQL\ninjection attacks](https://xkcd.com/327/).\nThe required [paramstyle](https://www.python.org/dev/peps/pep-0249/#paramstyle)\ncan be checked with `MY_DB.paramstyle`.\nA tuple is used for positional placeholders, or a dictionary for named\nplaceholders.\n\n```python\nselect_sql = \"SELECT * FROM src WHERE id = :id\"\n\nwith ORACLEDB.connect(\"ORA_PASSWORD\") as conn:\n    get_rows(sql, conn, parameters={'id': 1})\n```\n\n#### Row factories\n\nRow factories control the output format of returned rows.\nTo return each row as a dictionary, use the following:\n\n```python\nfrom etlhelper import get_rows\nfrom etlhelper.row_factories import dict_row_factory\n\nsql = \"SELECT * FROM my_table\"\n\nwith ORACLEDB.connect('ORACLE_PASSWORD') as conn:\n    for row in get_rows(sql, conn, row_factory=dict_row_factory):\n        print(row['id'])\n```\n\nThe `dict_row_factory` is useful when data are to be serialised to JSON/YAML,\nor when modifying individual fields with a `transform` function (see below).\nWhen using `dict_row_factory` with `copy_rows`, it is necessary to use named\nplaceholders for the INSERT query (e.g. `%(id)s` instead of `%s` for\nPostgreSQL, `:id` instead of `:1` for Oracle).\n\n\n#### Transform\n\nThe `transform` parameter allows passing of a function to transform the data\nbefore returning it.\nThe function must take a list of rows and return a list of modified rows.\nSee `copy_rows` for more details.\n\n\n#### Chunk size\n\nAll data extraction functions use `iter_chunks` behind the scenes.\nThis reads rows from the database in chunks to prevent them all being loaded\ninto memory at once.\nThe default `chunk_size` is 5000 and this can be set via keyword argument.\n\n\n### Insert rows\n\n`execute` can be used to insert a single row or to execute other single\nstatements e.g. \"CREATE TABLE ...\".\nThe `executemany` function is used to insert multiple rows of data.\nLarge datasets are broken into chunks and inserted in batches to reduce the\nnumber of queries.\nA tuple with counts of rows processed and failed is returned.\n\n```python\nfrom etlhelper import executemany\n\nrows = [(1, 'value'), (2, 'another value')]\ninsert_sql = \"INSERT INTO some_table (col1, col2) VALUES (%s, %s)\"\n\nwith POSTGRESDB.connect('PGPASSWORD') as conn:\n    processed, failed = executemany(insert_sql, conn, rows, chunk_size=1000)\n```\n\nThe `chunk_size` default is 5,000 and it can be set with a keyword argument.\nThe `commit_chunks` flag defaults to `True`.\nThis ensures that an error during a large data transfer doesn't require all the\nrecords to be sent again.\nSome work may be required to determine which records remain to be sent.\nSetting `commit_chunks` to `False` will roll back the entire transfer in case\nof an error.\n\nSome database engines can return autogenerated values (e.g. primary key IDs)\nafter INSERT statements.\nTo capture these values, use the `fetchone` method to execute the SQL command\ninstead.\n\n```python\ninsert_sql = \"INSERT INTO my_table (message) VALUES ('hello') RETURNING id\"\n\nwith POSTGRESDB.connect('PGPASSWORD') as conn:\n    result = fetchone(insert_sql, conn)\n\nprint(result.id)\n```\n\nThe `load` function is similar to `executemany` except that it autogenerates\nan insert query based on the data provided. It uses `generate_insert_query`\nto remove the need to explicitly write the query for simple cases. By\ncalling this function manually, users can create a base insert query that can\nbe extended with clauses such as `ON CONFLICT DO NOTHING`.\n\n\n#### Handling insert errors\n\nThe default behaviour of `etlhelper` is to raise an exception on the first\nerror and abort the transfer.\nSometimes it is desirable to ignore the errors and to do something else with\nthe failed rows.\nThe `on_error` parameter allows a function to be passed that is applied to the\nfailed rows of each chunk.\nThe input is a list of (row, exception) tuples.\n\nDifferent examples are given here.  The simplest approach is to collect all the\nerrors into a list to process at the end.\n\n```python\nerrors = []\nexecutemany(sql, conn, rows, on_error=errors.extend)\n\nif errors:\n    do_something()\n```\n\nErrors can be logged to the `etlhelper` logger.\n\n```python\nfrom etlhelper import logger\n\ndef log_errors(failed_rows):\n    for row, exception in failed_rows:\n        logger.error(exception)\n\nexecutemany(sql, conn, rows, on_error=log_errors)\n```\n\nThe IDs of failed rows can be written to a file.\n\n```python\ndef write_bad_ids(failed_rows):\n    with open('bad_ids.txt', 'at') as out_file:\n        for row, exception in failed_rows:\n            out_file.write(f\"{row.id}\\n\")\n\nexecutemany(sql, conn, rows, on_error=write_bad_ids)\n```\n\n`executemany`, `load`, `copy_rows` and `copy_table_rows` can all take an\n`on_error` parameter.  They each return a tuple containing the number of rows\nprocessed and the number of rows that failed.\n\n### Copy table rows\n\n`copy_table_rows` provides a simple way to copy all the data from one table to\nanother.\nIt can take a `transform` function in case some modification of the data, e.g.\nchange of case of column names, is required.\n\n```python\nfrom my_databases import POSTGRESDB, ORACLEDB\nfrom etlhelper import copy_table_rows\n\nwith ORACLEDB.connect(\"ORA_PASSWORD\") as src_conn:\n    with POSTGRESDB.connect(\"PG_PASSWORD\") as dest_conn:\n\tcopy_table_rows('my_table', src_conn, dest_conn)\n```\n\nThe `chunk_size`, `commit_chunks` and `on_error` parameters can all be set.\nA tuple with counts of rows processed and failed is returned.\n\n\n### Combining `iter_rows` with `load`\n\nFor extra control selecting the data to be transferred, `iter_rows` can be\ncombined with `load`.\n\n```python\nfrom my_databases import POSTGRESDB, ORACLEDB\nfrom etlhelper import iter_rows, load\n\nselect_sql = \"\"\"\n    SELECT id, name, value FROM my_table\n    WHERE value > :min_value\n\"\"\"\n\nwith ORACLEDB.connect(\"ORA_PASSWORD\") as src_conn:\n    with POSTGRESDB.connect(\"PG_PASSWORD\") as dest_conn:\n        rows = iter_rows(select_sql, src_conn, parameters={'min_value': 99})\n\tload('my_table', dest_conn, rows)\n```\n\n### Copy rows\n\nCustomising both queries gives the greatest control on data selection and loading.\n`copy_rows` takes the results from a SELECT query and applies them as parameters\nto an INSERT query.\nThe source and destination tables must already exist.\nFor example, here we use GROUP BY and WHERE in the SELECT query and insert extra\nauto-generated values via the INSERT query.\n\n```python\nfrom my_databases import POSTGRESDB, ORACLEDB\nfrom etlhelper import copy_rows\n\nselect_sql = \"\"\"\n    SELECT\n      customer_id,\n      SUM (amount) AS total_amount\n    FROM payment\n    WHERE id > 1000\n    GROUP BY customer_id\n\"\"\"\ninsert_sql = \"\"\"\n    INSERT INTO dest (customer_id, total_amount, loaded_by, load_time)\n    VALUES (%s, %s, current_user, now())\n\"\"\"\n\nwith ORACLEDB.connect(\"ORA_PASSWORD\") as src_conn:\n    with POSTGRESDB.connect(\"PG_PASSWORD\") as dest_conn:\n        copy_rows(select_sql, src_conn, insert_sql, dest_conn)\n```\n\n`parameters` can be passed to the SELECT query as before and the\n`commit_chunks`, `chunk_size` and `on_error` options can be set.\n\nA tuple of rows processed and failed is returned.\n\n\n### Transform\n\nData can be transformed in-flight by applying a transform function.  This is\nany Python callable (e.g. function) that takes an iterator (e.g. list) and returns\nanother iterator.\nTransform functions are applied to data as they are read from the database and\ncan be used with `get_rows`-type methods and with `copy_rows`.\n\nThe following code demonstrates that the returned chunk can have a different number\nof rows, and be of different length, to the input.\nWhen used with `copy_rows`, the INSERT query must contain the correct placeholders for the\ntransform result.\nExtra data can result from a calculation, a call to a webservice or another database.\n\n```python\nimport random\n\ndef my_transform(chunk):\n    # Append random integer (1-10), filter if <5.\n\n    new_chunk = []\n    for row in chunk:  # each row is a namedtuple\n        extra_value = random.randrange(10)\n        if extra_value >= 5:\n            new_chunk.append((*row, extra_value))\n\n    return new_chunk\n\ncopy_rows(select_sql, src_conn, insert_sql, dest_conn,\n          transform=my_transform)\n```\n\nIt can be easier to modify individual columns when using the\n`dict_row_factory` (see above).\n\n```python\nfrom etlhelper.row_factories import dict_row_factory\n\ndef my_transform(chunk):\n    # Add prefix to id, remove newlines, set lower case email addresses\n\n    new_chunk = []\n    for row in chunk:  # each row is a dictionary\n        row['id'] += 1000\n        row['description'] = row['description'].replace('\\n', ' ')\n        row['email'] = row['email'].lower()\n        new_chunk.append(row)\n\n    return new_chunk\n\nget_rows(select_sql, src_conn, row_factory=dict_row_factory,\n         transform=my_transform)\n```\n\nThe `iter_chunks` and `iter_rows` functions that are used internally return\ngenerators.  Each chunk or row of data is only accessed when it is required.\nThe transform function can also be written to return a generator instead of\na list.  Data transformation can then be performed via [memory-efficient\niterator-chains](https://dbader.org/blog/python-iterator-chains).\n\n\n## Recipes\n\nThe following recipes demonstrate how `etlhelper` can be used.\n\n\n### Debug SQL and monitor progress with logging\n\nETL Helper provides a custom logging handler.\nTime-stamped messages indicating the number of rows processed can be enabled by\nsetting the log level to INFO.\nSetting the level to DEBUG provides information on the query that was run,\nexample data and the database connection.\n\n```python\nimport logging\nfrom etlhelper import logger\n\nlogger.setLevel(logging.INFO)\n```\n\nOutput from a call to `copy_rows` will look like:\n\n```\n2019-10-07 15:06:22,411 iter_chunks: Fetching rows\n2019-10-07 15:06:22,413 executemany: 1 rows processed\n2019-10-07 15:06:22,416 executemany: 2 rows processed\n2019-10-07 15:06:22,419 executemany: 3 rows processed\n2019-10-07 15:06:22,420 iter_chunks: 3 rows returned\n2019-10-07 15:06:22,420 executemany: 3 rows processed in total\n```\n\nNote: errors on database connections output messages that include login\ncredentials in clear text.\n\n\n### Database to database copy ETL script template\n\nThe following is a template for an ETL script.\nIt copies copy all the sensor readings from the previous day from an Oracle\nsource to PostgreSQL destination.\n\n```python\n# copy_readings.py\n\nimport datetime as dt\nfrom etl_helper import copy_rows\nfrom my_databases import ORACLEDB, POSTGRESDB\n\nCREATE_SQL = dedent(\"\"\"\n    CREATE TABLE IF NOT EXISTS sensordata.readings\n    (\n      sensor_data_id bigint PRIMARY KEY,\n      measure_id bigint,\n      time_stamp timestamp without time zone,\n      meas_value double precision\n    )\n    \"\"\").strip()\n\nDELETE_SQL = dedent(\"\"\"\n    DELETE FROM sensordata.readings\n    WHERE time_stamp BETWEEN %(startdate)s AND %(enddate)s\n    \"\"\").strip()\n\nSELECT_SQL = dedent(\"\"\"\n    SELECT id, measure_id, time_stamp, reading\n    FROM sensor_data\n    WHERE time_stamp BETWEEN :startdate AND :enddate\n    ORDER BY time_stamp\n    \"\"\").strip()\n\nINSERT_SQL = dedent(\"\"\"\n    INSERT INTO sensordata.readings (sensor_data_id, measure_id, time_stamp,\n      meas_value)\n    VALUES (%s, %s, %s, %s)\n    \"\"\").strip()\n\n\ndef copy_readings(startdate, enddate):\n    params = {'startdate': startdate, 'enddate': enddate}\n\n    with ORACLEDB.connect(\"ORA_PASSWORD\") as src_conn:\n        with POSTGRESDB.connect(\"PG_PASSWORD\") as dest_conn:\n            execute(CREATE_SQL dest_conn)\n            execute(DELETE_SQL, dest_conn, parameters=params)\n            copy_rows(SELECT_SQL, src_conn,\n                      INSERT_SQL, dest_conn,\n                      parameters=params)\n\n\nif __name__ == \"__main__\":\n    # Copy data from 00:00:00 yesterday to 00:00:00 today\n    today = dt.combine(dt.date.today(), dt.time.min)\n    yesterday = today - dt.timedelta(1)\n\n    copy_readings(yesterday, today)\n```\n\nIt is valuable to create [idempotent](https://stackoverflow.com/questions/1077412/what-is-an-idempotent-operation) scripts to ensure that they can be rerun without problems.\nIn this example, the \"CREATE TABLE IF NOT EXISTS\" command can be called repeatedly.\nThe DELETE_SQL command clears existing data prior to insertion to prevent duplicate key errors.\nSQL syntax such as \"INSERT OR UPDATE\", \"UPSERT\" or \"INSERT ... ON CONFLICT\" may be more efficient, but the the exact commands depend on the target database type.\n\n\n### Calling ETL Helper scripts from Apache Airflow\n\nThe following is an [Apache Airflow\nDAG](https://airflow.apache.org/docs/stable/concepts.html) that uses the `copy_readings` function\ndefined in the script above.\nThe Airflow scheduler will create tasks for each day since 1 August 2019 and\ncall `copy_readings` with the appropriate start and end times.\n\n```python\n# readings_dag.py\n\nimport datetime as dt\nfrom airflow import DAG\nfrom airflow.operators.python_operator import PythonOperator\nimport copy_readings\n\n\ndef copy_readings_with_args(**kwargs):\n    # Set arguments for copy_readings from context\n    start = kwargs.get('prev_execution_date')\n    end = kwargs.get('execution_date')\n    copy_readings.copy_readings(start, end)\n\ndag = DAG('readings',\n          schedule_interval=dt.timedelta(days=1),\n          start_date=dt.datetime(2019, 8, 1),\n          catchup=True)\n\nt1 = PythonOperator(\n    task_id='copy_readings',\n    python_callable=copy_readings_with_args,\n    provide_context=True,\n    dag=dag)\n```\n\n\n### Spatial ETL\n\nNo specific drivers are required for spatial data if they are transferred as\nWell Known Text.\n\n```python\nselect_sql_oracle = \"\"\"\n    SELECT\n      id,\n      SDO_UTIL.TO_WKTGEOMETRY(geom)\n    FROM src\n    \"\"\"\n\ninsert_sql_postgis = \"\"\"\n    INSERT INTO dest (id, geom) VALUES (\n      %s,\n      ST_Transform(ST_GeomFromText(%s, 4326), 27700)\n    )\n    \"\"\"\n```\n\nOther spatial operations e.g. coordinate transforms, intersections and\nbuffering can be carried out in the SQL.\nTransform functions can manipulate geometries using the [Shapely](https://pypi.org/project/Shapely/) library.\n\n\n### Database to API / NoSQL copy ETL script template\n\n`etlhelper` can be combined with Python's\n[aiohttp](https://docs.aiohttp.org/en/stable/) library to create an ETL\nfor posting data from a database into an HTTP API.\nThe API could be a NoSQL document store (e.g. ElasticSearch, Cassandra) or some other\nweb service.\n\nThis example transfers data from Oracle to ElasticSearch.\nIt uses `iter_chunks` to fetch data from the database without loading it all into\nmemory at once.\nA custom transform function creates a dictionary structure from each row\nof data.\nThis is \"dumped\" into JSON and posted to the API via `aiohttp`.\n\n`aiohttp` allows the records in each chunk to be posted to the API\nasynchronously.\nThe API is often the bottleneck in such pipelines and we have seen significant\nspeed increases (e.g. 10x) using asynchronous transfer as opposed to posting\nrecords in series.\n\n\n```python\n# copy_sensors_async.py\nimport asyncio\nimport datetime as dt\nimport json\nimport logging\n\nimport aiohttp\nfrom etlhelper import iter_chunks\n\nfrom db import ORACLE_DB\n\nlogger = logging.getLogger(\"copy_sensors_async\")\n\nSELECT_SENSORS = \"\"\"\n    SELECT CODE, DESCRIPTION\n    FROM BGS.DIC_SEN_SENSOR\n    WHERE date_updated BETWEEN :startdate AND :enddate\n    ORDER BY date_updated\n    \"\"\"\nBASE_URL = \"http://localhost:9200/\"\nHEADERS = {'Content-Type': 'application/json'}\n\n\ndef copy_sensors(startdate, enddate):\n    \"\"\"Read sensors from Oracle and post to REST API.\"\"\"\n    logger.info(\"Copying sensors with timestamps from %s to %s\",\n                startdate.isoformat(), enddate.isoformat())\n    row_count = 0\n\n    with ORACLE_DB.connect('ORACLE_PASSWORD') as conn:\n        # chunks is a generator that yields lists of dictionaries\n        chunks = iter_chunks(SELECT_SENSORS, conn,\n                             parameters={\"startdate\": startdate,\n                                         \"enddate\": enddate},\n                             transform=transform_sensors)\n\n        for chunk in chunks:\n            result = asyncio.run(post_chunk(chunk))\n            row_count += len(result)\n            logger.info(\"%s items transferred\", row_count)\n\n    logger.info(\"Transfer complete\")\n\n\ndef transform_sensors(chunk):\n    \"\"\"Transform rows to dictionaries suitable for converting to JSON.\"\"\"\n    new_chunk = []\n\n    for row in chunk:\n        new_row = {\n            'sample_code': row.CODE,\n            'description': row.DESCRIPTION,\n            'metadata': {\n                'source': 'ORACLE_DB',  # fixed value\n                'transferred_at': dt.datetime.now().isoformat()  # dynamic value\n                }\n            }\n        logger.debug(new_row)\n        new_chunk.append(new_row)\n\n    return new_chunk\n\n\nasync def post_chunk(chunk):\n    \"\"\"Post multiple items to API asynchronously.\"\"\"\n    async with aiohttp.ClientSession() as session:\n        # Build list of tasks\n        tasks = []\n        for item in chunk:\n            tasks.append(post_one(item, session))\n\n        # Process tasks in parallel.  An exception in any will be raised.\n        result = await asyncio.gather(*tasks)\n\n    return result\n\n\nasync def post_one(item, session):\n    \"\"\"Post a single item to API using existing aiohttp Session.\"\"\"\n    # Post the item\n    response = await session.post(BASE_URL + 'sensors/_doc', headers=HEADERS,\n                                  data=json.dumps(item))\n\n    # Log responses before throwing errors because error info is not included\n    # in generated Exceptions and so cannot otherwise be seen for debugging.\n    if response.status >= 400:\n        response_text = await response.text()\n        logger.error('The following item failed: %s\\nError message:\\n(%s)',\n                     item, response_text)\n        await response.raise_for_status()\n\n    return response.status\n\n\nif __name__ == \"__main__\":\n    # Configure logging\n    handler = logging.StreamHandler()\n    formatter = logging.Formatter('%(asctime)s %(levelname)-8s %(message)s')\n    handler.setFormatter(formatter)\n    logger.setLevel(logging.INFO)\n    logger.addHandler(handler)\n\n    # Copy data from 1 January 2000 to 00:00:00 today\n    today = dt.datetime.combine(dt.date.today(), dt.time.min)\n    copy_sensors(dt.datetime(2000, 1, 1), today)\n```\n\nIn this example, failed rows will fail the whole job.  Removing the\n`raise_for_status()` call will let them just be logged instead.\n\n\n### CSV load script template\n\nThe following script is an example of using the `load` function to import data\nfrom a CSV file into a database.\nIt shows how a `transform` function can perform common parsing tasks such as\nrenaming columns and converting timestamps into datetime objects.\nThe database has a `CHECK` constraint that rejects any rows with an ID\ndivisible by 1000.\nAn example `on_error` function prints the IDs of rows that fail to insert.\n\n```python\n\"\"\"\nScript to create database and load observations data from csv file. It also\ndemonstrates how an `on_error` function can handle failed rows.\n\nGenerate observations.csv with:\ncurl 'https://sensors.bgs.ac.uk/FROST-Server/v1.1/Observations?$select=@iot.id,result,phenomenonTime&$top=20000&$resultFormat=csv' -o observations.csv\n\"\"\"\nimport csv\nimport datetime as dt\nfrom typing import Iterable, List, Tuple\n\nfrom etlhelper import execute, load, DbParams\n\n\ndef load_observations(csv_file, conn):\n    \"\"\"Load observations from csv_file to db_file.\"\"\"\n    # Drop table (helps with repeated test runs!)\n    drop_table_sql = \"\"\"\n        DROP TABLE IF EXISTS observations\n        \"\"\"\n    execute(drop_table_sql, conn)\n\n    # Create table (reject ids with no remainder when divided by 1000)\n    create_table_sql = \"\"\"\n        CREATE TABLE IF NOT EXISTS observations (\n          id INTEGER PRIMARY KEY CHECK (id % 1000),\n          time TIMESTAMP,\n          result FLOAT\n          )\"\"\"\n    execute(create_table_sql, conn)\n\n    # Load data\n    with open(csv_file, 'rt') as f:\n        reader = csv.DictReader(f)\n        load('observations', conn, transform(reader), on_error=on_error)\n\n\n# The on_error function is called after each chunk with all the failed rows\ndef on_error(failed_rows: List[Tuple[dict, Exception]]) -> None:\n    \"\"\"Print the IDs of failed rows\"\"\"\n    rows, exceptions = zip(*failed_rows)\n    failed_ids = [row['id'] for row in rows]\n    print(f\"Failed IDs: {failed_ids}\")\n\n\n# A transform function that takes an iterable and yields one row at a time\n# returns a \"generator\".  The generator is also iterable, and records are\n# processed as they are read so the whole file is never held in memory.\ndef transform(rows: Iterable[dict]) -> Iterable[dict]:\n    \"\"\"Rename time column and convert to Python datetime.\"\"\"\n    for row in rows:\n        row['time'] = row.pop('phenomenonTime')\n        row['time'] = dt.datetime.strptime(row['time'], \"%Y-%m-%dT%H:%M:%S.%fZ\")\n        yield row\n\n\nif __name__ == \"__main__\":\n    import logging\n    from etlhelper import logger\n    logger.setLevel(logging.INFO)\n\n    db = DbParams(dbtype=\"SQLITE\", filename=\"observations.sqlite\")\n    with db.connect() as conn:\n        load_observations('observations.csv', conn)\n```\n\n\n### Export data to CSV\n\nThe [Pandas](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.read_sql.html) library can connect to databases via SQLAlchemy.\nIt has powerful tools for manipulating tabular data.\nETL Helper makes it easy to prepare the SQL Alchemy connection.\n\n```python\nimport pandas as pd\nfrom sqlalchemy import create_engine\n\nfrom my_databases import ORACLEDB\n\nengine = create_engine(ORACLEDB.get_sqlalchemy_connection_string(\"ORACLE_PASSWORD\"))\n\nsql = \"SELECT * FROM my_table\"\ndf = pd.read_sql(sql, engine)\ndf.to_csv('my_data.csv', header=True, index=False, float_format='%.3f')\n```\n\n\n## Development\n\n### Maintainers\n\nETL Helper was created by and is maintained by British Geological Survey Informatics.\n\n+ John A Stevenson ([volcan01010](https://github.com/volcan01010))\n+ Jo Walsh ([metazool](https://github.com/metazool))\n+ Declan Valters ([dvalters](https://github.com/dvalters))\n+ Colin Blackburn ([ximenesuk](https://github.com/ximenesuk))\n+ Daniel Sutton ([kerberpolis](https://github.com/kerberpolis))\n\n### Development status\n\nThe code is still under active development and breaking changes are possible.\nUsers should pin the version in their dependency lists and\n[watch](https://docs.github.com/en/github/managing-subscriptions-and-notifications-on-github/viewing-your-subscriptions#configuring-your-watch-settings-for-an-individual-repository)\nthe repository for new releases.\nSee [CONTRIBUTING.md](CONTRIBUTING.md) for details on how to contribute.\n\n\n### Licence\n\nETL Helper is distributed under the [LGPL v3.0 licence](LICENSE).\nCopyright: \u00a9 BGS / UKRI 2019\n\n\n## References\n\n+ [PEP249 DB API2](https://www.python.org/dev/peps/pep-0249/#cursor-objects)\n+ [psycopg2](http://initd.org/psycopg/docs/cursor.html)\n+ [cx_Oracle](https://cx-oracle.readthedocs.io/en/latest/cursor.html)\n+ [pyodbc](https://pypi.org/project/pyodbc/)\n+ [sqlite3](https://docs.python.org/3/library/sqlite3.html)\n\n\n",
        "description_content_type": "text/markdown",
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://github.com/BritishGeologicalSurvey/etlhelper",
        "keywords": "",
        "license": "",
        "maintainer": "",
        "maintainer_email": "",
        "name": "etlhelper",
        "package_url": "https://pypi.org/project/etlhelper/",
        "platform": null,
        "project_url": "https://pypi.org/project/etlhelper/",
        "project_urls": {
            "Homepage": "https://github.com/BritishGeologicalSurvey/etlhelper"
        },
        "release_url": "https://pypi.org/project/etlhelper/0.12.2/",
        "requires_dist": [
            "flake8 ; extra == 'dev'",
            "ipdb ; extra == 'dev'",
            "ipython ; extra == 'dev'",
            "pytest ; extra == 'dev'",
            "pytest-cov ; extra == 'dev'",
            "versioneer ; extra == 'dev'",
            "pyodbc ; extra == 'mssql'",
            "cx-oracle ; extra == 'oracle'",
            "psycopg2-binary ; extra == 'postgres'"
        ],
        "requires_python": ">=3.6",
        "summary": "A Python library to simplify data transfer into and out of databases.",
        "version": "0.12.2",
        "yanked": false,
        "yanked_reason": null
    },
    "last_serial": 13347876,
    "releases": {
        "0.10.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "ff201b3f3ba634dbf7a53536e193fa88",
                    "sha256": "3fff562fa00e8db7830414df5ad3f966f0db53e2066c499328b635534636924a"
                },
                "downloads": -1,
                "filename": "etlhelper-0.10.0-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "ff201b3f3ba634dbf7a53536e193fa88",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": ">=3.6",
                "size": 36200,
                "upload_time": "2021-10-29T16:43:13",
                "upload_time_iso_8601": "2021-10-29T16:43:13.560619Z",
                "url": "https://files.pythonhosted.org/packages/e3/46/f9929e178c0e3c8483be61df863b9495bd69759d6543af0f11004ebb2895/etlhelper-0.10.0-py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "3e033174c91421322647fda68d28ab4c",
                    "sha256": "50192d8f0c4051f1e27e12e4f45077acd066f9738a87ad091aab8a2335ef61aa"
                },
                "downloads": -1,
                "filename": "etlhelper-0.10.0.zip",
                "has_sig": false,
                "md5_digest": "3e033174c91421322647fda68d28ab4c",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=3.6",
                "size": 75522,
                "upload_time": "2021-10-29T16:43:15",
                "upload_time_iso_8601": "2021-10-29T16:43:15.140551Z",
                "url": "https://files.pythonhosted.org/packages/5d/6a/e2c640f5363a6ebe036b53fd56b0d3067471b571ea0cd0862c3a4bd8ec0e/etlhelper-0.10.0.zip",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "0.11.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "7bf6618511bbeb40e2b035d8f9352414",
                    "sha256": "adb4354f54229edbafa71547168ec3e7e25625e9ab60ac17371b300cc7896cdf"
                },
                "downloads": -1,
                "filename": "etlhelper-0.11.0-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "7bf6618511bbeb40e2b035d8f9352414",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": ">=3.6",
                "size": 37825,
                "upload_time": "2022-01-14T09:52:39",
                "upload_time_iso_8601": "2022-01-14T09:52:39.575296Z",
                "url": "https://files.pythonhosted.org/packages/31/3e/d898ebf236e46c85ec934898e41b67e2b746eaeee8cbf50356df4abb4c6b/etlhelper-0.11.0-py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "5cabc877329dfbbf62ec95e2d454beb6",
                    "sha256": "cf295c0cc184d884a91d9a308b8191efd0a69c840cfeb7b8273e876601f19020"
                },
                "downloads": -1,
                "filename": "etlhelper-0.11.0.zip",
                "has_sig": false,
                "md5_digest": "5cabc877329dfbbf62ec95e2d454beb6",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=3.6",
                "size": 79118,
                "upload_time": "2022-01-14T09:52:41",
                "upload_time_iso_8601": "2022-01-14T09:52:41.491834Z",
                "url": "https://files.pythonhosted.org/packages/af/5d/75fd0e31e6df79a42b810bc9b7fe10ed93a02d5042170e0cafc037411607/etlhelper-0.11.0.zip",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "0.12.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "ff2e62d11199e66ee56cc4031e65b3b5",
                    "sha256": "f3e874a9bf09f3a1ead41a5704253c47ecd2070f487a41a1afea928156c4c7c1"
                },
                "downloads": -1,
                "filename": "etlhelper-0.12.0-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "ff2e62d11199e66ee56cc4031e65b3b5",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": ">=3.6",
                "size": 38226,
                "upload_time": "2022-03-29T09:18:30",
                "upload_time_iso_8601": "2022-03-29T09:18:30.528702Z",
                "url": "https://files.pythonhosted.org/packages/6c/fe/f51d5a41bf3721f357a5bd2deb5071a7639e8abe6c95f4899eb94db77228/etlhelper-0.12.0-py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "e8a567df6849cdf3d6a3d0cb655dbb75",
                    "sha256": "76999a344db35cf16c95ec4a45314c1528f10d103f4333e2847cc1a561ce93f1"
                },
                "downloads": -1,
                "filename": "etlhelper-0.12.0.zip",
                "has_sig": false,
                "md5_digest": "e8a567df6849cdf3d6a3d0cb655dbb75",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=3.6",
                "size": 79893,
                "upload_time": "2022-03-29T09:18:32",
                "upload_time_iso_8601": "2022-03-29T09:18:32.502516Z",
                "url": "https://files.pythonhosted.org/packages/18/79/23d9301d7f23552d24a77d362c604440b82c7d5e1a868d282b28cbcc134c/etlhelper-0.12.0.zip",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "0.12.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "cc2a31612fb831178176f9aecf692713",
                    "sha256": "669eb79cbacafed5da2891ad75d091204b1460a9bd4c55d59464f84a4fb6ab6f"
                },
                "downloads": -1,
                "filename": "etlhelper-0.12.1-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "cc2a31612fb831178176f9aecf692713",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": ">=3.6",
                "size": 38272,
                "upload_time": "2022-03-29T16:59:04",
                "upload_time_iso_8601": "2022-03-29T16:59:04.247882Z",
                "url": "https://files.pythonhosted.org/packages/8f/db/38f5e78d982f14d6e820fc7970a02a7abad85d6ca969d9fecc1b015c7a77/etlhelper-0.12.1-py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "aa72911a3a48dfcfaa594ea7b5234f87",
                    "sha256": "2af64d7287ca1223245ca7d788d25718caeae9f7e1a4d7682bd51b23ad684598"
                },
                "downloads": -1,
                "filename": "etlhelper-0.12.1.zip",
                "has_sig": false,
                "md5_digest": "aa72911a3a48dfcfaa594ea7b5234f87",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=3.6",
                "size": 79937,
                "upload_time": "2022-03-29T16:59:06",
                "upload_time_iso_8601": "2022-03-29T16:59:06.931606Z",
                "url": "https://files.pythonhosted.org/packages/a7/dc/1a6a35a9953ab124f6c34edff59dec1735ba7c01c513584535045bb11fa8/etlhelper-0.12.1.zip",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "0.12.2": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "fc4af0e36e748057c7feca8933b0132f",
                    "sha256": "6ef6b9d7e6e268e711d5da6d9bd21f3ede2e3425632a2d5e46a2ea22bbac125f"
                },
                "downloads": -1,
                "filename": "etlhelper-0.12.2-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "fc4af0e36e748057c7feca8933b0132f",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": ">=3.6",
                "size": 38260,
                "upload_time": "2022-03-30T15:17:43",
                "upload_time_iso_8601": "2022-03-30T15:17:43.499190Z",
                "url": "https://files.pythonhosted.org/packages/e6/4c/448b9debcbb585a569ea420e2f665e5f347c49116c13d3470958947793ae/etlhelper-0.12.2-py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "e417dc47e157109c31d35eed62572547",
                    "sha256": "bfd561fe1f9410840f0087d29e1e90843eee2bb6c604f0864f3a114a998e59ef"
                },
                "downloads": -1,
                "filename": "etlhelper-0.12.2.zip",
                "has_sig": false,
                "md5_digest": "e417dc47e157109c31d35eed62572547",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=3.6",
                "size": 79927,
                "upload_time": "2022-03-30T15:17:45",
                "upload_time_iso_8601": "2022-03-30T15:17:45.860604Z",
                "url": "https://files.pythonhosted.org/packages/6b/05/024045e2b206c9d339734fa7f37b158dad1b5e527706671dcee075670cf5/etlhelper-0.12.2.zip",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "0.5.4": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "b76bb0e28f1f74dd630f4f659920be05",
                    "sha256": "70e23972aa3843ca2514e82384679122346e1b7408b3e174b3474b8a9a52725f"
                },
                "downloads": -1,
                "filename": "etlhelper-0.5.4.zip",
                "has_sig": false,
                "md5_digest": "b76bb0e28f1f74dd630f4f659920be05",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=3.6",
                "size": 47428,
                "upload_time": "2019-09-18T23:50:37",
                "upload_time_iso_8601": "2019-09-18T23:50:37.194622Z",
                "url": "https://files.pythonhosted.org/packages/da/96/f7e0cbbe5d47e9b27635cb337cbea9075b372d149ae8fc8a59cabdab793a/etlhelper-0.5.4.zip",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "0.5.5": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "ec149f2f2bc12dad6fafbd184b3639e1",
                    "sha256": "1bac147e193245e757266859ce21f52d17d90dad68760ae6beb8452e9741d1b4"
                },
                "downloads": -1,
                "filename": "etlhelper-0.5.5.zip",
                "has_sig": false,
                "md5_digest": "ec149f2f2bc12dad6fafbd184b3639e1",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=3.6",
                "size": 49333,
                "upload_time": "2019-09-27T15:04:09",
                "upload_time_iso_8601": "2019-09-27T15:04:09.318180Z",
                "url": "https://files.pythonhosted.org/packages/e6/84/897e53a054e68fe3ceccdabb169e248616d9dfb35ed6e923d66620a31acf/etlhelper-0.5.5.zip",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "0.6.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "60c3d24553b0f35753052d7827a9ed7b",
                    "sha256": "6a9febecf896a16b90b07e0760ad8c9e5c3d3b2d5d9b226c6dafe2fd70da74e9"
                },
                "downloads": -1,
                "filename": "etlhelper-0.6.0.zip",
                "has_sig": false,
                "md5_digest": "60c3d24553b0f35753052d7827a9ed7b",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=3.6",
                "size": 50855,
                "upload_time": "2019-10-22T10:35:34",
                "upload_time_iso_8601": "2019-10-22T10:35:34.238784Z",
                "url": "https://files.pythonhosted.org/packages/7d/63/176ea458ad1c88aff6fb28ddb69c8776bf4611b3af09038c94676656cca0/etlhelper-0.6.0.zip",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "0.7.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "35bf37a8e868e6ca4d5a1ead9d0a495e",
                    "sha256": "6e0d2002ce4560bb96ce36a862e6297173ec942833a5116ac1d8792069f0992f"
                },
                "downloads": -1,
                "filename": "etlhelper-0.7.0.zip",
                "has_sig": false,
                "md5_digest": "35bf37a8e868e6ca4d5a1ead9d0a495e",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=3.6",
                "size": 51690,
                "upload_time": "2019-11-04T18:25:17",
                "upload_time_iso_8601": "2019-11-04T18:25:17.866961Z",
                "url": "https://files.pythonhosted.org/packages/fb/9f/e04ed5a3f391390a2c4ad678bfdf8d275b4f5b3f884d31fa86a61515d43b/etlhelper-0.7.0.zip",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "0.7.4": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "7b1be2d86291c15ae74646c9b39bb226",
                    "sha256": "58a10e4e8f92a9e33f331dfdf59d0aef2bc83b0ed958d8db64a5f8b2e3e8eedf"
                },
                "downloads": -1,
                "filename": "etlhelper-0.7.4.zip",
                "has_sig": false,
                "md5_digest": "7b1be2d86291c15ae74646c9b39bb226",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=3.6",
                "size": 51776,
                "upload_time": "2019-12-11T10:22:20",
                "upload_time_iso_8601": "2019-12-11T10:22:20.958034Z",
                "url": "https://files.pythonhosted.org/packages/ea/50/cf12904ed2b496694831eb5f80bac3950854d91e38b54d0e1e52c1a514b9/etlhelper-0.7.4.zip",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "0.7.5": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "50d6fcb25b89eea69c4406a0bf007874",
                    "sha256": "5aae5fe488bb0fb9ac08f51f8c9a07e168ec556a17b9e2882395986076a9f6be"
                },
                "downloads": -1,
                "filename": "etlhelper-0.7.5.zip",
                "has_sig": false,
                "md5_digest": "50d6fcb25b89eea69c4406a0bf007874",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=3.6",
                "size": 52308,
                "upload_time": "2019-12-11T10:51:35",
                "upload_time_iso_8601": "2019-12-11T10:51:35.885049Z",
                "url": "https://files.pythonhosted.org/packages/2c/a7/6cb6978b75a9536b688a989dfd343858e8dfed7e3d2100abafd10dbf5460/etlhelper-0.7.5.zip",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "0.7.6": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "ab0276e5653d638f8ae182808c01e49d",
                    "sha256": "9030cad2994f3eb31d0f4eb3b96b73550fc419137c7ed11575d7296fa872ca02"
                },
                "downloads": -1,
                "filename": "etlhelper-0.7.6.zip",
                "has_sig": false,
                "md5_digest": "ab0276e5653d638f8ae182808c01e49d",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=3.6",
                "size": 52308,
                "upload_time": "2019-12-11T11:52:03",
                "upload_time_iso_8601": "2019-12-11T11:52:03.427314Z",
                "url": "https://files.pythonhosted.org/packages/55/22/4b9f7a6cde71fc585cc24c547bc49253ea2ade1210f8a30472bb7a9e30f9/etlhelper-0.7.6.zip",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "0.8.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "a67d7518be6dc32d4b795fbdd0113ffb",
                    "sha256": "1a8a3d98500238730db13748a2a8711e15f2d3e0b7ef8abf49cfd45cd72b1666"
                },
                "downloads": -1,
                "filename": "etlhelper-0.8.0.zip",
                "has_sig": false,
                "md5_digest": "a67d7518be6dc32d4b795fbdd0113ffb",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=3.6",
                "size": 53877,
                "upload_time": "2020-06-18T15:56:34",
                "upload_time_iso_8601": "2020-06-18T15:56:34.826072Z",
                "url": "https://files.pythonhosted.org/packages/7a/dd/297324342b7339a1fbde18d3b99137ac9b27c93460209ee9b1cea07a03fd/etlhelper-0.8.0.zip",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "0.9.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "ad3fc08b58deb06381140dd53d42b064",
                    "sha256": "6e8bd3864898ec94d5c66a251faf812aa5002890fe8e30e1e31d2870b5f7578b"
                },
                "downloads": -1,
                "filename": "etlhelper-0.9.0.zip",
                "has_sig": false,
                "md5_digest": "ad3fc08b58deb06381140dd53d42b064",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=3.6",
                "size": 61400,
                "upload_time": "2020-08-26T08:58:56",
                "upload_time_iso_8601": "2020-08-26T08:58:56.913281Z",
                "url": "https://files.pythonhosted.org/packages/81/56/15f1570b3bd27faec34c3596b9cd12ffeb9d836bc52795dbb43312e73ac9/etlhelper-0.9.0.zip",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "0.9.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "8039fef2a7373a4d846268d4f4823635",
                    "sha256": "9f95be525932b3d6a80f5c33caaa9cd00fa7fd6fc4b9d5adc9c73288db6612d8"
                },
                "downloads": -1,
                "filename": "etlhelper-0.9.1.zip",
                "has_sig": false,
                "md5_digest": "8039fef2a7373a4d846268d4f4823635",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=3.6",
                "size": 61393,
                "upload_time": "2020-08-26T15:55:40",
                "upload_time_iso_8601": "2020-08-26T15:55:40.099731Z",
                "url": "https://files.pythonhosted.org/packages/cf/af/8ef9e4b037fd1057701c74150f4e4575543d588a6fe77e1e0f273710497e/etlhelper-0.9.1.zip",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "0.9.2": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "8b6401bc88e47a921fbd42c5c04b29bc",
                    "sha256": "7904cf34e39d72f919c6e356e0f12b1828389bf4dbea1bb583bc6c352c6c8176"
                },
                "downloads": -1,
                "filename": "etlhelper-0.9.2.zip",
                "has_sig": false,
                "md5_digest": "8b6401bc88e47a921fbd42c5c04b29bc",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=3.6",
                "size": 61805,
                "upload_time": "2020-09-09T14:24:28",
                "upload_time_iso_8601": "2020-09-09T14:24:28.398018Z",
                "url": "https://files.pythonhosted.org/packages/fd/76/acc6207cf6ba0b299ef4a41a604957b729fb35967a5c2eef104d2c709cf7/etlhelper-0.9.2.zip",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "0.9.3": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "955949378e2fd834d80ca60e033ea0f4",
                    "sha256": "34d455b25dbc974d7e2bdefc662faf983338e6d728f06dd0c69292599a17d867"
                },
                "downloads": -1,
                "filename": "etlhelper-0.9.3-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "955949378e2fd834d80ca60e033ea0f4",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": ">=3.6",
                "size": 31390,
                "upload_time": "2020-09-25T16:47:51",
                "upload_time_iso_8601": "2020-09-25T16:47:51.597136Z",
                "url": "https://files.pythonhosted.org/packages/7d/dd/e9cd13decad523722b3b64565d6f633cf638fa569ab1fd28fc2b892dcdd3/etlhelper-0.9.3-py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "a99a6093f8bfec392c7bb80fb0100420",
                    "sha256": "236dcfa74de3e3053de9889657f46b55209bc2aa03ca09c4a17a6571f76d83fb"
                },
                "downloads": -1,
                "filename": "etlhelper-0.9.3.zip",
                "has_sig": false,
                "md5_digest": "a99a6093f8bfec392c7bb80fb0100420",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=3.6",
                "size": 63017,
                "upload_time": "2020-09-25T16:47:53",
                "upload_time_iso_8601": "2020-09-25T16:47:53.049197Z",
                "url": "https://files.pythonhosted.org/packages/f8/91/146f63bb472c8c9cafa9286f69b742ea31ccf39f32de0796a7e4a933c141/etlhelper-0.9.3.zip",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "0.9.4": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "56b63788f4082731fceb8f9872cc8db2",
                    "sha256": "b5ad47625ca12d7c8f1c14595b1833189fac4285ff8a5e9292b5f3902cb6514b"
                },
                "downloads": -1,
                "filename": "etlhelper-0.9.4-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "56b63788f4082731fceb8f9872cc8db2",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": ">=3.6",
                "size": 31652,
                "upload_time": "2020-10-09T15:58:38",
                "upload_time_iso_8601": "2020-10-09T15:58:38.266162Z",
                "url": "https://files.pythonhosted.org/packages/4a/fa/f1e9ad97f933dc224f6632d6ef4f74d5a7ed1610450cb010e0c885806ea5/etlhelper-0.9.4-py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "b2f11fdfd8a2d450c0d6b6221e941943",
                    "sha256": "6d3e49ed554b5104a0d9c10704a3f70a18908bf0e4e9fc9d95c45db6a5626938"
                },
                "downloads": -1,
                "filename": "etlhelper-0.9.4.zip",
                "has_sig": false,
                "md5_digest": "b2f11fdfd8a2d450c0d6b6221e941943",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=3.6",
                "size": 63306,
                "upload_time": "2020-10-09T15:58:39",
                "upload_time_iso_8601": "2020-10-09T15:58:39.941424Z",
                "url": "https://files.pythonhosted.org/packages/d9/91/6911fc1f715e1ac511446cbefdee73caf0170d77854b21a335fb645b8901/etlhelper-0.9.4.zip",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "0.9.5": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "8f53b3a21c9e4088f66ec4c0c414aad4",
                    "sha256": "a92e6695ff42756f82858ae8a460c60b06cd697cb9228f7699d56acce82eacdf"
                },
                "downloads": -1,
                "filename": "etlhelper-0.9.5-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "8f53b3a21c9e4088f66ec4c0c414aad4",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": ">=3.6",
                "size": 32822,
                "upload_time": "2020-11-20T11:22:20",
                "upload_time_iso_8601": "2020-11-20T11:22:20.726528Z",
                "url": "https://files.pythonhosted.org/packages/fc/61/eb13e080aff82adc2157a2656a5ecc4e7da99f2136ead459d5a8c0cdccc8/etlhelper-0.9.5-py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "92c265bb0607bcec0a96174f4d9399fc",
                    "sha256": "76fd39e15a203613e7dd6085aab4242188c968f45019e65fd79b6cc2503dea1e"
                },
                "downloads": -1,
                "filename": "etlhelper-0.9.5.zip",
                "has_sig": false,
                "md5_digest": "92c265bb0607bcec0a96174f4d9399fc",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=3.6",
                "size": 66922,
                "upload_time": "2020-11-20T11:22:22",
                "upload_time_iso_8601": "2020-11-20T11:22:22.460101Z",
                "url": "https://files.pythonhosted.org/packages/54/94/769728abbd36cea10864453e5f328fe1aa1dd96e74fb3bb740bb6301a14b/etlhelper-0.9.5.zip",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "0.9.6": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "60d59dc37afcc6468fce504d092d590b",
                    "sha256": "6027d6acf49519364481a57315435dbbc3e960b693de3dd612ae67707c4c7310"
                },
                "downloads": -1,
                "filename": "etlhelper-0.9.6-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "60d59dc37afcc6468fce504d092d590b",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": ">=3.6",
                "size": 33580,
                "upload_time": "2020-12-15T11:52:50",
                "upload_time_iso_8601": "2020-12-15T11:52:50.421282Z",
                "url": "https://files.pythonhosted.org/packages/1b/f4/f6303fe77ed2a30a876b130d25818a335b12a85c5cf06d90221cb95e692a/etlhelper-0.9.6-py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "753e296619c9b9e7a2a7fbc9a254a95d",
                    "sha256": "4f0a281be8cea89987192632d4b6d716e1383d86e2a6eeba9618b548298c0517"
                },
                "downloads": -1,
                "filename": "etlhelper-0.9.6.zip",
                "has_sig": false,
                "md5_digest": "753e296619c9b9e7a2a7fbc9a254a95d",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=3.6",
                "size": 68615,
                "upload_time": "2020-12-15T11:52:52",
                "upload_time_iso_8601": "2020-12-15T11:52:52.165185Z",
                "url": "https://files.pythonhosted.org/packages/b8/c8/a5ee50809aa06b8384eefc0aa90d713b57bfb962df6f035cdb44ca49b809/etlhelper-0.9.6.zip",
                "yanked": false,
                "yanked_reason": null
            }
        ],
        "0.9.7": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "738d1008ce6237f11966867fa6cd8577",
                    "sha256": "bb06fe33adb331fa1ff868c123cbd4470fd81ca3a8bdbf43b0577bf45193e83a"
                },
                "downloads": -1,
                "filename": "etlhelper-0.9.7-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "738d1008ce6237f11966867fa6cd8577",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": ">=3.6",
                "size": 34053,
                "upload_time": "2021-09-10T15:55:58",
                "upload_time_iso_8601": "2021-09-10T15:55:58.243396Z",
                "url": "https://files.pythonhosted.org/packages/fc/03/4803f9f95b7c1952cdd777bb14bbc1814ac1b3604011e9cddc08e487524b/etlhelper-0.9.7-py3-none-any.whl",
                "yanked": false,
                "yanked_reason": null
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "6fcdf94bbb9ff311a0828503238f17ff",
                    "sha256": "97f9ceb81e06d0a61b2b085260f80fd9aaadf8117b793c0435695495eea00bf2"
                },
                "downloads": -1,
                "filename": "etlhelper-0.9.7.zip",
                "has_sig": false,
                "md5_digest": "6fcdf94bbb9ff311a0828503238f17ff",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": ">=3.6",
                "size": 71108,
                "upload_time": "2021-09-10T15:56:00",
                "upload_time_iso_8601": "2021-09-10T15:56:00.101258Z",
                "url": "https://files.pythonhosted.org/packages/db/3e/3ceae3b0e159d38d6956b02c2fe92d1d27cb0d8abe35eeeb537e50979291/etlhelper-0.9.7.zip",
                "yanked": false,
                "yanked_reason": null
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "fc4af0e36e748057c7feca8933b0132f",
                "sha256": "6ef6b9d7e6e268e711d5da6d9bd21f3ede2e3425632a2d5e46a2ea22bbac125f"
            },
            "downloads": -1,
            "filename": "etlhelper-0.12.2-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "fc4af0e36e748057c7feca8933b0132f",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": ">=3.6",
            "size": 38260,
            "upload_time": "2022-03-30T15:17:43",
            "upload_time_iso_8601": "2022-03-30T15:17:43.499190Z",
            "url": "https://files.pythonhosted.org/packages/e6/4c/448b9debcbb585a569ea420e2f665e5f347c49116c13d3470958947793ae/etlhelper-0.12.2-py3-none-any.whl",
            "yanked": false,
            "yanked_reason": null
        },
        {
            "comment_text": "",
            "digests": {
                "md5": "e417dc47e157109c31d35eed62572547",
                "sha256": "bfd561fe1f9410840f0087d29e1e90843eee2bb6c604f0864f3a114a998e59ef"
            },
            "downloads": -1,
            "filename": "etlhelper-0.12.2.zip",
            "has_sig": false,
            "md5_digest": "e417dc47e157109c31d35eed62572547",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": ">=3.6",
            "size": 79927,
            "upload_time": "2022-03-30T15:17:45",
            "upload_time_iso_8601": "2022-03-30T15:17:45.860604Z",
            "url": "https://files.pythonhosted.org/packages/6b/05/024045e2b206c9d339734fa7f37b158dad1b5e527706671dcee075670cf5/etlhelper-0.12.2.zip",
            "yanked": false,
            "yanked_reason": null
        }
    ],
    "vulnerabilities": []
}