{
    "info": {
        "author": "Media Integration Technologies Ltd.",
        "author_email": "info@mediaintegration.co.uk",
        "bugtrack_url": null,
        "classifiers": [],
        "description": "# twiddlepy\n\n`twiddlepy` is a Python library designed for end-to-end extract, transform and load pipline (or ETL for short). Using a mapper file,\nand optional functions your data can be transformed into a better suited format.\n\n## Features\n\n- Extract, Transform and Load pipelines\n- Multiple datasource options for extracting data\n- Multiple repository options for loading data\n- Support for mapping input data\n\n## Installation\n\nTwiddlepy is available on the PyPi repository\n\n`pip install twiddlepy`\n\nOr if you want to install directly from the repository: `python setup.py install`, or drop the twiddlepy directory anywhere on your PYTHONPATH.\n\n## Connectors\n\nThere are a number of data repository connectors available with Twiddlepy. Currently implemented connectors include:\n\n### Data Source (Input)\n\n- File Based\n  - CSV\n  - Excel Document\n  - Support for custom file loading (e.g. HTML)\n- Database\n  - MySQL\n  - MSSQL\n  - Oracle\n  - SQLite\n- MongoDB\n\n### Repository (Output)\n\n- File Based\n  - CSV\n- Apache Solr\n\n## Usage\n\nCreate a runnable python file with the following code:\n\n```python\nfrom twiddlepy.config import config\nfrom twiddlepy.driver import TwiddleDriver\n\ndriver = TwiddleDriver(config)\ndriver.process_data()\n```\n\n### Example Project Structure\n\n```\n.\n|-- mapper\n|   |-- mapper.csv\n|-- local_functions.py\n|-- run.py (File that runs Twiddle)\n|-- twiddle.cfg\n```\n\n### User Configuration\n\nImporting config from `twiddle.config` will import the default configuration items for each of the processes,\nand will also look for a user defined configuration file on the path where the application is being run from.\n\nAll of the configuration items, including all of the default options can be found [here](twiddlepy/data/twiddle_defaults.cfg)\n\n### Mapper File\n\nA mapper file defined by the user is used to defined the input data that will be extracted from\nthe data repository. The mapper file is a CSV in which there are multiple columns that can be filled in\nto specify the data mappings. Thw following columns must be defined in the mapper:\n\n|      Column Name      |                             Description                             |                        Options                         |\n| :-------------------: | :-----------------------------------------------------------------: | :----------------------------------------------------: |\n|        dataset        |             The dataset twiddlepy will use mappings for             |                   Any name (string)                    |\n|   source_field_name   |                      A name of a source field                       |                   Any name (string)                    |\n|   source_field_type   |                  The data type of the source field                  |  One of: \"str\", \"int\", \"float\", \"double\", \"timestamp\"  |\n|     allow_missing     |            Allow the column to be missing in the dataset            |              One of: \"y\", \"n\" (Yes or No)              |\n|          min          |               Data Validation: minimum allowed value                |                   Any numeric value                    |\n|          max          |               Data Validation: maximum allowed value                |                   Any numeric value                    |\n|    allowed_values     |               Data Validation: list of allowed values               |                  Any array of values                   |\n|         unit          |                The unit the column is represented by                |               Any name (string) e.g. kg                |\n|      repository       |              The repository name the column belongs to              |                   Any name (string)                    |\n| repository_field_name |       The name the column will be renamed to for data loading       |                   Any name (string)                    |\n| repository_field_type |    The data type that will be applied to the column when loading    | One of: \"string\", \"integer\", \"float\", \"double\", \"date\" |\n|        ignore         | Mark column to be ignore by mapping process (for historic datasets) |              One of: \"y\", \"n\" (Yes or No)              |\n\n## Contribute\n\nAs a company, we welcome any input to fix/improve the project. Whilst we don't have a style guide currently,\nthis is something we will be working on in the future to improve the project further. We're very interested to hear\nwhat you think about Twiddlepy, and any improvements you would like to see so please raise any issues in the tracker!\n\n## Contact\n\nGot a problem/query and want to discuss it with us personally? Contact us at <info@mediaintegration.co.uk>. We also have a website with more\ninformation about the company [here](http://www.mediaintegration.co.uk)\n\n\n",
        "description_content_type": "",
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://github.com/mediaintegration/twiddlepy",
        "keywords": "",
        "license": "",
        "maintainer": "",
        "maintainer_email": "",
        "name": "twiddlepy",
        "package_url": "https://pypi.org/project/twiddlepy/",
        "platform": "",
        "project_url": "https://pypi.org/project/twiddlepy/",
        "project_urls": {
            "Homepage": "https://github.com/mediaintegration/twiddlepy"
        },
        "release_url": "https://pypi.org/project/twiddlepy/0.1.3/",
        "requires_dist": [
            "certifi (==2018.11.29)",
            "chardet (==3.0.4)",
            "cx-oracle (==7.1.1)",
            "cython (==0.29.6)",
            "idna (==2.8)",
            "kazoo (==2.6.1)",
            "numpy (==1.16.2)",
            "pandas-schema (==0.3.2)",
            "pandas (==0.24.1)",
            "pymongo (==3.7.2)",
            "pymssql (==2.1.4)",
            "pymysql (==0.9.3)",
            "python-dateutil (==2.8.0)",
            "pytz (==2018.9)",
            "requests (==2.21.0)",
            "simplejson (==3.16.0)",
            "six (==1.12.0)",
            "sqlalchemy (==1.3.0)",
            "urllib3 (==1.24.1)",
            "xlrd (==1.2.0)"
        ],
        "requires_python": "",
        "summary": "Extract, Transform and Load pipeline application",
        "version": "0.1.3"
    },
    "last_serial": 5187918,
    "releases": {
        "0.0.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "25cf9a343b6f6250f06eb1cb8ebb8d98",
                    "sha256": "d4092a06483200078ff32e2da58f7aa9037aceb2dccbf8b437c1926e123c8402"
                },
                "downloads": -1,
                "filename": "twiddlepy-0.0.1-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "25cf9a343b6f6250f06eb1cb8ebb8d98",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 49144,
                "upload_time": "2019-03-28T18:51:52",
                "url": "https://files.pythonhosted.org/packages/46/b6/5b7cea3e4513b28e1e21ad2e0ae2bf0ab80cd7f6aaf08d5fccadbce3c561/twiddlepy-0.0.1-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "0d5736a09f95935e9b61d54dfb2258eb",
                    "sha256": "3c1f30fc54bf47eb70b9c72154307bd1650465f2dd0cff6ea64ae923e27e2ede"
                },
                "downloads": -1,
                "filename": "twiddlepy-0.0.1.tar.gz",
                "has_sig": false,
                "md5_digest": "0d5736a09f95935e9b61d54dfb2258eb",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 31989,
                "upload_time": "2019-03-28T18:51:55",
                "url": "https://files.pythonhosted.org/packages/42/4d/77d49455ad98967ca8636f073278698861e1ff805ea60590aa0f28ac55fd/twiddlepy-0.0.1.tar.gz"
            }
        ],
        "0.1.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "18123141a8a20781de93099df18a623e",
                    "sha256": "bf3b290c61fbef69aded6a9b340f12ae2753dd2fcedd442a33d9c25325b12b07"
                },
                "downloads": -1,
                "filename": "twiddlepy-0.1.1-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "18123141a8a20781de93099df18a623e",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 51793,
                "upload_time": "2019-04-24T13:38:44",
                "url": "https://files.pythonhosted.org/packages/2d/6b/b01ab0220e076e11bf5b51e6dee8a311eb9077e130419a988ea16d85c05f/twiddlepy-0.1.1-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "7808af3e1cd2218a96fe066a2374cf6d",
                    "sha256": "cffa97b4aebc0fa4d59033a564bb836c016253df08aacc0d59b66171bffb0473"
                },
                "downloads": -1,
                "filename": "twiddlepy-0.1.1.tar.gz",
                "has_sig": false,
                "md5_digest": "7808af3e1cd2218a96fe066a2374cf6d",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 34182,
                "upload_time": "2019-04-24T13:38:50",
                "url": "https://files.pythonhosted.org/packages/43/46/789e17f5b055d42338f4d4326ff7c1c1409204f4ff158ab16ff27b2a8354/twiddlepy-0.1.1.tar.gz"
            }
        ],
        "0.1.2": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "196d383025c33acd1b250331a6c14adf",
                    "sha256": "3eb555ba3fcf8b1b92105960d8b3eb60059cc09948125eb93af29298a2893d44"
                },
                "downloads": -1,
                "filename": "twiddlepy-0.1.2-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "196d383025c33acd1b250331a6c14adf",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 51797,
                "upload_time": "2019-04-24T14:44:15",
                "url": "https://files.pythonhosted.org/packages/b0/eb/2d326f8f25fbaf8986960cbdd69865bf94f09bae7f95c2ab8b21c920d63f/twiddlepy-0.1.2-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "52c528649b31e6048beab4d204224341",
                    "sha256": "cf60bb38ef5221c89b44e93695a44f3a9e822139631e7934441ff867c101c3ce"
                },
                "downloads": -1,
                "filename": "twiddlepy-0.1.2.tar.gz",
                "has_sig": false,
                "md5_digest": "52c528649b31e6048beab4d204224341",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 34196,
                "upload_time": "2019-04-24T14:44:17",
                "url": "https://files.pythonhosted.org/packages/28/94/be1bac83180e5100b304de2367634a5bab004def1334b6d6e0b10f075832/twiddlepy-0.1.2.tar.gz"
            }
        ],
        "0.1.3": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "07d518dd208ca2ee2aae98ae40e67d69",
                    "sha256": "0325c9e6191ddd33e2cbde53b62da48e807c6c579c97e002cecef130ba9c2e72"
                },
                "downloads": -1,
                "filename": "twiddlepy-0.1.3-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "07d518dd208ca2ee2aae98ae40e67d69",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 54074,
                "upload_time": "2019-04-25T13:07:01",
                "url": "https://files.pythonhosted.org/packages/70/f6/12bce06e7b86709783bbd207e87cfe3f7df6ca403d6068136aba6e33fd24/twiddlepy-0.1.3-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "5867c9598d8d69c1a873afb76f9286dc",
                    "sha256": "5b121b7e3dd3bda0734fb2ba1b50e38a9becbee19d1d91dcec98b53298a4c0ed"
                },
                "downloads": -1,
                "filename": "twiddlepy-0.1.3.tar.gz",
                "has_sig": false,
                "md5_digest": "5867c9598d8d69c1a873afb76f9286dc",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 34354,
                "upload_time": "2019-04-25T13:07:04",
                "url": "https://files.pythonhosted.org/packages/e8/78/fb2e991491418f403ba23f8b64311810b6debb297abab095f4741a54009f/twiddlepy-0.1.3.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "07d518dd208ca2ee2aae98ae40e67d69",
                "sha256": "0325c9e6191ddd33e2cbde53b62da48e807c6c579c97e002cecef130ba9c2e72"
            },
            "downloads": -1,
            "filename": "twiddlepy-0.1.3-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "07d518dd208ca2ee2aae98ae40e67d69",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": null,
            "size": 54074,
            "upload_time": "2019-04-25T13:07:01",
            "url": "https://files.pythonhosted.org/packages/70/f6/12bce06e7b86709783bbd207e87cfe3f7df6ca403d6068136aba6e33fd24/twiddlepy-0.1.3-py3-none-any.whl"
        },
        {
            "comment_text": "",
            "digests": {
                "md5": "5867c9598d8d69c1a873afb76f9286dc",
                "sha256": "5b121b7e3dd3bda0734fb2ba1b50e38a9becbee19d1d91dcec98b53298a4c0ed"
            },
            "downloads": -1,
            "filename": "twiddlepy-0.1.3.tar.gz",
            "has_sig": false,
            "md5_digest": "5867c9598d8d69c1a873afb76f9286dc",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 34354,
            "upload_time": "2019-04-25T13:07:04",
            "url": "https://files.pythonhosted.org/packages/e8/78/fb2e991491418f403ba23f8b64311810b6debb297abab095f4741a54009f/twiddlepy-0.1.3.tar.gz"
        }
    ]
}