{ "info": { "author": "Alex Wang", "author_email": "", "bugtrack_url": null, "classifiers": [ "Intended Audience :: Developers", "License :: OSI Approved :: BSD License", "Operating System :: OS Independent", "Programming Language :: Python", "Programming Language :: Python :: 3.5", "Topic :: Internet :: WWW/HTTP", "Topic :: Software Development :: Libraries :: Python Modules" ], "description": "================\nData-Conversion\n================\n\n.. image:: https://img.shields.io/pypi/v/data-conversion.svg\n :target: https://pypi.python.org/pypi/data-conversion\n :alt: PyPI Version\n\n.. image:: https://img.shields.io/travis/xiaowangwindow/data-conversion/master.svg\n :target: http://travis-ci.org/xiaowangwindow/data-conversion\n :alt: Build Status\n\nData-Conversion is a framework to convert data from origin style to target style easily.\nWith custom settings, data-conversion can read data from MongoDB, convert\ndata by MAPPING Rules in settings, and save to destination collection in MongoDB.\n\nHow to Install\n==============\n\nInstall by pip::\n\n $ pip install data-conversion\n\nHow to Use\n===========\n\nFirst, you should create a new settings file, for example, ``settings_release.py``.\nThen, define custom settings like Setting Template File ``settings.py`` in ``data_conversion/settings.py``, whose arguments also describe below.\nFinally, run asynchronously::\n\n $ etl async settings_release.py\n\nor run synchronously::\n\n $ etl sync settings_release.py\n\n\nSettings\n==========\n\n+--------------------------+--------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------+\n| Argument | Description | Value Example |\n+--------------------------+--------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------+\n| MONGODB_HOST | Host of MongoDB which store origin data | '127.0.0.1' |\n+--------------------------+--------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------+\n| MONGODB_PORT | Port of MongoDB which store origin data | 27017 |\n+--------------------------+--------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------+\n| MONGODB_USERNAME | Username of MongoDB which store origin data | None / 'admin' |\n+--------------------------+--------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------+\n| MONGODB_PASSWORD | Password of MongoDB which store origin data | None / '123456' |\n+--------------------------+--------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------+\n| MONGODB_AUTHDB | DB of authorization which store username and password | 'admin' |\n+--------------------------+--------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------+\n| MONGODB_DB | DB of MongoDB which store origin data and will store result data | 'data' |\n+--------------------------+--------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------+\n| MONGODB_SRC_COLL | Source Collection of MongoDB which store origin data | 'src_coll' |\n+--------------------------+--------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------+\n| MONGODB_DST_COLL | Destination Collection of MongoDB which will store result data | 'dst_coll' |\n+--------------------------+--------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------+\n| MONGODB_DST_COLL_INDEX | Destination Collection Index of MongoDB which store result data | [([('url', pymongo.ASCENDING)], {'unique':True}), ([('domain', pymongo.ASCENDING)], {})] |\n+--------------------------+--------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------+\n| MONGODB_ERROR_COLL | Error Collection of MongoDB which will store error data when convert raise exception | 'error_coll' |\n+--------------------------+--------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------+\n| MONGODB_ERROR_COLL_INDEX | Collection Index of Error Collection of MongoDB | [([('url', pymongo.ASCENDING)], {'unique': True})] |\n+--------------------------+--------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------+\n| SRC_COLL_QUERY | Query condition to select documents to be converted | { 'filter': {}, 'projection': None, 'start': 0, 'limit': 1000 } |\n+--------------------------+--------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------+\n| WRITE_CONDITION_DICT | write to dst_coll which collection.update({CONDITION}, {$set:{dst_document}}, upsert=True) | {'$set': ['url']} |\n+--------------------------+--------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------+\n| MAPPING | list to mapper, rules of conversion | [Mapper('url', 'url', str, None)] // src_key, dst_key, dst_type, custom_convert_function |\n+--------------------------+--------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------+\n| OPERATE_MAPPING_DICT | dict to mapper, rules of conversion | {'$set':MAPPING, '$push': MAPPING2, '$addToSet': MAPPING3} |\n+--------------------------+--------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------+\n| PROCESS_NUM | Number of process to run conversion | 1 |\n+--------------------------+--------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------+\n| CONCURRENT_PER_PROCESS | number of concurrent group to run in one process | 100 |\n+--------------------------+--------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------+\n| LOG_LEVEL | Level of logging | logging.INFO |\n+--------------------------+--------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------+\n\nSettings explain\n==================\nThe most important part in settings is MAPPING. MAPPING contains a list of Mapper,\nwhich is a namedtuple (src_key, dst_key, dst_type, custom_convert).\ndst_type and custom_convert can be ``None`` if you want to preserve origin type and value.\n\nNow, we support '$set', '$push', '$addToSet' operation when update document,\nif you want to add each array element to an existed array,\nplease add '$each_' by custom_convert_function.\n.. _$each https://docs.mongodb.com/manual/reference/operator/update/addToSet/#each-modifier\n\n\nException Handling\n===================\nException occured in convert function will be save into error collection which\ndefined in settings.\n\nIf you want to record the key of document which excpetion raise, you can\n``raise ValueError('key')`` contains ``key`` as an argument.", "description_content_type": "", "docs_url": null, "download_url": "", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "https://github.com/xiaowangwindow/data-conversion", "keywords": "", "license": "BSD", "maintainer": "Alex Wang", "maintainer_email": "xiaowangwindow@163.com", "name": "data-conversion", "package_url": "https://pypi.org/project/data-conversion/", "platform": "", "project_url": "https://pypi.org/project/data-conversion/", "project_urls": { "Homepage": "https://github.com/xiaowangwindow/data-conversion" }, "release_url": "https://pypi.org/project/data-conversion/0.0.4/", "requires_dist": null, "requires_python": "", "summary": "A ETL framework to convert data", "version": "0.0.4" }, "last_serial": 3719242, "releases": { "0.0.1": [ { "comment_text": "", "digests": { "md5": "6c8bb3f3c07d9bce4a9cdfb05874f04d", "sha256": "f02d564c5e1e6e24c7e6781cadd8049cb653be81dfe449dab98cf690061801f5" }, "downloads": -1, "filename": "data-conversion-0.0.1.tar.gz", "has_sig": false, "md5_digest": "6c8bb3f3c07d9bce4a9cdfb05874f04d", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 7311, "upload_time": "2018-01-04T08:24:14", "url": "https://files.pythonhosted.org/packages/76/88/26c73ae56131ac0de46eec25f875e34059fbcfd465c74d55fac70701c319/data-conversion-0.0.1.tar.gz" } ], "0.0.2": [ { "comment_text": "", "digests": { "md5": "b3bdf0a53f5a508a9cc4c8e3104f51ab", "sha256": "25617c55bce5c5a30f1731c07bc177b707e54287d8cea6f6edd03d7e3b1cc391" }, "downloads": -1, "filename": "data-conversion-0.0.2.tar.gz", "has_sig": false, "md5_digest": "b3bdf0a53f5a508a9cc4c8e3104f51ab", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 7611, "upload_time": "2018-01-05T02:39:28", "url": "https://files.pythonhosted.org/packages/e1/94/a1d6e92294e690c1f61d1b88f220c1d29cf8c732c2dd76295f3c5e14d99f/data-conversion-0.0.2.tar.gz" } ], "0.0.3": [ { "comment_text": "", "digests": { "md5": "d6b235d56bf390827d883932589b7eed", "sha256": "cc48ca0205513537f633991b47b74ea8a3261844e4a65287a6c793d7f74d971f" }, "downloads": -1, "filename": "data-conversion-0.0.3.tar.gz", "has_sig": false, "md5_digest": "d6b235d56bf390827d883932589b7eed", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 7691, "upload_time": "2018-01-10T06:51:58", "url": "https://files.pythonhosted.org/packages/87/b6/b340b866efd91b64e82fa0e6f040dfa2cd82cbaa4f47a06b4acf5f93c692/data-conversion-0.0.3.tar.gz" } ], "0.0.4": [ { "comment_text": "", "digests": { "md5": "b1147123357bbc59e3f69efe91c01739", "sha256": "0918ac699d6dea6b1c09a0b7ffe4495f8d5533d5f1844ab66b25806bf4a94f4a" }, "downloads": -1, "filename": "data-conversion-0.0.4.tar.gz", "has_sig": false, "md5_digest": "b1147123357bbc59e3f69efe91c01739", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 8017, "upload_time": "2018-03-30T08:38:25", "url": "https://files.pythonhosted.org/packages/8d/06/da62579e18adb5cfdf9d5f627a2760a316dbf294e593a120bcfc42b69cb8/data-conversion-0.0.4.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "b1147123357bbc59e3f69efe91c01739", "sha256": "0918ac699d6dea6b1c09a0b7ffe4495f8d5533d5f1844ab66b25806bf4a94f4a" }, "downloads": -1, "filename": "data-conversion-0.0.4.tar.gz", "has_sig": false, "md5_digest": "b1147123357bbc59e3f69efe91c01739", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 8017, "upload_time": "2018-03-30T08:38:25", "url": "https://files.pythonhosted.org/packages/8d/06/da62579e18adb5cfdf9d5f627a2760a316dbf294e593a120bcfc42b69cb8/data-conversion-0.0.4.tar.gz" } ] }