{ "info": { "author": "Nishant Nath, Aniket Shenoy", "author_email": "", "bugtrack_url": null, "classifiers": [], "description": "# pyFission\n\n\n#### Description\npyFission syncs tables/schemas across databases as defined in the **configs/fission.py** file. \nIf *src_table* and *src_schema* args are not provided, it will sync all tables and schemas specified in the \n**fission.py** config file by summoning multiple bash commands in parallel. \n\nThe **pyfission** module has 2 methods of syncing tables - \n1. full: truncate + full sync of table, as name suggests\n2. incremental: syncs changes based on primary key and replication key provided in *fission.py* config file\n\npyFission automatically detects and builds the schema definition for tables, including nested and repeated fields \nfor JSON-like schemas e.g:- BigQuery. Also, pyFission patches tables automatically if any new fields are added. \n\n\n### Getting started\n#### Usage\n* Modify **configs/custom_config.py** and **configs/fission.py** with DB creds and sync config respectively\n* Add service account private key files/secrets to **secrets_storage** dir if needed\n\n```bash\n$ python -m pyfission --src [] --dest [] sync --help\nusage: __main__.py pyfission [-h] [--src_table SRC_TABLE]\n [--src_schema SRC_SCHEMA] [--src_db SRC_DB]\n [--dest_table DEST_TABLE]\n [--dest_schema DEST_SCHEMA] [--dest_db DEST_DB]\n [--method {full,incremental}]\n [--out_format {csv,json}]\n\noptional arguments:\n -h, --help show this help message and exit\n --src_table SRC_TABLE\n Overrides table definition from pyfission configs\n --src_schema SRC_SCHEMA\n Overrides schema definition from pyfission configs\n --src_db SRC_DB Overrides database definition from pyfission configs\n --dest_table DEST_TABLE\n Overrides table definition from pyfission configs\n --dest_schema DEST_SCHEMA\n Overrides schema definition from pyfission configs\n --dest_db DEST_DB Overrides database definition from pyfission configs\n --method {full,incremental}\n Method of Replication\n --out_format {csv,json}\n Format of output files\n```\n\n* Recommended *out_format*:\n * *json* for BigQuery\n * *csv* for others\n\n* To sync a particular table - specify the src_schema and src_table args\n * *dest_table* will have same name as *src_table* unless explicitly specified\n * *dest_schema* will be set from **fission.py** config file's *fission_dest* dict\n * *src_db* and *dest_db* will derived from *src* and *dest*\n```bash\npython -m pyfission --src [] --dest [] sync --src_table [] --src_schema [] --out_format []\n```\n\n\n* To sync an entire DB - only provide *src* and *dest* args\n```bash\npython -m pyfission --src [] --dest [] sync --out_format []\n```\n\n\n#### Installation\n```bash\npip install pyfission\n```\n\n#### Contribution/Local installation\n1. Clone the repo\n```bash\n$ git clone https://github.com/nishantnath/pyfission.git\n```\n\n2. Change the working directory to fission\n```bash\n$ cd pyfission\n```\n\n3. Install python3 and the requirements if they don't exist\n```bash\n$ pip install -r requirements.txt\n```\n\n\n", "description_content_type": "text/markdown", "docs_url": null, "download_url": "", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "https://github.com/nishantnath/pyfission/", "keywords": "", "license": "MIT", "maintainer": "", "maintainer_email": "", "name": "pyFission", "package_url": "https://pypi.org/project/pyFission/", "platform": "", "project_url": "https://pypi.org/project/pyFission/", "project_urls": { "Homepage": "https://github.com/nishantnath/pyfission/" }, "release_url": "https://pypi.org/project/pyFission/1.0.2/", "requires_dist": [ "google-cloud-storage", "google-cloud-bigquery", "pandas-gbq", "pandas", "sqlalchemy", "pymysql", "pymssql", "pyodbc", "cx-Oracle", "boto3" ], "requires_python": "", "summary": "A tool to sync data across data sources", "version": "1.0.2" }, "last_serial": 5573705, "releases": { "1.0": [ { "comment_text": "", "digests": { "md5": "2bdc9c0fca8886a5194223230df43a52", "sha256": "f3eaa8ec165877b519c93e5de1c1feb5b7e0607100a7450cdbec1e42b744d49a" }, "downloads": -1, "filename": "pyFission-1.0-py3-none-any.whl", "has_sig": false, "md5_digest": "2bdc9c0fca8886a5194223230df43a52", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": null, "size": 33857, "upload_time": "2019-07-23T00:09:01", "url": "https://files.pythonhosted.org/packages/fa/1c/f827d7549efc9309dfb6557b409398335863c91f77eb7c1088fff75395f9/pyFission-1.0-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "eb2feb785f7705fe6019fc065e5e1ba4", "sha256": "a3c6f55e829f7b4ad24c033080e971e520ad1be63cc8b7f8fd53012a601fa071" }, "downloads": -1, "filename": "pyFission-1.0.tar.gz", "has_sig": false, "md5_digest": "eb2feb785f7705fe6019fc065e5e1ba4", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 25116, "upload_time": "2019-07-23T00:09:04", "url": "https://files.pythonhosted.org/packages/d1/70/7adf62c23fa7c3c4be3984c0dc3476741a48ee78150ba8caaa7479181e0f/pyFission-1.0.tar.gz" } ], "1.0.2": [ { "comment_text": "", "digests": { "md5": "a8de2fb7edb5b68de19f223b720380ad", "sha256": "0e30cf517f48a69ff9b3fa2b9ad7d90b70b3804a91d78b115006e8a59f1aa087" }, "downloads": -1, "filename": "pyFission-1.0.2-py3-none-any.whl", "has_sig": false, "md5_digest": "a8de2fb7edb5b68de19f223b720380ad", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": null, "size": 33796, "upload_time": "2019-07-23T17:57:18", "url": "https://files.pythonhosted.org/packages/5c/44/201246c0fd0768c4183b2d09ba450d4e80a085eda754b32b436513762047/pyFission-1.0.2-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "be7c45407b4ba5d422166fcbf434f2ee", "sha256": "91c7189e2f66bc8c042ffeb87e7cb7b436e712b630aeda1f33664496b3fa142a" }, "downloads": -1, "filename": "pyFission-1.0.2.tar.gz", "has_sig": false, "md5_digest": "be7c45407b4ba5d422166fcbf434f2ee", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 24086, "upload_time": "2019-07-23T17:57:19", "url": "https://files.pythonhosted.org/packages/64/40/3eee14e6afab19364b683d256bae031d056e59a2840661b7acb8f732884f/pyFission-1.0.2.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "a8de2fb7edb5b68de19f223b720380ad", "sha256": "0e30cf517f48a69ff9b3fa2b9ad7d90b70b3804a91d78b115006e8a59f1aa087" }, "downloads": -1, "filename": "pyFission-1.0.2-py3-none-any.whl", "has_sig": false, "md5_digest": "a8de2fb7edb5b68de19f223b720380ad", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": null, "size": 33796, "upload_time": "2019-07-23T17:57:18", "url": "https://files.pythonhosted.org/packages/5c/44/201246c0fd0768c4183b2d09ba450d4e80a085eda754b32b436513762047/pyFission-1.0.2-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "be7c45407b4ba5d422166fcbf434f2ee", "sha256": "91c7189e2f66bc8c042ffeb87e7cb7b436e712b630aeda1f33664496b3fa142a" }, "downloads": -1, "filename": "pyFission-1.0.2.tar.gz", "has_sig": false, "md5_digest": "be7c45407b4ba5d422166fcbf434f2ee", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 24086, "upload_time": "2019-07-23T17:57:19", "url": "https://files.pythonhosted.org/packages/64/40/3eee14e6afab19364b683d256bae031d056e59a2840661b7acb8f732884f/pyFission-1.0.2.tar.gz" } ] }