{ "info": { "author": "Catherine Devlin", "author_email": "catherine.devlin@gsa.gov", "bugtrack_url": null, "classifiers": [ "Development Status :: 3 - Alpha", "License :: CC0 1.0 Universal (CC0 1.0) Public Domain Dedication", "Natural Language :: English", "Programming Language :: Python :: 2", "Programming Language :: Python :: 2.7", "Programming Language :: Python :: 3", "Programming Language :: Python :: 3.4", "Topic :: Database", "Topic :: Software Development :: Testing" ], "description": "rdbms-subsetter\n===============\n\n.. image:: https://img.shields.io/pypi/v/rdbms-subsetter.svg\n :target: https://pypi.python.org/pypi/rdbms-subsetter\n :alt: PyPI status\n\n.. image:: https://circleci.com/gh/18F/rdbms-subsetter.svg?style=svg\n :target: https://circleci.com/gh/18F/rdbms-subsetter\n :alt: CircleCI status\n\n.. image:: https://api.codeclimate.com/v1/badges/8fe62703d4999c07968c/maintainability\n :target: https://codeclimate.com/github/18F/rdbms-subsetter/maintainability\n :alt: Maintainability\n\n.. image:: https://api.codeclimate.com/v1/badges/8fe62703d4999c07968c/test_coverage\n :target: https://codeclimate.com/github/18F/rdbms-subsetter/test_coverage\n :alt: Test Coverage\n\n.. image:: https://gemnasium.com/badges/github.com/18F/rdbms-subsetter.svg\n :target: https://gemnasium.com/github.com/18F/rdbms-subsetter\n :alt: Gemnasium\n\nGenerate a random sample of rows from a relational database that preserves\nreferential integrity - so long as constraints are defined, all parent rows\nwill exist for child rows.\n\nGood for creating test/development databases from production. It's slow,\nbut how often do you need to generate a test/development database?\n\nUsage::\n\n rdbms-subsetter \n\nExample::\n\n rdbms-subsetter postgresql://:@/bigdb postgresql://:@/littledb 0.05\n\nValid SQLAlchemy connection strings are described\n`here `_.\n\n``rdbms-subsetter`` promises that each child row will have whatever parent rows are\nrequired by its foreign keys. It will also *try* to include most child rows belonging\nto each parent row (up to the supplied ``--children`` parameter, default 3 each), but it\ncan't make any promises. (Demanding all children can lead to infinite propagation in\nthoroughly interlinked databases, as every child record demands new parent records,\nwhich demand new child records, which demand new parent records...\nso increase ``--children`` with caution.)\n\nWhen row numbers in your tables vary wildly (tens to billions, for example),\nconsider using the ``-l`` flag, which sets row number targets\nby a logarithmic formula.\nWhen ``-l`` is set, if ``f`` is the fraction specified,\nand the original table has ``n`` rows,\nthen each new table's row target will be::\n\n math.pow(10, math.log10(n)*f)\n\nA fraction of ``0.5`` seems to produce good results, converting 10 rows to 3,\n1,000,000 to 1,000, and 1,000,000,000 to 31,622.\n\nRows are selected randomly, but for tables with a single primary key column, you\ncan force rdbms-subsetter to include specific rows (and their dependencies) with\n``force=:``. The children, grandchildren, etc. of\nthese rows\nare exempted from the ``--children`` limit.\n\n``rdbms-subsetter`` only performs the INSERTS; it's your responsibility to set\nup the target database first, with its foreign key constraints. The easiest\nway to do this is with your RDBMS's dump utility. For example, for PostgreSQL,\n\n::\n\n pg_dump --schema-only -f schemadump.sql bigdb\n createdb littledb\n psql -f schemadump.sql littledb\n\nRows are taken from the schema visible by default to your\ndatabase connection. You can also include rows from non-default schemas\nwith the ``--schema=`` parameter (which can be used multiple times).\nCurrently the target database must contain the corresponding tables in its own\nschema of the same name (moving between schemas of different names is not yet\nsupported).\n\nYou can restrict the tables included in the sample via the ``--table``\n(``-t``) and ``--exclude-table`` (``-T``) parameters (which can be used\nmultiple times). These parameters take a table name or pattern with wildcards\n(``*``), and supports both qualified names (i.e. ``schema.table``) and simple\nnames. When both ``-t`` and ``-T`` are given, the behavior is to include just\nthe tables that match at least one ``-t`` switch, but no ``-T`` switches.\n\nWhen target db write buffering is enabled (which it is by default), subset\nextraction for complex schemas that have tangled foreign key relationships may\nfail. To disable write buffering, set the buffer parameter to 0::\n\n rdbms-subsetter postgresql://:@/bigdb postgresql://:@/littledb 0.05 -b 0\n\nConfiguration file\n------------------\n\nIf you need to honor relationships that aren't actually defined as foreign-key\nconstraints in the database - for example, if you are using MySQL MyISAM\nand can't define constraints - you can specify a\nconfiguration file with ``--config``. The config file should specify constraints\nin JSON. For example,\n\n {\n \"constraints\": {\n \"(child_table_name)\": [\n {\n \"referred_schema\": null,\n \"referred_table\": \"(name of parent table)\",\n \"referred_columns\": [\"(constraint col 1 in parent)\", \"(constraint col 2 in parent)\",],\n \"constrained_columns\": [\"(constrained col 1 in child)\", \"(constrained col 2 in child)\",],\n }\n ],\n },\n \"tables\": [ \"SCHEMA1.TABLE1\", \"SCHEMA2.TABLE2\" ],\n \"schemas\": [ \"SCHEMA1\", \"SCHEMA2\" ]\n }\n\nOptionally, you can qualify ``child_table_name``, i.e.\n``schema_name.child_table_name``. Cross-schema constraints are also supported.\n\n``rdbms-subsetter`` treats these constraints like real foreign keys and fetches\nparent and child rows as described above.\n\n``tables`` and ``schemas`` are optional.\n\n``tables`` are merged with the ``--table`` elements passed on commandline.\n\n``schemas`` are merged with the ``--schema`` elements passed on commandline.\n\n\nSignal handlers\n---------------\nIf you provide a python module with appropriate signal handling functions, and specify that module\nwhen calling the script like ``--import=my.signals.signal_handlers``, then any signal handlers that you\nhave registered in your module will be called when the corresponding signals are sent during\nthe DB subsetting process.\n\nAt the moment, the only signal is ``subsetter.SIGNAL_ROW_ADDED``.\n\nAn example signal handling module::\n\n from blinker import signal\n import subsetter\n\n row_added_signal = signal(subsetter.SIGNAL_ROW_ADDED)\n @row_added_signal.connect\n def row_added(source_db, **kwargs):\n print(\"row_added called with source db: {}, and kwargs: {}\".format(source_db, kwargs))\n\nSIGNAL_ROW_ADDED\n^^^^^^^^^^^^^^^^\nThis signal will be sent when a new row has been selected for adding to the target database.\nThe associated signal handler should have the following signature::\n\n def row_added(source_db, **kwargs):\n\n``source_db`` is a ``subsetter.Db`` instance.\n\n``kwargs`` contains:\n\n- ``target_db``: a ``subsetter.Db`` instance.\n\n- ``source_row``: an ``sqlalchemy.engine.RowProxy`` with the values from the row that will be inserted.\n\n- ``target_table``: an ``sqlalchemy.Table``.\n\n- ``prioritized``: a ``bool`` representing whether of not all child, grandchild, etc. rows should be included.\n\nInstalling\n----------\n\n::\n\n pip install rdbms-subsetter\n\nThen the DB-API2 module for your RDBMS; for example, for PostgreSQL,\n\n::\n\n pip install psycopg2\n\nMemory\n------\n\nWill consume memory roughly equal to the size of the *extracted* database.\n(Not the size of the *source* database!)\n\nDevelopment\n-----------\n\nhttps://github.com/18F/rdbms-subsetter\n\nSee also\n--------\n\n* `Jailer `_\n\n\n", "description_content_type": null, "docs_url": null, "download_url": "", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "https://github.com/18F/rdbms-subsetter", "keywords": "database testing", "license": "CC0", "maintainer": "", "maintainer_email": "", "name": "rdbms-subsetter", "package_url": "https://pypi.org/project/rdbms-subsetter/", "platform": "", "project_url": "https://pypi.org/project/rdbms-subsetter/", "project_urls": { "Homepage": "https://github.com/18F/rdbms-subsetter" }, "release_url": "https://pypi.org/project/rdbms-subsetter/0.2.6/", "requires_dist": [ "blinker", "sqlalchemy" ], "requires_python": "", "summary": "Generate consistent subset of an RDBMS", "version": "0.2.6" }, "last_serial": 3842110, "releases": { "0.1.0": [ { "comment_text": "", "digests": { "md5": "40c008ae75cd7af536b0c719763c1eaa", "sha256": "7184765dfae9916d0070d3ee63f9798a706e0d75e07db63f6ff27353375a611b" }, "downloads": -1, "filename": "rdbms_subsetter-0.1.0-py2.7.egg", "has_sig": false, "md5_digest": "40c008ae75cd7af536b0c719763c1eaa", "packagetype": "bdist_egg", "python_version": "2.7", "requires_python": null, "size": 3090, "upload_time": "2014-11-12T22:55:38", "url": "https://files.pythonhosted.org/packages/5e/47/517305ebfe4d584690bef7f01a21ad8d4aaddae56a3077d15fd5aa6ed30e/rdbms_subsetter-0.1.0-py2.7.egg" }, { "comment_text": "", "digests": { "md5": "a4c12490ee90c3e013a13271e222d72d", "sha256": "adfd7ffa444aa8d11bd01b6b3279a541940b70b5e0a19b90fb44c0337b2623ff" }, "downloads": -1, "filename": "rdbms-subsetter-0.1.0.tar.gz", "has_sig": false, "md5_digest": "a4c12490ee90c3e013a13271e222d72d", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 3414, "upload_time": "2014-11-12T22:55:35", "url": "https://files.pythonhosted.org/packages/e7/24/9bb74aed4b1d01f2a7987372e9da0fb2a88de447f4631c969fd4d35c2d79/rdbms-subsetter-0.1.0.tar.gz" } ], "0.1.1": [ { "comment_text": "", "digests": { "md5": "80d1e44a4d0ea00989b1accb13983f60", "sha256": "6a707ac992ffacf09111c23262619bd1f9f00450c055e47dd21219dac88834f2" }, "downloads": -1, "filename": "rdbms_subsetter-0.1.1-py3.4.egg", "has_sig": false, "md5_digest": "80d1e44a4d0ea00989b1accb13983f60", "packagetype": "bdist_egg", "python_version": "3.4", "requires_python": null, "size": 3183, "upload_time": "2014-11-12T23:36:40", "url": "https://files.pythonhosted.org/packages/d7/e6/927647b2198911b59b5bcfa886350abcd8aecea6b54ecaa7c7d7b9a21729/rdbms_subsetter-0.1.1-py3.4.egg" }, { "comment_text": "", "digests": { "md5": "3ef384ba0c81668bf61a948dd5abeba2", "sha256": "95fdd83d0b121881693d038a3115ec140a9df6b61eac54be4794f1d1b1d459f7" }, "downloads": -1, "filename": "rdbms-subsetter-0.1.1.tar.gz", "has_sig": false, "md5_digest": "3ef384ba0c81668bf61a948dd5abeba2", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 3568, "upload_time": "2014-11-12T23:36:31", "url": "https://files.pythonhosted.org/packages/c0/57/52a8864908ba593605f6bbde953f4231ec004802412db529dd3ef74f6d6d/rdbms-subsetter-0.1.1.tar.gz" } ], "0.1.2": [ { "comment_text": "", "digests": { "md5": "77ade6c02f8e8c1af5cdaf776035453c", "sha256": "b5b8c9e6103552c828b5f2c8e2653ad43f6c43b75b866a5d42e8b61e57ad6457" }, "downloads": -1, "filename": "rdbms_subsetter-0.1.2-py3.4.egg", "has_sig": false, "md5_digest": "77ade6c02f8e8c1af5cdaf776035453c", "packagetype": "bdist_egg", "python_version": "3.4", "requires_python": null, "size": 14721, "upload_time": "2014-11-14T15:02:07", "url": "https://files.pythonhosted.org/packages/7d/e4/ae829aef22f0c27992e8084c09af132f59ae9168da5ba0fbe93a833e3e13/rdbms_subsetter-0.1.2-py3.4.egg" }, { "comment_text": "", "digests": { "md5": "2c5b7544eae0bc27194d045b9eab3110", "sha256": "89fa8fc4a9c76e35366e131eac41e60068f8504c74d8c7173605a56872065dba" }, "downloads": -1, "filename": "rdbms-subsetter-0.1.2.tar.gz", "has_sig": false, "md5_digest": "2c5b7544eae0bc27194d045b9eab3110", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 7030, "upload_time": "2014-11-14T15:02:03", "url": "https://files.pythonhosted.org/packages/d7/69/fd65115ef2ac38dc7cb4e7e0980935f79e7789777bdc8e29dfe9529442f1/rdbms-subsetter-0.1.2.tar.gz" } ], "0.2.0": [ { "comment_text": "", "digests": { "md5": "71649ebaf905a074b3fed1f572051c71", "sha256": "ceabb92f54b63beba4b1dcace2ddbc836f80c2c3c086ace9e2e5700bd8d688f5" }, "downloads": -1, "filename": "rdbms_subsetter-0.2.0-py3.4.egg", "has_sig": false, "md5_digest": "71649ebaf905a074b3fed1f572051c71", "packagetype": "bdist_egg", "python_version": "3.4", "requires_python": null, "size": 14355, "upload_time": "2014-12-05T23:59:25", "url": "https://files.pythonhosted.org/packages/b0/78/b035963a8fa4f692d9f661a29dec3979863eaf127262841641850f0cfc95/rdbms_subsetter-0.2.0-py3.4.egg" }, { "comment_text": "", "digests": { "md5": "70a3ffb5a30438a231f0cddb3d26cd26", "sha256": "d9b3193c01f8c5ad2655e20e3135163b6278020d0a84e7efc8a942ea42674bab" }, "downloads": -1, "filename": "rdbms-subsetter-0.2.0.tar.gz", "has_sig": false, "md5_digest": "70a3ffb5a30438a231f0cddb3d26cd26", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 6651, "upload_time": "2014-12-05T23:59:22", "url": "https://files.pythonhosted.org/packages/0a/50/fbb7685af0cdf0c545b69b7e9d9c056313cbe275375b68cfd4ace125dab6/rdbms-subsetter-0.2.0.tar.gz" } ], "0.2.1": [ { "comment_text": "", "digests": { "md5": "02d2230a512fed94792dda7445ea1619", "sha256": "b8956bc8799f1b5a896b4cbd27a28589c701cc3583b08ac7341b41831a2a47fe" }, "downloads": -1, "filename": "rdbms_subsetter-0.2.1-py3.4.egg", "has_sig": false, "md5_digest": "02d2230a512fed94792dda7445ea1619", "packagetype": "bdist_egg", "python_version": "3.4", "requires_python": null, "size": 15373, "upload_time": "2015-01-04T01:47:13", "url": "https://files.pythonhosted.org/packages/50/db/66eaed3d7564ca203b492a99c7ced26c4de870daf21fd6aab943df7a3a0b/rdbms_subsetter-0.2.1-py3.4.egg" }, { "comment_text": "", "digests": { "md5": "cec2c4910c9f6fecba654a58ec417954", "sha256": "21af5eaeadfc7eb7b8a167e3349bb2667ae64b5cf69b8f3ac22251a478cb268b" }, "downloads": -1, "filename": "rdbms-subsetter-0.2.1.tar.gz", "has_sig": false, "md5_digest": "cec2c4910c9f6fecba654a58ec417954", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 7067, "upload_time": "2015-01-04T01:47:09", "url": "https://files.pythonhosted.org/packages/31/be/c3cbd82a428cc2417e1b8c7e136565615b87ae711737ab724df7ced3e61a/rdbms-subsetter-0.2.1.tar.gz" } ], "0.2.2": [ { "comment_text": "", "digests": { "md5": "87a5c82daa83140f92eaeb5962441935", "sha256": "478b34e9a5f4f8790d27b27917c406701239fbbee915fcebe33f03c5285342c7" }, "downloads": -1, "filename": "rdbms_subsetter-0.2.2-py2.7.egg", "has_sig": false, "md5_digest": "87a5c82daa83140f92eaeb5962441935", "packagetype": "bdist_egg", "python_version": "2.7", "requires_python": null, "size": 16327, "upload_time": "2015-02-07T00:13:18", "url": "https://files.pythonhosted.org/packages/3d/03/344f7f8352a99b24cc962207365a8b7fba4abbc8af0e86a8ce45f23497d5/rdbms_subsetter-0.2.2-py2.7.egg" }, { "comment_text": "", "digests": { "md5": "252e9c7191389700e9adf3c6bca4957f", "sha256": "034575d6db67e29e4057f3811005443fc533efd6948cee18f58d8847ec212246" }, "downloads": -1, "filename": "rdbms-subsetter-0.2.2.tar.gz", "has_sig": false, "md5_digest": "252e9c7191389700e9adf3c6bca4957f", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 7632, "upload_time": "2015-02-07T00:13:15", "url": "https://files.pythonhosted.org/packages/a2/e6/58c4ab840956f500ad985abbb256e1c8c019ef23fb73a414a636dfd06351/rdbms-subsetter-0.2.2.tar.gz" } ], "0.2.3": [ { "comment_text": "", "digests": { "md5": "14d9f5c05b350da4dc52ec69d4fd8425", "sha256": "717522a9507cd6a89be04aa8c0cdc1b901f45509359e20ee48e278d047eef40d" }, "downloads": -1, "filename": "rdbms_subsetter-0.2.3-py3.4.egg", "has_sig": false, "md5_digest": "14d9f5c05b350da4dc52ec69d4fd8425", "packagetype": "bdist_egg", "python_version": "3.4", "requires_python": null, "size": 17737, "upload_time": "2015-05-24T14:35:39", "url": "https://files.pythonhosted.org/packages/69/0d/a62ab37697bff3d8e1a4449bc520406f42212ef6f258b4eefdb059103ff9/rdbms_subsetter-0.2.3-py3.4.egg" }, { "comment_text": "", "digests": { "md5": "2199efff698b3edbe2562a9612a8996a", "sha256": "ff80e1decaab2faf4aff4209041788b5ba147ebec9582a1041d954779c1e5f4d" }, "downloads": -1, "filename": "rdbms-subsetter-0.2.3.tar.gz", "has_sig": false, "md5_digest": "2199efff698b3edbe2562a9612a8996a", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 8364, "upload_time": "2015-05-24T14:35:35", "url": "https://files.pythonhosted.org/packages/55/bc/db61b5ba41e78a8054b391cedb964fa47a4700a0fd160d7f4fcc424785be/rdbms-subsetter-0.2.3.tar.gz" } ], "0.2.5": [ { "comment_text": "", "digests": { "md5": "705beec2933daf3657e358bff8be398e", "sha256": "4460ca1418470f2987c7a38ec33d9c0d7ba5c203bc5253e94f8c724f3e855028" }, "downloads": -1, "filename": "rdbms_subsetter-0.2.5-py3-none-any.whl", "has_sig": false, "md5_digest": "705beec2933daf3657e358bff8be398e", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": null, "size": 15603, "upload_time": "2015-12-14T03:23:58", "url": "https://files.pythonhosted.org/packages/74/93/b4ca29c77602d2bb55d00e6c399649f37c40dad9b3a09860dc70ab039b22/rdbms_subsetter-0.2.5-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "b43c1c4032b7266562b36c1b0fa9af91", "sha256": "e013270b544b90c9f004517603e90c2beb309c6a8af188dd16c6af10709185f1" }, "downloads": -1, "filename": "rdbms-subsetter-0.2.5.tar.gz", "has_sig": false, "md5_digest": "b43c1c4032b7266562b36c1b0fa9af91", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 10976, "upload_time": "2015-12-14T03:24:04", "url": "https://files.pythonhosted.org/packages/fc/ff/579501c1bc7c98613b414fdf72561062bb395d1a68bddd913014a4980439/rdbms-subsetter-0.2.5.tar.gz" } ], "0.2.6": [ { "comment_text": "", "digests": { "md5": "ed8abd33d50b47b0e2f5986a5ccd4a38", "sha256": "9222a5781504b2cf67a4cc2bf6297b346829537da6aa16e6dc94ba5eceed6e6d" }, "downloads": -1, "filename": "rdbms_subsetter-0.2.6-py3-none-any.whl", "has_sig": false, "md5_digest": "ed8abd33d50b47b0e2f5986a5ccd4a38", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": null, "size": 16045, "upload_time": "2017-10-17T21:11:19", "url": "https://files.pythonhosted.org/packages/b9/33/27920601543ea4bca04cfdf0ac1f878f611d559737ee6dd2f24d50cea64c/rdbms_subsetter-0.2.6-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "4338d66b86f72693accad187147b1c13", "sha256": "0aea9ebe778f3d028854e4870a5b8964accd311ad02b5c2e99da83e4b8d64339" }, "downloads": -1, "filename": "rdbms-subsetter-0.2.6.tar.gz", "has_sig": false, "md5_digest": "4338d66b86f72693accad187147b1c13", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 11214, "upload_time": "2017-10-17T21:11:22", "url": "https://files.pythonhosted.org/packages/90/76/8dff046cd3dde86accfbf5d5abb33798a0e840c2781e0fca154536c8f8ec/rdbms-subsetter-0.2.6.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "ed8abd33d50b47b0e2f5986a5ccd4a38", "sha256": "9222a5781504b2cf67a4cc2bf6297b346829537da6aa16e6dc94ba5eceed6e6d" }, "downloads": -1, "filename": "rdbms_subsetter-0.2.6-py3-none-any.whl", "has_sig": false, "md5_digest": "ed8abd33d50b47b0e2f5986a5ccd4a38", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": null, "size": 16045, "upload_time": "2017-10-17T21:11:19", "url": "https://files.pythonhosted.org/packages/b9/33/27920601543ea4bca04cfdf0ac1f878f611d559737ee6dd2f24d50cea64c/rdbms_subsetter-0.2.6-py3-none-any.whl" }, { "comment_text": "", "digests": { "md5": "4338d66b86f72693accad187147b1c13", "sha256": "0aea9ebe778f3d028854e4870a5b8964accd311ad02b5c2e99da83e4b8d64339" }, "downloads": -1, "filename": "rdbms-subsetter-0.2.6.tar.gz", "has_sig": false, "md5_digest": "4338d66b86f72693accad187147b1c13", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 11214, "upload_time": "2017-10-17T21:11:22", "url": "https://files.pythonhosted.org/packages/90/76/8dff046cd3dde86accfbf5d5abb33798a0e840c2781e0fca154536c8f8ec/rdbms-subsetter-0.2.6.tar.gz" } ] }