{ "info": { "author": "Daniele Varrazzo", "author_email": "daniele.varrazzo@gmail.com", "bugtrack_url": null, "classifiers": [ "Development Status :: 3 - Alpha", "License :: OSI Approved :: BSD License", "Programming Language :: Python :: 2.7", "Topic :: Database" ], "description": "The project is currently in pre-production phase. We are hacking on it.\n\n**replisome** is an output plugin for `PostgreSQL logical decoding`__ Using\nthe plugin a client can receive a stream of changes describing the data\nmanipulation inside the database (INSERT, UPDATE, DELETE of records) for all\nor a specified subset of tables, with the possibility of limiting the columns\nand rows received. Changes are received in JSON format.\n\n.. __: https://www.postgresql.org/docs/current/static/logicaldecoding-explanation.html\n\n**replisome** doesn't try to be a complete replication solution: take a look\nat pglogical_ for that. Things you can do with it:\n\n* receive data from the database in a readable way, not for replication (e.g.\n for logging, message passing to different systems...)\n* integrate data update from a database into a receiving database with an\n arbitrary different schemas.\n\n**replisome** doesn't need persistent configuration in the database sending\nthe changes, apart from the creation of a logical replication slot. The\nconfiguration, i.e. what records to send, is entirely chosen by the client\nconnecting. Changing configuration is as simple as stopping the replica and\nrestarting it with different parameters (the data stream will recover from\nwhere previously interrupted).\n\n**replisome** is released under PostgreSQL license.\n\n.. _pglogical: https://www.2ndquadrant.com/en/resources/pglogical/\n\n\nRequirements\n============\n\n* PostgreSQL 9.4+\n\n\nBuild and Install\n=================\n\nThis thing will be packaged as an extension, have a version number, be\nreleased on PGXN... but currently it is only available on github::\n\n $ git clone https://github.com/dvarrazzo/wal2json.git -b replisome replisome\n\n\nUnix based Operating Systems\n----------------------------\n\nThe extension should be compiled and installed in a PostgreSQL installation,\nafter which it will be available in the database clusters run by that\ninstallation.\n\nIn order to build the extension you will need a C compiler, the PostgreSQL\nserver development packages and maybe something else that google will friendly\ntell you. ::\n\n $ git clone https://github.com/dvarrazzo/wal2json.git -b replisome replisome\n $ cd replisome\n $ export PATH=\n $ make PG_CONFIG=/path/to/bin/pg_config\n $ sudo make PG_CONFIG=/path/to/bin/pg_config install\n\n\nWindows\n-------\n\nIn a world without walls you don't need windows. But if you happen to do, try\nit and send some patches back.\n\n\nConfiguration\n=============\n\nYou need to set up at least two parameters into ``postgresql.conf``::\n\n wal_level = logical\n max_replication_slots = 1\n\nAfter changing these parameters, a restart is needed.\n\n\nExamples\n========\n\nThere are a few ways to obtain the changes (JSON objects) from the\n**replisome** plugin:\n\n* using `SQL functions`__ such as ``pg_logical_slot_get_changes()``\n* using pg_recvlogical__ from command line.\n* using `psycopg replication protocol support`__.\n\n.. __: https://www.postgresql.org/docs/9.4/static/functions-admin.html#FUNCTIONS-REPLICATION-TABLE\n.. __: https://www.postgresql.org/docs/current/static/app-pgrecvlogical.html\n.. __: http://initd.org/psycopg/docs/advanced.html#replication-protocol-support\n\n\nExamples using ``pg_recvlogical``\n---------------------------------\n\nBesides the configuration above, it is necessary to configure a replication\nconnection to use ``pg_recvlogical``.\n\nFirst, add an entry into ``pg_hba.conf``::\n\n local replication myuser trust\n host replication myuser 10.1.2.3/32 trust\n\nAlso, set ``max_wal_senders`` into ``postgresql.conf``::\n\n max_wal_senders = 1\n\nA restart is necessary if you change ``max_wal_senders``.\n\nYou are ready to try replisome. In one terminal create a replication slot and\nstart a replica::\n\n $ pg_recvlogical -d postgres --slot test_slot --create-slot -P replisome\n $ pg_recvlogical -d postgres --slot test_slot --start -o pretty-print=1 -f -\n\nIn another terminal connect to the database and enter some commands::\n\n =# create table test (\n id serial primary key, data text, ts timestamptz default now());\n CREATE TABLE\n\n =# insert into test default values;\n INSERT 0 1\n =# insert into test (data) values ('hello');\n INSERT 0 1\n\n =# begin;\n BEGIN\n *=# update test set data = 'world' where id = 2;\n UPDATE 1\n *=# delete from test where id = 1;\n DELETE 1\n *=# commit;\n COMMIT\n\n\nThe streaming connection should display a description of the operations\nperformed::\n\n {\n \"change\": [\n {\n \"op\": \"I\",\n \"schema\": \"public\",\n \"table\": \"test\",\n \"colnames\": [\"id\", \"data\", \"ts\"],\n \"coltypes\": [\"int4\", \"text\", \"timestamptz\"],\n \"values\": [1, null, \"2017-05-13 13:15:28.052318+01\"]\n }\n ]\n }\n {\n \"change\": [\n {\n \"op\": \"I\",\n \"schema\": \"public\",\n \"table\": \"test\",\n \"values\": [2, \"hello\", \"2017-05-13 13:15:35.140594+01\"]\n }\n ]\n }\n {\n \"change\": [\n {\n \"op\": \"U\",\n \"schema\": \"public\",\n \"table\": \"test\",\n \"values\": [2, \"world\", \"2017-05-13 13:15:35.140594+01\"],\n \"keynames\": [\"id\"],\n \"keytypes\": [\"int4\"],\n \"oldkey\": [2]\n }\n ,{\n \"op\": \"D\",\n \"schema\": \"public\",\n \"table\": \"test\",\n \"oldkey\": [1]\n }\n ]\n }\n\n\nOptions\n=======\n\nThe plugin output content and format is configured by several options passed\nto the START_REPLICATION__ command (e.g. using the ``-o`` option of\n``pg_recvlogical``, the psycopg `start_replication()`__ method etc.\n\n.. __: https://www.postgresql.org/docs/9.4/static/protocol-replication.html\n.. __: http://initd.org/psycopg/docs/extras.html#psycopg2.extras.ReplicationCursor.start_replication\n\n``pretty-print`` [``bool``] (default: ``false``)\n Add whitespaces in the output for readibility.\n\n``include`` [``json``]\n Choose what tables and what content to see of these tables. The command,\n together with ``exclude``, can be used several times: each table will be\n considered for inclusion or exclusion by matching it against all the\n commands specified in order. The last matching command will take effect\n (e.g. you may exclude an entire schema and then include only one specific\n table into it).\n\n The parameter is a JSON object which may contain the following keys:\n\n - ``table``: match a table with this name, in any schema\n - ``tables``: match all the tables whose name matches a regular\n expression, in any schema\n - ``schema``: match all the tables in a schema\n - ``schemas``: match all the tables in all the schemas whose name matches\n a regular expression\n\n These keys will establish if a table matches or not the configuration\n object. At least a schema or a table must be specified. The following\n options can be specified too, and they will affect any table whose\n inclusion is decided by the object:\n\n - ``columns``: only emit the columns specified (as a JSON array)\n - ``skip_columns``: don't emit the columns specified\n - ``where``: only emit the row matching the condition specified as a SQL\n expression matching the table columns, like in a ``CHECK`` clause.\n\n Example (as ``pg_recvlogical`` option)::\n\n -o ' {\"tables\": \"^test.*\", \"skip_columns\": [\"ts\", \"wat\"], \"where\": \"id % 2 = 0\"}'\n\n``exlcude`` [``json``]\n Choose what table to exclude. The format is the same of ``include`` but\n only the tables/schems can be specified, no rows or columns.\n\n``include-xids`` [``bool``] (default: ``false``)\n If ``true``, include the id of each transaction::\n\n {\n \"xid\": 5360,\n \"change\": [\n { ...\n\n``include-lsn`` [``bool``] (default: ``false``)\n Include the Log Sequence Number of the transaction::\n\n {\n \"nextlsn\": \"0/3784C40\",\n \"change\": [\n { ...\n\n\n``include-timestamp`` [``bool``] (default: ``false``)\n Include the commit time of the transaction::\n\n {\n \"timestamp\": \"2017-05-13 03:19:29.828474+01\",\n \"change\": [\n { ...\n\n``include-schemas`` [``bool``] (default: ``true``)\n Include the schema name of the tables.\n\n``include-types`` [``bool``] (default: ``true``)\n Include the types of the table columns.\n\n``include-empty-xacts`` [``bool``] (default: ``false``)\n If ``true``, send information about transactions not containing data\n changes (e.g. ones only performing DDL statements. Only the metadata (e.g.\n time, txid) of the transaction are sent.\n\n``write-in-chunks`` [``bool``] (default: ``false``)\n If ``true``, data may be sent in several chunks instead of a single\n message for the entire transaction. Please note that a single chunk may\n not be a valid JSON document and the client is responsible to aggregate\n the parts received.\n\n\nLicense\n=======\n\n| Copyright (c) 2013-2017, Euler Taveira de Oliveira\n| Copyright (c) 2017, Gambit Research Ltd.\n| All rights reserved.\n\nRedistribution and use in source and binary forms, with or without\nmodification, are permitted provided that the following conditions are met:\n\n* Redistributions of source code must retain the above copyright notice,\n this list of conditions and the following disclaimer.\n\n* Redistributions in binary form must reproduce the above copyright notice,\n this list of conditions and the following disclaimer in the documentation\n and/or other materials provided with the distribution.\n\n* Neither the name of Gambit Research Ltd. nor the names of its contributors\n may be used to endorse or promote products derived from this software\n without specific prior written permission.\n\nTHIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS \"AS IS\"\nAND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\nIMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE\nARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE\nLIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR\nCONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF\nSUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS\nINTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN\nCONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)\nARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE\nPOSSIBILITY OF SUCH DAMAGE.", "description_content_type": null, "docs_url": null, "download_url": "UNKNOWN", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "https://github.com/dvarrazzo/wal2json/tree/replisome", "keywords": "database,replication,PostgreSQL", "license": "UNKNOWN", "maintainer": null, "maintainer_email": null, "name": "replisome", "package_url": "https://pypi.org/project/replisome/", "platform": "UNKNOWN", "project_url": "https://pypi.org/project/replisome/", "project_urls": { "Download": "UNKNOWN", "Homepage": "https://github.com/dvarrazzo/wal2json/tree/replisome" }, "release_url": "https://pypi.org/project/replisome/0.0.1/", "requires_dist": null, "requires_python": null, "summary": "replisome - handsomely replicate something", "version": "0.0.1" }, "last_serial": 2876169, "releases": { "0.0.1": [] }, "urls": [] }