{ "info": { "author": "UNKNOWN", "author_email": "UNKNOWN", "bugtrack_url": null, "classifiers": [], "description": "PostgreSQL failover managed by ZooKeeper\n========================================\n\nLeverage the co-ordination capability of ZooKeeper\n(http://zookeeper.apache.org) to provide a robust implementation of automated\nfailover for PostgreSQL.\n\nBasic Design\n------------\n\nAll inter-server communication occurs via ZooKeeper. No SSH or network\nconnections of any kind.\n\nThere are 2 different daemons which make up the system: \"deadman\" and\nthe \"sync\" daemons. On the nodes which actually use the PostgreSQL\ncluster, the \"deadman\" daemon is run to control the PostgreSQL process\nand optionally the sync daemon to update PostgreSQL configuration files\n(e.g. pg\\_hba.conf). On application servers or proxies the \"sync\" daemon\nis run.\n\nTerminology\n-----------\n\n- database cluster or cluster:\n http://www.postgresql.org/docs/9.4/static/creating-cluster.html\n- znode: a ZooKeeper data node in the ZooKeeper database\n- group, cluster group, database group: A group of master/replica providing one logical\n database, made up of one or more database clusters.\n\n\nZooKeeper Directory Layout\n--------------------------\n\nIn ZooKeeper, we have one directory which contains the information on\nall the nodes in multiple master-replica database groups. It is a\ndirectory with 4 sub-directories types:\n\n /state/{DATABASE_GROUP_NAME}_{IPADDR}:{PORT}\n This contains all the information about a database cluster nodes,\n healthy or not. It is updated frequently with data such as the WAL log\n replay status. It is an ephemeral node and will dissapear if\n the connection to zookeeper is lost.\n Ephemeral, created/maintained by the \"deadman\" daemon.\n\n /conn/{DATABASE_GROUP_NAME}_{IPADDR}:{PORT}\n This znode contains a subset of information from the state-\n node. It is the static connection information/metadata about a single\n healthy (i.e. connectable) cluster. If the node is not \"healthy\", this\n entry will not exist. The information in this znode is not vollatile and\n is gaurenteed not to change over the lifespan of the znode. Ephemeral,\n created/maintained by the \"deadman\" daemon.\n\n /lock/{DATABASE_GROUP_NAME}\n This contains the IPADDR:PORT of the current master for the\n database group. Connection info should be looked up in the\n \"_conn_\" node (if it exists).\n Created/maintained by the \"deadman\" daemon on the current\n\n /static/{DATABASE_GROUP_NAME}-db-id\n Contains the database identifier of the database group.\n\nMost of the above znodes contain a JSON encoded\ndictionary/object/hashmap.\n\nSync Daemon\n-----------\n\nThis daemon runs on any node which wishes to connect to a database group\nand maintains the local machine's configuration files. For example, it\ncan rewrite a pgbouncer configuration if the master of one database\ngroup fails over. It can also start a new pgbouncer daemon if a new\ndatabase group is created. Another example is dynamically changing\nHAProxy weight according to the node location (e.g. availability zone)\nor replication lag.\n\nActually applying the configuration changes is the job of plugins, the\nsync daemon will not apply any changes by itself. Plugins can be\nspecified in 2 ways:\n\n * Using setuptools entry points to subscribe in-process to the\n changes. This allows subscribers to subscribe to events from\n either state-, master- or healthy- znodes.\n * Provide an executable which will be called with the path to a\n JSON encoded file containing the information from the healthy-\n and master- znodes. This is provided by the zgres-apply package\n which plugs into zgres-sync using the previous plugin method.\n This plugin does not recieve state- events for performance\n reasons.\n\nThese plugins MUST be idempotent, they will be called repeatedly with\nthe same data.\n\nDeadman Daemon\n--------------\n\nThis daemon controls one PostgreSQL database cluster and registers it in\nzookeeper (creating/maintaining the state-, conn- and master-\nznodes). It must run on the same machine as the database cluster.\n\nIt is responsible for promoting or shutting down it's postgresql\ndatabase cluster.\n\nCurrently, remastering and starting PostgreSQL should be handled outside\nbefore deadman is started.\n\nPlugins for the deadman daemon should be able to do 2 things:\n * Provide extra metadata (i.e. availability-zone or replication lag)\n * Veto the cluster being up (aside from a builtin SELECT 1)\n\nChallenges\n----------\n\n * multiple replicas: How to fail over to the \"best\" replica.\n * multiple replicas: How to connect replicas to new master.\n * Would it be a good idea to know where the basebackups are and initialize new replicas on startup?\n\n * PLUGINS PLUGINS PLUGINS:\n - Provide a lot of plugins builtin, allow them to be seen and enabled via the\n \"config\" in zookeeper? EEEK: dynamic reconfiguration of daemons?\n - What happens if a few nodes don't have some plugins?\n - Configuration on a \"cluster group\" level\n\nThe Good\n--------\n\n * Relatively simple configuration. No ssh config.\n * Fully distributed.\n\nImplementation Thoughts\n-----------------------\n\n * Implement daemons in python, log to stdout and stderr. Have them be\n run via systemd with configured to restart on fail. Fail noisily!\n\nDependencies\n------------\n\n * systemd\n * kazoo - ZooKeeper client\n * psycopg2 - connections to PostgreSQL\n\nAcknowledgment\n--------------\n\nZgres is heavily influenced by HandyRep\n(https://github.com/pgexperts/handyrep) and Patroni\n(https://github.com/zalando/patroni). Many thanks to the developers of\nthose for some great ideas.\n", "description_content_type": null, "docs_url": null, "download_url": "UNKNOWN", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "UNKNOWN", "keywords": null, "license": "UNKNOWN", "maintainer": null, "maintainer_email": null, "name": "zgres", "package_url": "https://pypi.org/project/zgres/", "platform": "UNKNOWN", "project_url": "https://pypi.org/project/zgres/", "project_urls": { "Download": "UNKNOWN", "Homepage": "UNKNOWN" }, "release_url": "https://pypi.org/project/zgres/0.4/", "requires_dist": null, "requires_python": null, "summary": "Database Connection and failover manager for PostgreSQL", "version": "0.4" }, "last_serial": 2241924, "releases": { "0.4": [ { "comment_text": "", "digests": { "md5": "a6a9db542fb18c730fbb4d33ecda2621", "sha256": "8d187055b32dd0159520a7c77a6938ff6bcd248025d7807c5cd5b28b774de052" }, "downloads": -1, "filename": "zgres-0.4.tar.gz", "has_sig": false, "md5_digest": "a6a9db542fb18c730fbb4d33ecda2621", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 47957, "upload_time": "2016-07-25T08:26:04", "url": "https://files.pythonhosted.org/packages/32/c3/a53c902e0cea5df1c47f8e26217af60c4f1e31e8c4d265ae558cbfb72241/zgres-0.4.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "a6a9db542fb18c730fbb4d33ecda2621", "sha256": "8d187055b32dd0159520a7c77a6938ff6bcd248025d7807c5cd5b28b774de052" }, "downloads": -1, "filename": "zgres-0.4.tar.gz", "has_sig": false, "md5_digest": "a6a9db542fb18c730fbb4d33ecda2621", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 47957, "upload_time": "2016-07-25T08:26:04", "url": "https://files.pythonhosted.org/packages/32/c3/a53c902e0cea5df1c47f8e26217af60c4f1e31e8c4d265ae558cbfb72241/zgres-0.4.tar.gz" } ] }