{
    "info": {
        "author": "Vincent Pelletier",
        "author_email": "vincent@nexedi.com",
        "bugtrack_url": null,
        "classifiers": [
            "Framework :: Zope2",
            "Operating System :: OS Independent"
        ],
        "description": "TIDStorage\n==========\n\nSYNOPSIS\n--------\n\nThis product provides a way to have consistent backups when running a\nmulti-storage instance (only ZEO is supported at the moment).\n\nDoing backups of individual Data.fs who are part of the same instance\n(one mounted in another) is a problem when there are transactions involving\nmultiple storages: if there is a crash during transaction commit, there is no\nway to tell which storage was committed and which was not (there is no TID\nconsistency between databases).\nThere is an even more tricky case. Consider the following::\n\n 2 transactions running in parallel:\n  T1: modifies storage A and B\n  T2: modifies storage A\n Commit order scenario:\n  T1 starts committing (takes commit lock on A and B)\n  T2 starts committing (waits for commit lock on A)\n  T1 commits A (commit lock released on A)\n  T2 commits A (takes & releases commit lock on A)\n  [crash]\n  T1 commits B (commit lock released on B) <- never happens because of crash\n\nHere, T2 was able to commit entirely, but it must not be saved. This is\nbecause transactions are stored in ZODB in the order they are committed.\nSo is T2 is in the backup, a part of T1 will also be, and backup will be\ninconsistent (T1 commit on B never happened).\n\nTIDStorage log and server log\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\nTIDStorage uses two logfiles - one which is used to inform administrator\nabout server state (logfile_name in configuration) and TIDStorage log to which\nTIDs are appended (status_file in configuration).\n\nUSAGE\n-----\n\nPut product in Zope Products to activate Zope-side patches.\n\nCreate configuration, example is provided in repozo/sample_configuration.py\n\nRun bin/tidstorage.py with created configuration. As Zope commits transations\nit will connect to TIDStorage server, which will be shown in Zope and TIDStorage\nserver logs.\n\nZope 2.12+ configuration\n~~~~~~~~~~~~~~~~~~~~~~~~\n\nPut in zope.conf section::\n\n  <product-config TIDStorage>\n    backend-ip ip-of-tidstorage-server\n    backend-port port-of-tidstorage-server\n  </product-config>\n\n\nPYTHONPATH issues\n~~~~~~~~~~~~~~~~~\n\nTo run server and scripts there is a need to set correct PYTHONPATH - at least\nto product directory and for some tools to Zope lib/python.\n\nExample:\n\nPYTHONPATH=/usr/lib/erp5/lib/python:/usr/lib/erp5/lib/python/Products/TIDStorage\n\nTypical scenario with failure, restoring from backup\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n * Zopes and Zeos running\n * TIDStorage running\n * backups done using repozo/repozo_tidstorage.py (they might contain\n   incoherency), for every backup tidstorage.tid is saved\n * system failure\n * restore using repozo/repozo_tidstorage.py with -t tidstorage.tid from last\n   backup\n\nIn this scenario only on restoration destination file is cut at point of last\nknown TID position. This step is optional, as in some cases administrator\nmight want to not cut this file.\n\nTypical scenario with failure, no restoring needed\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n\n * Zopes and Zeos running\n * TIDStorage running\n * system failure\n * no need to restore from backup, but there might be some laying transactions\n   in different ZODB files, system is incoherent\n * administrator use repozo/restore_tidstorage.py to cut not correctly commited\n   transactions, system is coherent again\n\nTECHNICAL DETAILS\n-----------------\n\nTIDStorage fixes those issues by keeping track of transaction-to-tid relations\nfor all (ZODB, via ZEO) storages involved in any transaction, and by tracking\ninter-transaction dependencies.\n\nTIDStorage is composed of 3 parts:\n - A Zope product, which monkey-patches \"ZEO\" and \"transaction\" products.\n\n   transaction patch\n\n     TIDStorage works at transaction boundaries, so we hook around\n     _commitResource method to know when it happens.\n     It must be configured to fit your network setup (TID_STORAGE_ADDRESS)\n\n   ZEO patch\n\n     With regular ZEO, there is no way to know last committed TID at\n     transaction-code level. This patch stores last committed TID on ZEO\n     connection object, to be read by transaction patch.\n\n - A daemon\n   This is TIDStorage itself, receiving TIDs from Zopes and delivering\n   coherency points to backup scripts.\n - Backup scripts and other utilities\n   Those scripts are (mostly) wrappers for repozo backup script, fetching\n   coherency points from TIDStorage daemon and invoking repozo.\n   No changes to repozo.py are needed, as it is used only as subsystem\n   to do reliable backups and restore.\n   Using provided utils in utils/ directory is it possible to query\n   for last known TID from server and operate on TIDStorage log.\n\nConstraints under which TIDStorage was designed:\n - Zope performance\n   Protocol (see below) was designed as one-way only (Zope pushes data to\n   TIDStorage, and does not expect an answer), so that TIDStorage speed do not\n   limit Zope performance.\n - No added single-point-of-failure\n   Even if Zope cannot connect to TIDStorage, it will still work. It will only\n   emit one log line when connection is lost or at first attempt if it did not\n   succeed. When connection is established, another log line is emitted.\n - Bootstrap\n   As TIDStorage can be started and stopped while things still happen on\n   ZODBs, it must be able to bootstrap its content before any backup can\n   happen. This is done by creating artificial Zope transactions whose only\n   purpose is to cause a commit to happen on each ZODB, filling TIDStorage and\n   making sure there is no pending commit on any storage (since all locks\n   could be taken by those transactions, it means that all transaction started\n   before that TIDStorage can receive their notification have ended).\n - Restoration from Data.fs\n   In addition to the ability to restore from repozo-style backups, and in\n   order to provide greater backup frequency than repozo can offer on big\n   databases, TIDStorage offers the possibility to restore coherent Data.fs\n   from crashed ones - as long as they are not corrupted.\n\nLimits:\n - Restore \"lag\"\n   As TIDStorage can only offer a coherency point when inderdependent\n   transactions are all finished (committed or aborted), TIDStorage log file\n   backup from time T might actually contain data from moments before.\n   So while doing restore with -t option data will be cut to state as\n   time T - undefined, small lag.\n\n   There are even pathologic cases where no coherency point can be found,\n   so TIDStorage log file won't have any information.\n\nDaemeon signal support:\n - HUP repoens all log files\n - USR1 dumps tid configuration into log file\n - TERM kills daemon\n\nPROTOCOL SPECIFICATION\n----------------------\n\n All characters allowed in data, except \\n and \\r (0x0A & 0x0D).\n Each field ends with \\n, \\r is ignored.\n No escaping.\n When transferring a list, it is prepended by the number of included fields.\n Example::\n\n     3\\n\n     foo\\n\n     bar\\n\n     baz\\n\n\n When transferring a dict, it is prepended by the number of items, followed by\n keys and then values. Values must be integers represented as strings.\n Example::\n\n     2\\n\n     key1\\n\n     key2\\n\n     1\\n\n     2\\n\n\n Commands are case-insensitive.\n\n1) Start of commit command:\n\nBEGIN\\n\n<commit id>\\n\n<list of involved storages>\n\n <commit id>: must be identical to the one given when commit finishes (be it ABORT or COMMIT)\n <list of involved storages>: list of storage ids involved in the transaction\n NB: final \\n is part of list representation, so it's not displayed above.\n\nResponse: (nothing)\n\n2) Transaction abort command:\n\nABORT\\n\n<commit id>\\n\n\n  <commit id>: (cf. BEGIN)\n\nResponse: (nothing)\n\n3) Transaction finalisation command:\n\nCOMMIT\\n\n<commit id>\\n\n<dict of involved storages and committed TIDs>\n\n <commit id>: (cf. BEGIN)\n involved storages: (cf. BEGIN)\n committed TIDs: TIDs for each storage, as int.\n NB: final \\n is part of list representation, so it's not displayed above.\n\nResponse: (nothing)\n\n4) Data read command:\n\nDUMP\\n\n\nResponse:\n<dict of storages and TIDs>\n\n5) Connection termination command:\n\nQUIT\\n\n\nResponse: (nothing, server closes connection)\n\n6) Bootstrap status command:\n\nBOOTSTRAPED\\n\n\nResponse: 1 if bootstrap was completely done, 0 otherwise.\n\nChange History\n==============\n\n5.5.0 (2019-06-21)\n------------------\n\n * Do not depend on Medusa HTTP server. This requires ERP5 > 2018-04-26\n   (commit 6d74ba22a962d08624ef35342708f333a21ceff5).\n\n5.4.9 (2013-09-12)\n------------------\n\n * keep only the latest index files saved by recent repozo\n\n5.4.8 (2012-07-31)\n------------------\n\n * drop base_url parameter [Romain Courteaud]\n * add missing __init_.py files [Romain Courteaud]\n\n5.4.7 (2012-07-30)\n------------------\n\n * drop circular dependency with Zope [Romain Courteaud]\n * <product-config tidstorage> section is supported in zope.conf [\u0141ukasz Nowak]\n * write always pidfile, even if not forking [\u0141ukasz Nowak]",
        "description_content_type": "",
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://svn.erp5.org/repos/public/erp5/trunk/utils/Products.TIDStorage/",
        "keywords": "Zope TIDStorage",
        "license": "",
        "maintainer": "",
        "maintainer_email": "",
        "name": "Products.TIDStorage",
        "package_url": "https://pypi.org/project/Products.TIDStorage/",
        "platform": "",
        "project_url": "https://pypi.org/project/Products.TIDStorage/",
        "project_urls": {
            "Homepage": "https://svn.erp5.org/repos/public/erp5/trunk/utils/Products.TIDStorage/"
        },
        "release_url": "https://pypi.org/project/Products.TIDStorage/5.5.0/",
        "requires_dist": null,
        "requires_python": "",
        "summary": "TIDStorage Product provides a way to have consistent backups when running a multi-storage instance (only ZEO is supported at the moment).",
        "version": "5.5.0"
    },
    "last_serial": 5432333,
    "releases": {
        "5.4.7": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "22d91b22ab727ae4df123af69f568e84",
                    "sha256": "85787f20e5fed1e8f35eb4f6cc0bd467c12cab06b933d667b60b9f3e9fd57dae"
                },
                "downloads": -1,
                "filename": "Products.TIDStorage-5.4.7.tar.gz",
                "has_sig": false,
                "md5_digest": "22d91b22ab727ae4df123af69f568e84",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 19355,
                "upload_time": "2012-07-30T17:48:47",
                "url": "https://files.pythonhosted.org/packages/e8/13/1679a8d4afaac67d443d88b146ac3ef6162cbd6458dbdea3ceac493541df/Products.TIDStorage-5.4.7.tar.gz"
            }
        ],
        "5.4.7.dev-r45824": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "b885835564077db4fd2c663d349d2bff",
                    "sha256": "fa003c46b05c751f9ac4000038d451f871f4ae0ec916ad140aa7991363b13735"
                },
                "downloads": -1,
                "filename": "Products.TIDStorage-5.4.7.dev-r45824.tar.gz",
                "has_sig": false,
                "md5_digest": "b885835564077db4fd2c663d349d2bff",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 32026,
                "upload_time": "2011-05-04T14:46:19",
                "url": "https://files.pythonhosted.org/packages/fa/01/951baa07d10d9d86c67f01ab4ea7edcc554f244ead67ca48330915373d31/Products.TIDStorage-5.4.7.dev-r45824.tar.gz"
            }
        ],
        "5.4.7.dev-r45831": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "e2e4afa3f2fc1d9f3829cf14bdee203b",
                    "sha256": "496fac294c887484fc1a30cc32200af7d707ef68f1c855b87a7ce907189b7585"
                },
                "downloads": -1,
                "filename": "Products.TIDStorage-5.4.7.dev-r45831.tar.gz",
                "has_sig": false,
                "md5_digest": "e2e4afa3f2fc1d9f3829cf14bdee203b",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 32193,
                "upload_time": "2011-05-04T15:36:44",
                "url": "https://files.pythonhosted.org/packages/1f/02/fa4a8df9562b34912b1a850f419f1f50c3974da286e59a331b4554672de9/Products.TIDStorage-5.4.7.dev-r45831.tar.gz"
            }
        ],
        "5.4.7.dev-r45842": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "b3cd7bf66bc7985553cbbf27ea837c13",
                    "sha256": "de77a0f73008cbf8c1f67340411ffde88132194107724a540101aa84e7fc1498"
                },
                "downloads": -1,
                "filename": "Products.TIDStorage-5.4.7.dev-r45842.tar.gz",
                "has_sig": false,
                "md5_digest": "b3cd7bf66bc7985553cbbf27ea837c13",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 32258,
                "upload_time": "2011-05-05T10:53:45",
                "url": "https://files.pythonhosted.org/packages/bc/be/e67463ab57598d7b2e0127a146f4a90e35d20f6ef5e07316f7459f87b690/Products.TIDStorage-5.4.7.dev-r45842.tar.gz"
            }
        ],
        "5.4.8": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "83782c60804480c34d7317ee1f9cb762",
                    "sha256": "c67cfb20e6989d2b573db761fe0f6c03ab31b94d2e74a74826117318c213dd87"
                },
                "downloads": -1,
                "filename": "Products.TIDStorage-5.4.8.tar.gz",
                "has_sig": false,
                "md5_digest": "83782c60804480c34d7317ee1f9cb762",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 29680,
                "upload_time": "2012-07-31T13:30:13",
                "url": "https://files.pythonhosted.org/packages/0a/3a/1cc936b1b00832ce929b432780272fe9a2871c392fa204f6b8f59fa95258/Products.TIDStorage-5.4.8.tar.gz"
            }
        ],
        "5.4.9": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "5c35e99b971353234273f220a01a9971",
                    "sha256": "7112fd2434e463300fc95d2b302c3b6aa9fefcda481117f3a9383262aadb3328"
                },
                "downloads": -1,
                "filename": "Products.TIDStorage-5.4.9.tar.gz",
                "has_sig": true,
                "md5_digest": "5c35e99b971353234273f220a01a9971",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 28552,
                "upload_time": "2013-09-16T15:49:21",
                "url": "https://files.pythonhosted.org/packages/93/09/a17e39d395a2618b8699f16589de62a350ddd9e1881c3ad5c7f45c14f901/Products.TIDStorage-5.4.9.tar.gz"
            }
        ],
        "5.5.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "f32df2a6f330116d60b6929af6080ad0",
                    "sha256": "1a8b97af3c751715a0007d23e254c2b750b64eef3678707e529c3201d9891774"
                },
                "downloads": -1,
                "filename": "Products.TIDStorage-5.5.0.tar.gz",
                "has_sig": true,
                "md5_digest": "f32df2a6f330116d60b6929af6080ad0",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 28092,
                "upload_time": "2019-06-21T18:48:20",
                "url": "https://files.pythonhosted.org/packages/a4/87/9ebac8e649cd0531e75a5bddad758a2a732fe7ee3d1f35de3ea46ee01f94/Products.TIDStorage-5.5.0.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "f32df2a6f330116d60b6929af6080ad0",
                "sha256": "1a8b97af3c751715a0007d23e254c2b750b64eef3678707e529c3201d9891774"
            },
            "downloads": -1,
            "filename": "Products.TIDStorage-5.5.0.tar.gz",
            "has_sig": true,
            "md5_digest": "f32df2a6f330116d60b6929af6080ad0",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 28092,
            "upload_time": "2019-06-21T18:48:20",
            "url": "https://files.pythonhosted.org/packages/a4/87/9ebac8e649cd0531e75a5bddad758a2a732fe7ee3d1f35de3ea46ee01f94/Products.TIDStorage-5.5.0.tar.gz"
        }
    ]
}