{
    "info": {
        "author": "Stijn De Weirdt;Kenneth Waegeman",
        "author_email": "stijn.deweirdt@ugent.be;Kenneth.Waegeman@UGent.be",
        "bugtrack_url": null,
        "classifiers": [
            "License :: OSI Approved :: GNU Lesser General Public License v2 or later (LGPLv2+)"
        ],
        "description": "# Description\nvsc-zk contains scripts to build zookeeper trees and tools using zookeeper.\n\n\n\nvsc-zk\n======\n\n## zookeeper tools\n\nvsc-zk contains scripts to build zookeeper trees and tools using zookeeper.\n\n *  zkinitree builds a tree in zookeeper from a config file.\n \n *  vsc-zkrsync uses zookeeper to distribute rsync across multiple nodes \nwhen rsyncing (large) (shared) filesystems.\n\n    A large fraction of rsync calls is spent in gathering the required metadata, \nbefore performing the actual data synchronisation. This issue becomes worse in the case of an incremental rsync. \nDistributing the load across multiple processes may lead to a significant performance gain \neven though other bottlenecks, such as access to the metadata, can become apparent.\n\n    The implementation uses zookeeper to coordinate the distribution of collecting \nand sync the data across many processes and/or nodes.\n\n\n## Installation of Zookeeper \n\n\nA zookeeper server with proper ACLs to a base znode (zookeeper node) is required.\n\nIf no such server is available, installation can be performed as follows:\n~~~~bash\nzkversion=3.4.6\nbasepath=`mktemp -d` # not permanent, choose other directory as desired\n\nmkdir -p $basepath/zk/server/data\ncd $basepath/zk/server\n# use another mirror if needed\nwget http://apache.belnet.be/zookeeper/zookeeper-$zkversion/zookeeper-$zkversion.tar.gz\ntar xzf zookeeper-$zkversion.tar.gz\n\nrm -f zkServer.sh\ncat > zkServer.sh <<EOF\n#!/bin/bash\n./zookeeper-$zkversion/bin/zkServer.sh \\$@\nEOF\nchmod +x zkServer.sh\n\ncat > zk.conf <<EOF\ntickTime=2000\ndataDir=$PWD/data\nclientPort=2181\nEOF\n~~~~\n\nStart zookeeper server\n\n    `./zkServer.sh start $PWD/zk.conf `\n\n\nNote: You should have Java (JDK 6 or greater) installed to run Zookeeper.\nMore information can be found in the Zookeeper documentation. \n\n## Dependencies \n\nVsc-zk requires:\n \n * ZooKeeper 3.4 or above, since it is using transactions.\n * Kazoo 1.3.1 or above (`pip install kazoo`)\n * recent version of vsc-base (>= 1.6.7)\n\n## Build the initial zookeeper tree\n\nUse zkinitree to build an initial tree in zookeeper from a config file.\nIt will create paths with specified ACLs.\n\nAn example config file can be found in the examples folder (zkinitree.cfg) \n\n\n## Usage of zkrsync\n\nExample usage for N-parallelised rsync :\n\nPut reusable values in a config file, e.g. zkrs.conf:\n```\n[MAIN]\nservers=<servers> \nsession=<session>\ndomain=<domain>\ndaemon=true\nuser=<user>\npasswd=<password>\ndepth=<depth>\n#debug=true\n#dryrun=false\n#delete=false\n```\nStart N+1 sources (first source client will be the Master)\n```    \nzkrsync -S -r <sourcepath> --configfiles=zkrs.conf\n```\nStart N destinations:\n```    \nzkrsync -D -r <destpath> --configfiles=zkrs.conf\n```\nTesting pathbuilding: add the `pathsonly` option\n\nrun `zkrsync -H` to see all options\n\nDefault logging goes to `/tmp/zkrsync/<session>-<source|dest>-<pid>.log`. Use the `logfile` option to change the template. Use the `debug` option (or short `-d`) for more verbose logging.\n\nWhen running in daemon mode, a pidfile will be generated. Default location is `/tmp/zkrsync/<session>-<source|dest>-<pid>.pid` (pid is pid of process that starts the daemon). This can also be templated with the `pidfile` option.\n\nIf anything (zookeeper-related) goes wrong (no cleanup has been done)\n\n - kill all running source and destination clients (of that session)\n - wait about 20 seconds (ymmv), making sure all zookeeper connections have timed out\n - run exactly one source client, if the previous shutdown was not finished cleanly, \nzookeeper will recognise this and attempt a new cleanup before exiting.\n - Should the previous step fail to clean up, there might still be a running client. \nMake sure they are all killed \n - Start a new zkrsync process\n\nGlobal remarks:\n\n - Always define a session to make sure you do not mix up different sessions\n - Make sure parameters path, dryrun and delete are all the same: These parameters are not checked \n(at this moment) and each client will use its own parameters. \nThis can lead to inconsistencies (e.g., some paths runs with the dry-run option while others are effectively synced.)\n - Parameter depth is only used on pathbuilding, so the Source Master will always provide this.",
        "description_content_type": null,
        "docs_url": null,
        "download_url": "UNKNOWN",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://github.com/hpcugent/vsc-zk",
        "keywords": null,
        "license": "LGPLv2+",
        "maintainer": null,
        "maintainer_email": null,
        "name": "vsc-zk",
        "package_url": "https://pypi.org/project/vsc-zk/",
        "platform": "UNKNOWN",
        "project_url": "https://pypi.org/project/vsc-zk/",
        "project_urls": {
            "Download": "UNKNOWN",
            "Homepage": "https://github.com/hpcugent/vsc-zk"
        },
        "release_url": "https://pypi.org/project/vsc-zk/0.9.7/",
        "requires_dist": null,
        "requires_python": null,
        "summary": "vsc-zk contains scripts to build zookeeper trees and tools using zookeeper.",
        "version": "0.9.7"
    },
    "last_serial": 2315477,
    "releases": {
        "0.6.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "c53687ff0e3f367ad8036804f45289a6",
                    "sha256": "84fa1c9788e1423a9f2bd4023258c15901cc5072708e1b58fbffb2e7c35d8377"
                },
                "downloads": -1,
                "filename": "vsc-zk-0.6.1.tar.gz",
                "has_sig": false,
                "md5_digest": "c53687ff0e3f367ad8036804f45289a6",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 13072,
                "upload_time": "2014-04-14T09:53:03",
                "url": "https://files.pythonhosted.org/packages/e5/35/344ed52e9feb4ada3981986af7fb7eda185ce98140d51f69626bdbfa0bd1/vsc-zk-0.6.1.tar.gz"
            }
        ],
        "0.6.2": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "1380bc7afbe04f46b6bd69a1d40196bb",
                    "sha256": "e8bb98d5b3ab70e4395804f536fbf0d9f3f4780507d787b57519531747490671"
                },
                "downloads": -1,
                "filename": "vsc-zk-0.6.2.tar.gz",
                "has_sig": false,
                "md5_digest": "1380bc7afbe04f46b6bd69a1d40196bb",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 13302,
                "upload_time": "2014-04-14T13:19:00",
                "url": "https://files.pythonhosted.org/packages/c7/3d/fbdbf452e383728875e436603ff4b0b02a58ce7b9a2db74140711792d639/vsc-zk-0.6.2.tar.gz"
            }
        ],
        "0.7.2": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "46111d6bffe81123ae47c142af617e53",
                    "sha256": "f470209a0915219bd29873a6d1bd2899bcc894b76bf7150411dd3c50f0429d69"
                },
                "downloads": -1,
                "filename": "vsc-zk-0.7.2.tar.gz",
                "has_sig": false,
                "md5_digest": "46111d6bffe81123ae47c142af617e53",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 14667,
                "upload_time": "2015-07-06T09:16:28",
                "url": "https://files.pythonhosted.org/packages/dc/dd/9488290840facf558be8d78ba25a5cf3949b96785b9dc754dbaed22da715/vsc-zk-0.7.2.tar.gz"
            }
        ],
        "0.9.7": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "b5fda5bdfc5dd53bd25029919bce6256",
                    "sha256": "013390e945a6a22049e245a875310356abb26471e3580428e6ba4f079e12bbed"
                },
                "downloads": -1,
                "filename": "vsc-zk-0.9.7.tar.gz",
                "has_sig": false,
                "md5_digest": "b5fda5bdfc5dd53bd25029919bce6256",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 45119,
                "upload_time": "2016-08-31T11:46:45",
                "url": "https://files.pythonhosted.org/packages/ad/94/5ae5a7ae6031b296d9f475b95081910cba8a23d9169c838b002ec1a865f0/vsc-zk-0.9.7.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "b5fda5bdfc5dd53bd25029919bce6256",
                "sha256": "013390e945a6a22049e245a875310356abb26471e3580428e6ba4f079e12bbed"
            },
            "downloads": -1,
            "filename": "vsc-zk-0.9.7.tar.gz",
            "has_sig": false,
            "md5_digest": "b5fda5bdfc5dd53bd25029919bce6256",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 45119,
            "upload_time": "2016-08-31T11:46:45",
            "url": "https://files.pythonhosted.org/packages/ad/94/5ae5a7ae6031b296d9f475b95081910cba8a23d9169c838b002ec1a865f0/vsc-zk-0.9.7.tar.gz"
        }
    ]
}