{
    "info": {
        "author": "Konstantin Ryabitsev",
        "author_email": "konstantin@linuxfoundation.org",
        "bugtrack_url": null,
        "classifiers": [],
        "description": "GROKMIRROR\n==========\n--------------------------------------------\nFramework to smartly mirror git repositories\n--------------------------------------------\n\n:Author:    konstantin@linuxfoundation.org\n:Date:      2019-02-14\n:Copyright: The Linux Foundation and contributors\n:License:   GPLv3+\n:Version:   1.2.1\n\nDESCRIPTION\n-----------\nGrokmirror was written to make mirroring large git repository\ncollections more efficient. Grokmirror uses the manifest file published\nby the master mirror in order to figure out which repositories to\nclone, and to track which repositories require updating. The process is\nextremely lightweight and efficient both for the master and for the\nmirrors.\n\nCONCEPTS\n--------\nGrokmirror master publishes a json-formatted manifest file containing\ninformation about all git repositories that it carries. The format of\nthe manifest file is as follows::\n\n    {\n      \"/path/to/bare/repository.git\": {\n        \"description\": \"Repository description\",\n        \"reference\":   \"/path/to/reference/repository.git\",\n        \"modified\":    timestamp,\n        \"fingerprint\": sha1sum(git show-ref),\n        \"symlinks\": [\n            \"/location/to/symlink\",\n            ...\n        ],\n       }\n       ...\n    }\n\nThe manifest file is usually gzip-compressed to preserve bandwidth.\n\nEach time a commit is made to one of the git repositories, it\nautomatically updates the manifest file using an appropriate git hook,\nso the manifest.js file always contains the most up-to-date information\nabout the repositories provided by the git server and their\nlast-modified date.\n\nThe mirroring clients will constantly poll the manifest.js file and\ndownload the updated manifest if it is newer than the locally stored\ncopy (using ``Last-Modified`` and ``If-Modified-Since`` http headers).\nAfter downloading the updated manifest.js file, the mirrors will parse\nit to find out which repositories have been updated and which new\nrepositories have been added.\n\nFor all newly-added repositories, the clients will do::\n\n    git clone --mirror git://server/path/to/repository.git \\\n        /local/path/to/repository.git\n\nFor all updated repositories, the clients will do::\n\n    GIT_DIR=/local/path/to/repository.git git remote update\n\nWhen run with ``--purge``, the clients will also purge any repositories\nno longer present in the manifest file received from the server.\n\nShared repositories\n~~~~~~~~~~~~~~~~~~~\nGrokmirror will automatically recognize when repositories share objects\nvia alternates. E.g. if repositoryB is a shared clone of repositoryA\n(that is, it's been cloned using ``git clone -s repositoryA``), the\nmanifest will mention the referencing repository, so grokmirror will\nmirror repositoryA first, and then mirror repositoryB with a\n``--reference`` flag. This greatly reduces the bandwidth and disk use\nfor large repositories.\n\nSee man git-clone_ for more info.\n\n.. _git-clone: https://www.kernel.org/pub/software/scm/git/docs/git-clone.html\n\nSERVER SETUP\n------------\nInstall grokmirror on the server using your preferred way.\n\n**IMPORTANT: Currently, only bare git repositories are supported.**\n\nYou will need to add a hook to each one of your repositories that would\nupdate the manifest upon repository modification. This can either be a\npost-receive hook, or a post-update hook. The hook must call the\nfollowing command::\n\n    /usr/bin/grok-manifest -m /repos/manifest.js.gz -t /repos -n `pwd`\n\nThe **-m** flag is the path to the manifest.js file. The git process must be\nable to write to it and to the directory the file is in (it creates a\nmanifest.js.randomstring file first, and then moves it in place of the\nold one for atomicity).\n\nThe **-t** flag is to help grokmirror trim the irrelevant toplevel disk\npath. E.g. if your repository is in /var/lib/git/repository.git, but it\nis exported as git://server/repository.git, then you specify ``-t\n/var/lib/git``.\n\nThe **-n** flag tells grokmirror to use the current timestamp instead of the\nexact timestamp of the commit (much faster this way).\n\nBefore enabling the hook, you will need to generate the manifest.js of\nall your git repositories. In order to do that, run the same command,\nbut omit the -n and the \\`pwd\\` argument. E.g.::\n\n    /usr/bin/grok-manifest -m /repos/manifest.js.gz -t /repos\n\nThe last component you need to set up is to automatically purge deleted\nrepositories from the manifest. As this can't be added to a git hook,\nyou can either run the ``--purge`` command from cron::\n\n    /usr/bin/grok-manifest -m /repos/manifest.js.gz -t /repos -p\n\nOr add it to your gitolite's ``D`` command using the ``--remove`` flag::\n\n    /usr/bin/grok-manifest -m /repos/manifest.js.gz -t /repos -x $repo.git\n\nIf you would like grok-manifest to honor the ``git-daemon-export-ok``\nmagic file and only add to the manifest those repositories specifically\nmarked as exportable, pass the ``--check-export-ok`` flag. See\n``git-daemon(1)`` for more info on ``git-daemon-export-ok`` file.\n\nMIRROR SETUP\n------------\nInstall grokmirror on the mirror using your preferred way.\n\nLocate repos.conf and modify it to reflect your needs. The default\nconfiguration file is heavily commented.\n\nAdd a cronjob to run as frequently as you like. For example, add the\nfollowing to ``/etc/cron.d/grokmirror.cron``::\n\n    # Run grok-pull every minute as user \"mirror\"\n    * * * * * mirror /usr/bin/grok-pull -p -c /etc/grokmirror/repos.conf\n\nMake sure the user \"mirror\" (or whichever user you specified) is able to\nwrite to the toplevel and log locations specified in repos.conf.\n\nIf you already have a bunch of repositories in the hierarchy that\nmatches the upstream mirror and you'd like to reuse them instead of\nre-downloading everything from the master, you can pass the ``-r`` flag\nto tell grok-pull that it's okay to reuse existing repos. This will\ndelete any existing remotes defined in the repository and set the new\norigin to match what is configured in the repos.conf.\n\nGROK-FSCK\n---------\nGit repositories can get corrupted whether they are frequently updated\nor not, which is why it is useful to routinely check them using \"git\nfsck\". Grokmirror ships with a \"grok-fsck\" utility that will run \"git\nfsck\" on all mirrored git repositories. It is supposed to be run\nnightly from cron, and will do its best to randomly stagger the checks\nso only a subset of repositories is checked each night. Any errors will\nbe sent to the user set in MAILTO.\n\nTo enable grok-fsck, first locate the fsck.conf file and edit it to\nmatch your setup -- e.g., it must know where you keep your local\nmanifest. Then, add the following to ``/etc/cron.d/grok-fsck.cron``::\n\n    # Make sure MAILTO is set, for error reports\n    MAILTO=root\n    # Run nightly repacks to optimize the repos\n    0 2 1-6 * * mirror /usr/bin/grok-fsck -c /etc/grokmirror/fsck.conf --repack-only\n    # Run weekly fsck checks on Sunday\n    0 2 0 * * mirror /usr/bin/grok-fsck -c /etc/grokmirror/fsck.conf\n\nYou can force a full run using the ``-f`` flag, but unless you only have\na few smallish git repositories, it's not recommended, as it may take\nseveral hours to complete. See the man page for other flags grok-fsck\nsupports.\n\nBefore it runs, grok-fsck will put an advisory lock for the git-directory\nbeing checked (.repository.git.lock). Grok-pull will recognize the lock\nand will postpone any incoming updates to that repository until the lock\nis freed.\n\nFAQ\n---\nWhy is it called \"grok mirror\"?\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\nBecause it's developed at kernel.org and \"grok\" is a mirror of \"korg\".\nAlso, because it groks git mirroring.\n\nWhy not just use rsync?\n~~~~~~~~~~~~~~~~~~~~~~~\nRsync is extremely inefficient for the purpose of mirroring git trees\nthat mostly consist of a lot of small files that very rarely change.\nSince rsync must calculate checksums on each file during each run, it\nmostly results in a lot of disk thrashing.\n\nAdditionally, if several repositories share objects between each-other,\nunless the disk paths are exactly the same on both the remote and local\nmirror, this will result in broken git repositories.\n\nIt is also a bit silly, considering git provides its own extremely\nefficient mechanism for specifying what changed between revision X and\nrevision Y.\n\nWhy not just run \"git pull\" from cron every minute?\n~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\nThis is not a complete mirroring strategy, as this won't notify you when\nthe remote mirror adds new repositories. It is also not very nice to the\nremote server, especially the one that carries hundreds of repositories.\n\nAdditionally, this will not automatically take care of shared\nrepositories for you. See \"Shared repositories\" under \"CONCEPTS\".\n\n\n",
        "description_content_type": "text/x-rst",
        "docs_url": null,
        "download_url": "https://www.kernel.org/pub/software/network/grokmirror/grokmirror-1.2.1.tar.xz",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://git.kernel.org/pub/scm/utils/grokmirror/grokmirror.git",
        "keywords": "git,mirroring,repositories",
        "license": "GPLv3+",
        "maintainer": "",
        "maintainer_email": "",
        "name": "grokmirror",
        "package_url": "https://pypi.org/project/grokmirror/",
        "platform": "",
        "project_url": "https://pypi.org/project/grokmirror/",
        "project_urls": {
            "Download": "https://www.kernel.org/pub/software/network/grokmirror/grokmirror-1.2.1.tar.xz",
            "Homepage": "https://git.kernel.org/pub/scm/utils/grokmirror/grokmirror.git",
            "Source": "https://git.kernel.org/pub/scm/utils/grokmirror/grokmirror.git",
            "Tracker": "https://github.com/mricon/grokmirror/issues"
        },
        "release_url": "https://pypi.org/project/grokmirror/1.2.1/",
        "requires_dist": [
            "anyjson",
            "GitPython (>=2.1.8)",
            "enlighten"
        ],
        "requires_python": "",
        "summary": "Smartly mirror git repositories that use grokmirror",
        "version": "1.2.1"
    },
    "last_serial": 4926638,
    "releases": {
        "1.1.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "20aa9a607c4a0a835fba280f9b670adf",
                    "sha256": "e687f5242e15402352d425b350f69508f577aa4466c0c236b0fa1c8bda8c27fe"
                },
                "downloads": -1,
                "filename": "grokmirror-1.1.0.tar.gz",
                "has_sig": false,
                "md5_digest": "20aa9a607c4a0a835fba280f9b670adf",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 45140,
                "upload_time": "2018-04-24T19:16:04",
                "url": "https://files.pythonhosted.org/packages/7b/b2/558621b8e2e150bb1e0a102341aa5469fc23e9f01c8feb0ec0a7116cee81/grokmirror-1.1.0.tar.gz"
            }
        ],
        "1.1.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "0c34f15d1e5911cab65247e067408ee5",
                    "sha256": "c06124c6221d45e2615fa637c0e6f27d1582ad55883c77e921af39c120b2ab83"
                },
                "downloads": -1,
                "filename": "grokmirror-1.1.1.tar.gz",
                "has_sig": false,
                "md5_digest": "0c34f15d1e5911cab65247e067408ee5",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 45233,
                "upload_time": "2018-08-09T17:41:06",
                "url": "https://files.pythonhosted.org/packages/2e/ec/6c9f2e709b9089354f8e13c73e688d2d0b27e9f0e2a417474eec5e95056c/grokmirror-1.1.1.tar.gz"
            }
        ],
        "1.2.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "6b5056911b23fb4768e88132b386e050",
                    "sha256": "46c4e4daf04e0f4d0f2225b210f1cd68687c3619b6b42704b696b24ea0a74b54"
                },
                "downloads": -1,
                "filename": "grokmirror-1.2.0-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "6b5056911b23fb4768e88132b386e050",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 33874,
                "upload_time": "2019-02-14T17:13:38",
                "url": "https://files.pythonhosted.org/packages/82/37/765c638341b851263f512ef8470ac6d39d0b3bfd1b22e8103fe2516456d0/grokmirror-1.2.0-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "c639dc3fda16fa61513421f78ed4e6ca",
                    "sha256": "8278cefc03c515fa6f61831067644e4cc7b62b77fe761d79ad3f4a53f06f35c0"
                },
                "downloads": -1,
                "filename": "grokmirror-1.2.0.tar.gz",
                "has_sig": false,
                "md5_digest": "c639dc3fda16fa61513421f78ed4e6ca",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 48005,
                "upload_time": "2019-02-14T17:13:40",
                "url": "https://files.pythonhosted.org/packages/41/f4/0a52147abf521b6568f09ce778c11975aca6b13ddfadac1b4018c9823c28/grokmirror-1.2.0.tar.gz"
            }
        ],
        "1.2.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "5a0f5985a541af0f2fa255364d4a1e80",
                    "sha256": "757c9ee6b57aa276a4ec517b03240894602969ade1dcf22d0c6a635a59b91f56"
                },
                "downloads": -1,
                "filename": "grokmirror-1.2.1-py3-none-any.whl",
                "has_sig": false,
                "md5_digest": "5a0f5985a541af0f2fa255364d4a1e80",
                "packagetype": "bdist_wheel",
                "python_version": "py3",
                "requires_python": null,
                "size": 33994,
                "upload_time": "2019-03-11T19:28:24",
                "url": "https://files.pythonhosted.org/packages/17/2b/cee644b16768c8c7e85024ae30d1eaba2ef755750a230b35c3142dd08bdb/grokmirror-1.2.1-py3-none-any.whl"
            },
            {
                "comment_text": "",
                "digests": {
                    "md5": "b522a4ec1c317dcae14c9e27c7783a8b",
                    "sha256": "8a1391c298aa4c3ae6f9bd1248be587739cc5e55cac39925e1e7ac61241cba3f"
                },
                "downloads": -1,
                "filename": "grokmirror-1.2.1.tar.gz",
                "has_sig": false,
                "md5_digest": "b522a4ec1c317dcae14c9e27c7783a8b",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 48543,
                "upload_time": "2019-03-11T19:28:26",
                "url": "https://files.pythonhosted.org/packages/4c/74/01fc52d7cf5fbc18554629702af93069123bd5d789bc54801923af94f6df/grokmirror-1.2.1.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "5a0f5985a541af0f2fa255364d4a1e80",
                "sha256": "757c9ee6b57aa276a4ec517b03240894602969ade1dcf22d0c6a635a59b91f56"
            },
            "downloads": -1,
            "filename": "grokmirror-1.2.1-py3-none-any.whl",
            "has_sig": false,
            "md5_digest": "5a0f5985a541af0f2fa255364d4a1e80",
            "packagetype": "bdist_wheel",
            "python_version": "py3",
            "requires_python": null,
            "size": 33994,
            "upload_time": "2019-03-11T19:28:24",
            "url": "https://files.pythonhosted.org/packages/17/2b/cee644b16768c8c7e85024ae30d1eaba2ef755750a230b35c3142dd08bdb/grokmirror-1.2.1-py3-none-any.whl"
        },
        {
            "comment_text": "",
            "digests": {
                "md5": "b522a4ec1c317dcae14c9e27c7783a8b",
                "sha256": "8a1391c298aa4c3ae6f9bd1248be587739cc5e55cac39925e1e7ac61241cba3f"
            },
            "downloads": -1,
            "filename": "grokmirror-1.2.1.tar.gz",
            "has_sig": false,
            "md5_digest": "b522a4ec1c317dcae14c9e27c7783a8b",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 48543,
            "upload_time": "2019-03-11T19:28:26",
            "url": "https://files.pythonhosted.org/packages/4c/74/01fc52d7cf5fbc18554629702af93069123bd5d789bc54801923af94f6df/grokmirror-1.2.1.tar.gz"
        }
    ]
}