{ "info": { "author": "Miguel Garcia (aka Zeycus)", "author_email": "zeycus@gmail.com", "bugtrack_url": null, "classifiers": [ "Development Status :: 4 - Beta", "License :: OSI Approved :: MIT License", "Programming Language :: Python :: 3.5", "Topic :: System :: Archiving :: Backup" ], "description": "********\nOverview\n********\n\nA command-line script in Python is provided, to manage backups for large filesystems in multiple external disks.\n\nIt is intended as a minimalist system, to get the job done but with no GUI or other niceties. At least not yet!\nI just wanted to sleep well at night.\n\nSupport for Linux and Windows.\n\n\n\nMotivation\n==========\n\n\nThe Problem\n-----------\nFor more than a decade I had being gathering content and\nstoring it in external drives.\nFor backup purposes I used to buy them in pairs, so that one would work as the other's mirror.\nOf course the solution was far from ideal, there were TV-shows, movies, and documentaries in most disks,\nsparsed pretty much randomly, and when the number of disks reached 15 (plus backups) even finding content was a pain.\nI had simple text files with the ``ls`` contents of each disk, which needed to be updated, etc.\n\n\nAn Improvement\n------------------\nA friend talked to me about a NAS he had recently acquired. After little consideration I realized I had been needing\none myself for a long time, just did not know such a thing existed. Taking into account the size of the files I already had,\nplus reasonable mid-term foreseable needs, I bought a 6-slots NAS and put 8GB disks in it (5 of them currently).\n\nThanks to it, soon the content was neatly organised, easy to find and maintain.\nI was using RAID5, which is nice, but in several forums I found the clear warning\nthat `RAID does not work as backup `_\\ , so I started worring.\nI had the need of a real backup, and a bunch of external drives which content was already in the NAS.\nObviously they might be used to backup content, but I could not bring myself to even try to micro-manage it.\nIt would be particularly hard because some folders are way bigger that the external drives, so they would need to be split manually.\n\nI needed a proper tool to automatically manage a multi-volume backup.\n\n\n\nBackup System Overview\n======================\nThe idea behind the implementation of **fsbackup** is pretty simple, and everything gets done by a single command: ``fsbck``.\nGiven a list of one or more paths that we want backed-up, the backup system works in three stages.\n\n\nStage 1\n-------\n\nA command (intended to be scheduled nightly) keeps a collection in a `mongoDB `_ database updated with\nthe absolute path, size, last modification timestamp and a hash function (currently SHA-256) of each file in that list of paths.\nThey are interpreted as file-trees, so all the content buried in those paths is included.\nIt can be done with something like::\n\n fsbck.py refreshHashes -db=conn_multimedia.json\n\nOnly new files, or files with a more recent modification timestamp than the one in the database have their hash function recalculated,\nsince it is indeed time-consuming. As you might have guessed, the ``db`` argument refers\nto a `json `_ file with information regarding the location\nof the filesystem, as well as mongoDB collections where the information is stored.\n\n\nStage 2\n--------\nExternal hard disks work as backup volumes, containing files renamed with their hash function. The folder structure in the original filesystem\nis not replicated, all files are at root level. Except that, using git-style, they are divided in folders according to the first\nletters in the hash, to avoid having thousands of files in the same directory.\n\nIn order to update the backup, we can mount a disk that works as backup volume. For instance, if we are using Windows and the drive was assigned to G:, we run::\n\n fsbck.py processDrive -db=conn_multimedia.json --drive=G\n\nThis action:\n\n* Removes from the volume files that are not necessary anymore.\n* Copies new files that were not backed-up yet.\n* Provides a backup status report, with:\n\n * the number of files/size pending backup (if there was not enough room in that volume).\n * a summary of the number of files/size in each volume.\n * a file per volume is created with the detailed absolute paths of each file backed-up in it.\n\nFor this to work properly, another collection in the database stores the hashes backed in each volume.\n\nIn Linux it is slightly different, argument ``drive`` should be replaced by ``mountpoint``. If the volume had been\nmounted in ``/mnt/zeycus/FA03-E14F`` we should run::\n\n fsbck.py processDrive -db=conn_multimedia.json --mountpoint=/mnt/zeycus/FA03-E14F\n\nMost of the examples in the documentation are written for the Windows version, but ``mountpoint`` can always be used\ninstead of ``drive``.\n\n\n\nStage 3\n--------\nIf/when the time comes information needs to be retrieved from the volumes, the script handles that as well. For instance, the command::\n\n fsbck.py checkout -db=conn_multimedia.json --drive=G --sourcepath=Multimedia/video/movies --destpath=F:\\chekouts\\movies\n\nrecovers the relevant information in the actual (G:) volume for a particular folder. In a worst-case scenario, to recover all the files\nyou'd have to do this for every volume.\n\n\nSo, how do I start?\n===================\nIn a nutshell:\n\n1. Get a mongoDB server connection and create a database there. It could be local, mongoDB hosting (like `mlab `_ , just to name one), etc.\n\n2. Build a `JSON `_ config_file for the filesystem you want backed-up. For instance:\n\n.. code-block:: json\n\n {\n \"connstr\": \"mongodb://myuser:mypwd@ds21135.mlab.com:34562/fsbackup_tvs761_main\",\n \"mountpoint\": \"\\\\\\\\ZEYCUS-TVS671\",\n \"paths\": [\n \"Multimedia\",\n \"Resources\"\n ],\n \"reportpref\": \"F:\\\\Dropbox\\\\fsbackup\\\\reports\\\\main_\"\n }\n\nwhere ``connstr`` is the conection string to your mongoDB database (in this case, ``fsbackup_tvs761_main``).\nWith this, you'll build backup for all the content in\n``\\\\ZEYCUS-TVS671\\Multimedia`` and ``\\\\ZEYCUS-TVS671\\Resources``, more details in the documentation.\nMake sure the path in ``reportpref`` actually exists, reporting files are created there. In this case,\n``F:\\Dropbox\\fsbackup\\reports``.\n\n\n3. Create the actual collections in the database with::\n\n fsbck.py createDatabase -db= --force\n \n\n4. Gather the current filesystem information with::\n\n fsbck.py refreshHashes -db=\n\t\nThe first time hashes are calculated for all files, so this may take **long**.\n\n5. Connect a formated external drive. Assuming it gets mounted in ``driveLetter``, execute::\n\n fsbck.py processDrive -db= --drive=\n\nThis fills the volume with backup data. When finished, a message will clarify whether more volumes are needed to go on\nwith the backup.\n \n\n\n\nCollaboration\n=============\n\nYou may wish to improve or add features, in that case you are more than welcome, feel free to contact me at zeycus@gmail.com.\n\n\nComplete Documentation\n======================\nFind the complete documentation in `readthedocs `_.\n", "description_content_type": null, "docs_url": null, "download_url": "", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "https://github.com/zeycus/fsbackup/", "keywords": "backup volume", "license": "MIT", "maintainer": "", "maintainer_email": "", "name": "fsbackup", "package_url": "https://pypi.org/project/fsbackup/", "platform": "", "project_url": "https://pypi.org/project/fsbackup/", "project_urls": { "Homepage": "https://github.com/zeycus/fsbackup/" }, "release_url": "https://pypi.org/project/fsbackup/0.2.1/", "requires_dist": null, "requires_python": "", "summary": "Multi-Volume Backup for Large Filesystems", "version": "0.2.1" }, "last_serial": 3388113, "releases": { "0.1": [ { "comment_text": "", "digests": { "md5": "5d92e70397ee997e5b80cd56d25def2b", "sha256": "eeb482e5825dfd0bae4418a5a47aef2e65a833f22db1b5474317935ac310e4bb" }, "downloads": -1, "filename": "fsbackup-0.1.tar.gz", "has_sig": false, "md5_digest": "5d92e70397ee997e5b80cd56d25def2b", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 14845, "upload_time": "2017-11-05T10:36:11", "url": "https://files.pythonhosted.org/packages/06/07/314c53b3f6691baf88ac9199f9b45db0afb2b47d918e338dee26433fbc9e/fsbackup-0.1.tar.gz" } ], "0.1.1": [ { "comment_text": "", "digests": { "md5": "0f8de639f7724e887b11f02e1b608114", "sha256": "e50089229d8b8fcbeb32d9b1453ad09de8c90dc146669958817094b0c7ea6ff8" }, "downloads": -1, "filename": "fsbackup-0.1.1.tar.gz", "has_sig": false, "md5_digest": "0f8de639f7724e887b11f02e1b608114", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 749766, "upload_time": "2017-11-05T13:45:17", "url": "https://files.pythonhosted.org/packages/0a/c7/881b31031061f0556d86ca3034605842419c6eb82bcce2f255037b4d512c/fsbackup-0.1.1.tar.gz" } ], "0.1.2": [ { "comment_text": "", "digests": { "md5": "0d77a0640feb3524e2f7b27f4497d9f8", "sha256": "d8c98cffa1c9d9f9e203159604d06010f4837280fa32cdf17db4ce4541452725" }, "downloads": -1, "filename": "fsbackup-0.1.2.tar.gz", "has_sig": false, "md5_digest": "0d77a0640feb3524e2f7b27f4497d9f8", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 756848, "upload_time": "2017-11-09T08:07:17", "url": "https://files.pythonhosted.org/packages/1f/a0/8d2701d757b8b90d0ba69ec09310857fce997d0a4d76b7185c9922a3f4b7/fsbackup-0.1.2.tar.gz" } ], "0.1.3": [ { "comment_text": "", "digests": { "md5": "360f1fc7641a2d72b14a419c06cc23ed", "sha256": "dd78d4231068244b3009f503548cb4ce2bf307a033a36238a79ba669843b35ba" }, "downloads": -1, "filename": "fsbackup-0.1.3.tar.gz", "has_sig": false, "md5_digest": "360f1fc7641a2d72b14a419c06cc23ed", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 756845, "upload_time": "2017-11-12T20:55:56", "url": "https://files.pythonhosted.org/packages/d3/4c/77a9e5a168356759c866fca4a94c93b60486c04a86523306d4cbc59af648/fsbackup-0.1.3.tar.gz" } ], "0.2.1": [ { "comment_text": "", "digests": { "md5": "5fc37a7966601dae249b7276e9704715", "sha256": "c637508ed0a1e44e5f5ceb763613b4532b14f6967bc1e9cb692e0bb706ca9c42" }, "downloads": -1, "filename": "fsbackup-0.2.1.tar.gz", "has_sig": false, "md5_digest": "5fc37a7966601dae249b7276e9704715", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 413742, "upload_time": "2017-12-04T18:44:01", "url": "https://files.pythonhosted.org/packages/67/9e/925bbdfeda21225d9e5d5b580ac591dd77ccb5aead68958a9825b21c5559/fsbackup-0.2.1.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "5fc37a7966601dae249b7276e9704715", "sha256": "c637508ed0a1e44e5f5ceb763613b4532b14f6967bc1e9cb692e0bb706ca9c42" }, "downloads": -1, "filename": "fsbackup-0.2.1.tar.gz", "has_sig": false, "md5_digest": "5fc37a7966601dae249b7276e9704715", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 413742, "upload_time": "2017-12-04T18:44:01", "url": "https://files.pythonhosted.org/packages/67/9e/925bbdfeda21225d9e5d5b580ac591dd77ccb5aead68958a9825b21c5559/fsbackup-0.2.1.tar.gz" } ] }