{ "info": { "author": "Miguel Simoes", "author_email": "miguelrsimoes@yahoo.com", "bugtrack_url": null, "classifiers": [ "Development Status :: 3 - Alpha", "Environment :: Console", "License :: OSI Approved :: GNU General Public License v3 (GPLv3)", "Operating System :: POSIX :: Linux", "Programming Language :: Python :: 2.7", "Programming Language :: Python :: 3.6", "Topic :: System :: Archiving :: Backup", "Topic :: Utilities" ], "description": "# lnsync\n\n## Overview\n\n_lnsync_ provides unidirectional sync-by-rename of locally mounted directories, with support for hard links, without copying or deleting file data. To compare files, files hashes are computed and stored in a local database. Using these file hashes, _lnsync_ also provides other functions, such as finding duplicate files.\n\nThis tool allows arbitrary file renaming and moving in the source file tree to be quickly replicated in the target without copying any file data. It may used as a preprocessing step for other sync tools, such as _rsync_.\n\n### Hard Link Support\n\nOn many file systems (e.g. ext3/4, NTFS, btrfs), a file may be reached via multiple file paths, which function as aliases. A new path created for an existing file is a _hard link_, but all such aliases are on an equal footing, so each may be called a hard link. Removing the last hardlink to a file means deleting that file.\n\nIf both hard links are created or removed in the source (without deleting files), and the target supports hard linking, then _lnsync_ can create and removing hardlinks as required on the target when syncing. As before, this is done without copying or deleting file data.\n\n## Functionality\n\n### Syncing\n\nWhen you rename/move in the source file tree, _lnsync_ makes a best-effort to sync the target by only renaming files and generally creating/removing hard links on the target, without deleting or copying any file data. Empty directories on the target which do not exist on the source are also removed.\n\nFile content is compared using xxHash (a fast, non-cryptographic hash function). Hash values are stored in a single-file database at the top-level directory of each file tree. The hash value database is a single file at the top directory of processed trees, with basename matching `lnsync-[0-9]+.db`. (Only one such file should exist there.) These files are ignored by all _lnsync_ operations, and care should be taken not to sync them with other tools.\n\nFile modification times are used to detect stale hash values and are not synced. File ownership and permissions are ignored: files which cannot be read are skipped.\n\n### File Searching\n\nUsing file content hashes, _lnsync_ can also find duplicate files (like _fdupes_), and more generally find files according to the which file trees they appear on, compare file trees, and check for changes/bitrot.\n\n### Offline File Trees\n\n_lnsync_ can save the structure of a file tree under a local directory and save it in a single-file database. These offline file trees can be used in most _lnsync_ commands in place of an local directory, even as the source in a sync command.\n\n## Alternative Sync Solutions for Linux\n\nSome of the many tools for syncing with rename detection:\n\n- There are patches for _rsync_ (see --detect-renamed) that provide renaming on the target, relying on file size and modification time, for matching files in nearby directories. _rsync_ can preserve hard links and sync with remote rsync instances.\n\n- [rclone](https://rclone.org/) provides sync-by-rename for local as well as an amazing array of remote clients. It allows caching file hashes, after some configuration. However, it does not preserve hard links.\n\n- [unison](https://www.cis.upenn.edu/~bcpierce/unison/) and [bsync](https://github.com/dooblem/bsync) provide network syncing with rename detection, but do not preserve hard links.\n\n- _git_ itself identifies and stores files by content, and has been adapted for syncing.\n\n- Support in modern file systems (e.g. btrfs) for snapshots may be adapted for syncing.\n\nIn addition to syncing, _lnsync_ allows using the file hash database to search for files according to a variety of criteria.\n\n## Installation and Quickstart\n\n### Installing\n\nInstall the latest version from the PyPI repository with `pip install -U lnsync`.\n\nOr clone the repo with `git clone https://github.com/mrsimoes/lnsync.git` and run `python setup.py install`.\n\n### Example Usage\n\nIf you have your photo archive at `/home/you/Photos` and your backup is at `/mnt/disk/Photos`, run `lnsync sync -n /home/you/Photos /mnt/disk/Photos` for a dry-run, to see which sync operations would be performed. To sync, ommit the `-n` switch.\n\nYou will notice that two database files are created, one at the source and another at the backup directory. File hashes are cmoputed as needed and then stored in these files. The database filenames includes a random suffix to avoid accidental overwriting when syncing with a tool other than _lsync_.\n\nTo quickly obtain an _rsync_ command that will complete syncing, skipping `lnsync` database files, run `lnsync rsync /home/you/Photos /mnt/disk/Photos`. To also run this command, use the `-x` switch. Make sure the `rsync` options provided by this command are suitable for you.\n\nFinally, to check the target is in-sync by recursively comparing it to source, run `lnsync cmp /home/you/Photos /mnt/disk/Photos`.\n\nTo find duplicate files on the Photos directory, run `lnsync fdupes /home/you/Photos`. Use `-z` to compare by size only. Use `-H` to count different hardlinks to the same file as distinct files. If this option is not given, for each multiple-linked with other duplicates, a path is arbitrarily picked and printed.\n\nTo find all files in Photos which are not in the backup (under any name): `lnsync onfirstonly /home/you/Photos /mnt/disk/Photos`. \n\nTo sync the subdir `/home/you/Photos/Best` to your digital picture frame, using the database hashes at `/home/you/Photos`: `lnsync sync /home/you/Photos/Best --root=/home/you/Photos /mnt/eframe/`.\n\n## Command Reference\nAll _lnsync_ commands are `lnsync [] [] []`.\n\n### Syncing\n- `lnsync sync [options] ` syncs a target dir from a source dir (or offline tree).\n\n - First, target files are matched to source files. Each matched target file is associated to a single source file. If either file system supports hardlinks, a file may have multiple pathnames. _lnsync_ will not complain if the match is not unique or some files are not matched on either source and/or target.\n\n - For each matched target file, its pathnames are made to match those of the corresponding source file, by renaming, deleting, or creating hardlinks. New intermediate subdirectories are created as needed on the target and directories which become empty on the target are removed.\n\n - `-z` Match files by size only. In this case, hash databases are not created or updated.\n\n - `-M=` Excludes all files larger than , which may be given in human form, e.g. `10k`, `2.1M`, `3G`.\n\n - `-n` Dry-run, just show which operations would be performed.\n\n - `--exclude=` Exclude source files and directories by glob pattern. Patterns are interpreted similarly to `rsync --exclude= source/ target`, but with `*` matching across slashes and no `**` pattern. This option may be repeated, with each `--exclude` option affecting all file trees. An initial slash anchors the pattern to the file tree root. A trailing slash means the pattern applies only to directories. There is a corresponding `--include`. Some commands accept `--exclude-once=` and `--include-once=`, which apply only to the next file tree following the switch and gain precedence over global patterns.\n\n To sum up, a file or directory is excluded if it matches an `exclude` pattern before matching any `include` pattern.\n\n- `lnsync rsync [options] [rsync-options]` Prints an _rsync_ command that would sync target dir from source, skipping _lnsync_ database files. Source may be a dir or an offline tree. Check the default _rsync_ options provided are what you want. To also run the _rsync_ command, use the `-x` switch.\n\n### Creating, Updating, and Accessing the Hash Database\n- `lnsync update ` Update the hash database, creating a new database if none exists, and rehashing all new files and those with a changed modification time (mtime). Accepts `--exclude=` options.\n- `lnsync update ` Update the hash database, creating a new database if none exists, and rehashing all new files and those with a changed modification time (mtime). Accepts `--exclude=` options.\n- `lnsync rehash []+` Force rehashing files specified by paths relative to the root `dir`.\n- `lnsync subdir ` Update the database at `relsubdir` using any hash value already present in the hash database for `dir`.\n- `lnsync mkoffline ` Update database at `dir` and create corresponding offline database at `outputfile`.\n- `lnsync cleandb ` Remove outdated entries and re-compact the database.\n- `lnsync lookup [+]` Returns (either from db or by recomputing) the hash value for the files, where `tree` may be a a directory or an offline tree.\n\n### Finding Files\n\n- `lnsync cmp ` Recursively compares two file trees. Compares files at each path, does not compare the hard link structure. Accepts `--exclude=` options.\n- `lnsync fdupes [-h] []+` Find files duplicated anywhere on the given trees.\n- `lnsync onall []+`, `lnsync onfirstonly []+`, `lnsync onlastonly []+` Find files as advertised. Some options: `-M` prunes by maximum size; `-0` prunes empty files; `-1` prints each group of files in a single line, separated by spaces and with escaped backslashes and spaces, like `fdupes`; `-s` sorts output by size. Use `-H` to consider multiple links to the same file as distinct files; if this option is not used, print a single, arbitrarily picked path for each multiple-linked file found to satisfy the condition of the command.\n\n- `lnsync check [] []*` Recompute hashes for given files and compare to the hash stored in the database, to check for changes/bitrot.\n\n## More Information\n\nThis package started as a Python learning project. I've found it useful enough to polish for publication, but as with any work in progress, it should be used with adequate caution.\n\nFeedback, suggestions, comments, and corrections are very welcome.\n\nYou can support this project in bitcoin at [17HS828pkQMiXZGy7UNbA49TYCz7LAQ2ze](bitcoin:17HS828pkQMiXZGy7UNbA49TYCz7LAQ2ze?amount=.001).\n\nThis program comes with ABSOLUTELY NO WARRANTY. This is free software, and you are welcome to redistribute it under certain conditions. See the GNU General Public Licence v3 for details.\n\n## Limitations and Future Developments\n\n### Caveats and Limitations\n- Works only on locally mounted directories, no support whatsoever for remote servers.\n- Depends on mtime to detect file content changes.\n- If source files A, B, C (with pairwise distinct contents) are renamed on target in a cycle to C, A, B, sync is currently not supported.\n- Only readable files and readable+accessible directories are read. Other files and dirs, as well as symlinks, pipes, special devices are ignored.\n- Minimal support for case-insensitive but case-preserving file systems like vfat: if a target file name differs from source match in case only, target is not updated.\n- Supports Linux only.\n\n### Possible Improvements\n- Make `--include` and `--exclude` patterns more compatible with `rsync`.\n- Extend `cmp` to take hard links into account.\n- Filenames are NOT converted to Unicode. To allow using offline database across systems, conversion is required.\n- Detect renamed directories to obtain a more compact sync schedule.\n- Use multiple CPUs to hash files in parallel.\n- Support partial hashes for quicker comparison of same-size files.\n- Further optimize the sync algorithm, though it has been working well in practice.\n- Support for checking for duplicates by actual content, not just hash.\n- Update target mtimes from source.\n- Allow more output sorting options, e.g. by name or mtime.\n- Allow config files and maybe store database along with config files in some .lnsync-DDDD directory at the root.\n\n## Release Notes\n\n- Version 0.3.8\n-- Less hashing on `onfirstonly`.\n-- Sort file search output by size.\n-- Adjusted user output levels.\n- Version 0.3.7\n-- Bug fix on reading offline trees.\n-- Change output levels and some messages.\n- Version 0.3.6\n-- New: --include and --include-once options.\n - Bug fix: wrong exit code.\n- Version 0.3.5\n - Bug fix: not excluding dirs in offline mode.\n - Version 0.3.3\n - Python 3 support.\n- Version 0.3.2\n - New --root option to allow reading and updating a root tree database when querying subtrees.\n- Version 0.3.0\n - Exclude files by glob pattern in sync and other commands.\n - Better terminal output.\n - Major code overhaul.\n- Version 0.1.9\n - Improved sync algorithm.\n - Remove directories left empty after sync.\n- Version 0.1\n - Initial version.", "description_content_type": "text/markdown", "docs_url": null, "download_url": "https://github.com/mrsimoes/lnsync/archive/v0.3.8.tar.gz", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "https://github.com/mrsimoes/lnsync", "keywords": "sync rename detection backup file content hard link hardlink rsync", "license": "GNU General Public License v3", "maintainer": "", "maintainer_email": "", "name": "lnsync", "package_url": "https://pypi.org/project/lnsync/", "platform": "", "project_url": "https://pypi.org/project/lnsync/", "project_urls": { "Download": "https://github.com/mrsimoes/lnsync/archive/v0.3.8.tar.gz", "Homepage": "https://github.com/mrsimoes/lnsync" }, "release_url": "https://pypi.org/project/lnsync/0.3.8/", "requires_dist": null, "requires_python": "", "summary": "Sync-by-rename local directories with hard link support, and more.", "version": "0.3.8" }, "last_serial": 5819571, "releases": { "0.1.9": [ { "comment_text": "", "digests": { "md5": "9bbc6510837ffddede008fe2d98a760f", "sha256": "b23586739c9ed60b743cac8faa5b00478d67abd42fa2a390578e31e52c3ae1c8" }, "downloads": -1, "filename": "lnsync-0.1.9.tar.gz", "has_sig": false, "md5_digest": "9bbc6510837ffddede008fe2d98a760f", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 39628, "upload_time": "2019-01-05T10:10:59", "url": "https://files.pythonhosted.org/packages/be/fa/3fca2907c55109fe99e5bf0a36dfb9977515b2f46746aaeeb3879f93dd3a/lnsync-0.1.9.tar.gz" } ], "0.3.1": [ { "comment_text": "", "digests": { "md5": "e2f4546abb25ea5fd904ca97ab0fefc5", "sha256": "7777bc26dd0312276ff25baf14d8f42fd379ab23f2548f88aeeaa757b9c0e768" }, "downloads": -1, "filename": "lnsync-0.3.1-py2-none-any.whl", "has_sig": false, "md5_digest": "e2f4546abb25ea5fd904ca97ab0fefc5", "packagetype": "bdist_wheel", "python_version": "py2", "requires_python": null, "size": 71267, "upload_time": "2019-06-14T08:17:57", "url": "https://files.pythonhosted.org/packages/86/51/04ea0c2ece83bee0b39fa54827ddcdb8eece1b3c9252cc31a055a006c784/lnsync-0.3.1-py2-none-any.whl" }, { "comment_text": "", "digests": { "md5": "6d03b33558a4b8b5c05e29d8599912a5", "sha256": "65f1fc76d5f504f8704b3ef3a65d2d503dcc5dcf6b364f306b4296aa009d8260" }, "downloads": -1, "filename": "lnsync-0.3.1.tar.gz", "has_sig": false, "md5_digest": "6d03b33558a4b8b5c05e29d8599912a5", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 49003, "upload_time": "2019-06-14T08:18:00", "url": "https://files.pythonhosted.org/packages/74/fe/f7709b6ff31d432c4018a882a4cc3759f198f6e53b1fd385d0041759a4ac/lnsync-0.3.1.tar.gz" } ], "0.3.1.post3": [ { "comment_text": "", "digests": { "md5": "f40312823fe80ecb32ede130fda8ce3b", "sha256": "994eff4797718bac0c6fb7ca6c8c4f8e5f242895c4b476d92d93b4eeb639b457" }, "downloads": -1, "filename": "lnsync-0.3.1.post3.tar.gz", "has_sig": false, "md5_digest": "f40312823fe80ecb32ede130fda8ce3b", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 49329, "upload_time": "2019-06-17T19:32:30", "url": "https://files.pythonhosted.org/packages/a6/65/acdbc2ef3159032ad4feabe4957c75f2e4aed0c1f4d9e3b3d9433b202d19/lnsync-0.3.1.post3.tar.gz" } ], "0.3.3": [ { "comment_text": "", "digests": { "md5": "ddbfa2be72107f308b9b7ca4fad28b08", "sha256": "8b0c280808d27da987afe8bfa5cf3f849a3f5f7a668db7575b6481a739e110a6" }, "downloads": -1, "filename": "lnsync-0.3.3.tar.gz", "has_sig": false, "md5_digest": "ddbfa2be72107f308b9b7ca4fad28b08", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 57092, "upload_time": "2019-07-08T17:20:06", "url": "https://files.pythonhosted.org/packages/62/b8/11c2fbdd4fd7efaf8c5a07dda89a7e95356d95496270e3cd99cabe96ac8b/lnsync-0.3.3.tar.gz" } ], "0.3.4": [ { "comment_text": "", "digests": { "md5": "989b83fe2a2efc55e743c0b9472cff02", "sha256": "b0588077d6edb826b094febf22eb7c319af7c7d810a58c2c3c1a30f5490842f7" }, "downloads": -1, "filename": "lnsync-0.3.4.tar.gz", "has_sig": false, "md5_digest": "989b83fe2a2efc55e743c0b9472cff02", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 57564, "upload_time": "2019-07-21T13:39:20", "url": "https://files.pythonhosted.org/packages/61/e1/bb0c74dcc5354bcf36510dad24183170e0870adc333f1bb26e2632bcefbf/lnsync-0.3.4.tar.gz" } ], "0.3.5": [ { "comment_text": "", "digests": { "md5": "5654fdbd3c3f6598cd6b6a7ea045a52a", "sha256": "5051b4d37f411b0ff78106bea539e8c9f9e8bd172803853918d7201b2c6746e3" }, "downloads": -1, "filename": "lnsync-0.3.5.tar.gz", "has_sig": false, "md5_digest": "5654fdbd3c3f6598cd6b6a7ea045a52a", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 58073, "upload_time": "2019-08-06T13:18:10", "url": "https://files.pythonhosted.org/packages/cb/4b/c2ff9e40945b87b8f646b6289aae0a466dba70d1db6d3c471530fad38692/lnsync-0.3.5.tar.gz" } ], "0.3.6": [ { "comment_text": "", "digests": { "md5": "42a87a16b4da376188ede38c246648d3", "sha256": "c6dc73b1d142edae83dcb5fb0b07e16cdb0b21b26a1b77631753a1c672d0387c" }, "downloads": -1, "filename": "lnsync-0.3.6.tar.gz", "has_sig": false, "md5_digest": "42a87a16b4da376188ede38c246648d3", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 59374, "upload_time": "2019-08-14T11:29:42", "url": "https://files.pythonhosted.org/packages/b9/d7/dad68920fd6bf9ad51d23d5de1aa7d8b61774f5c697bee90b1329f1b58e7/lnsync-0.3.6.tar.gz" } ], "0.3.7": [ { "comment_text": "", "digests": { "md5": "9c00517eee8193615179a2d98a0592c4", "sha256": "85f189ab71b6131cced85a1a49430ed3a8d8e626582afd88b6c198482b09ab98" }, "downloads": -1, "filename": "lnsync-0.3.7.tar.gz", "has_sig": false, "md5_digest": "9c00517eee8193615179a2d98a0592c4", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 59987, "upload_time": "2019-08-23T18:41:03", "url": "https://files.pythonhosted.org/packages/d3/e1/e06c739ee4d642c1c33918c1475184c0fac191134339c0a448cd7f815497/lnsync-0.3.7.tar.gz" } ], "0.3.8": [ { "comment_text": "", "digests": { "md5": "d2e2ae76c94c29fd0f835f716cc3c81d", "sha256": "337ddbfd8ae5aba3c16304fc3c14036fe3bd9ebd245a82dfcaf45d511c88df4b" }, "downloads": -1, "filename": "lnsync-0.3.8.tar.gz", "has_sig": false, "md5_digest": "d2e2ae76c94c29fd0f835f716cc3c81d", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 60350, "upload_time": "2019-09-12T10:11:58", "url": "https://files.pythonhosted.org/packages/86/91/8f59dd16688edeadeb89109698877683d1ba28777f2ba56d11d9c2296ab6/lnsync-0.3.8.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "d2e2ae76c94c29fd0f835f716cc3c81d", "sha256": "337ddbfd8ae5aba3c16304fc3c14036fe3bd9ebd245a82dfcaf45d511c88df4b" }, "downloads": -1, "filename": "lnsync-0.3.8.tar.gz", "has_sig": false, "md5_digest": "d2e2ae76c94c29fd0f835f716cc3c81d", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 60350, "upload_time": "2019-09-12T10:11:58", "url": "https://files.pythonhosted.org/packages/86/91/8f59dd16688edeadeb89109698877683d1ba28777f2ba56d11d9c2296ab6/lnsync-0.3.8.tar.gz" } ] }