{
"info": {
"author": "Stephen Larroque",
"author_email": "LRQ3000@gmail.com",
"bugtrack_url": null,
"classifiers": [
"Development Status :: 5 - Production/Stable",
"Environment :: Console",
"Intended Audience :: Developers",
"Intended Audience :: End Users/Desktop",
"Intended Audience :: Information Technology",
"Intended Audience :: System Administrators",
"License :: OSI Approved :: MIT License",
"Operating System :: MacOS :: MacOS X",
"Operating System :: Microsoft :: Windows",
"Operating System :: POSIX :: Linux",
"Programming Language :: Python",
"Programming Language :: Python :: 2",
"Programming Language :: Python :: 2.7",
"Programming Language :: Python :: Implementation :: PyPy",
"Topic :: Software Development :: Libraries",
"Topic :: Software Development :: Libraries :: Python Modules",
"Topic :: System :: Archiving",
"Topic :: System :: Archiving :: Backup",
"Topic :: System :: Monitoring",
"Topic :: System :: Recovery Tools",
"Topic :: Utilities"
],
"description": "pyFileFixity\n============\n\n|Build Status| |Coverage Status| |PyPi Status| |PyPi Downloads|\n\nThis project aims to provide a set of open source, cross-platform, easy\nto use and easy to maintain (readable code) to protect and manage data\nfor long term storage. The project is done in pure-Python to meet those criteria.\n\nHere is an example of what pyFileFixity can do:\n\n|Example|\n\nOn the left, this is the original image.\n\nAt the center, the same image but\nwith a few symbols corrupted (only 3 in header and 2 in the rest of the file,\nwhich equals to 5 bytes corrupted in total, over 19KB which is the total file size).\nOnly a few corrupted bytes are enough to make the image looks like totally\nunrecoverable, and yet we are lucky, because the image could be unreadable at all\nif any of the \"magic bytes\" were to be corrupted!\n\nAt the right, the corrupted image was repaired using `header_ecc.py` of pyFileFixity.\nThis repaired only the image header (ie, the first part of the file), so only the first\n3 corrupted bytes were repaired, not the 2 bytes in the rest of the file, but we can see\nthe image looks like it's totally repaired! And the best thing is that it only costed the generation\nof a \"ecc repair file\", which size is only 3.3KB (17% of the original file)!\n\nThis works because most files will store the most important information to read them at\ntheir beginning, also called \"file's header\", so repairing this part will almost always ensure\nthe possibility to read the file (even if the rest of the file is still corrupted, if the header is safe,\nyou can read it).\n\nOf course, you can also protect the whole file, not only the header, using pyFileFixity's\n`structural_adaptive_ecc.py`. You can also detect any corruption using `rfigc.py`.\n\n------------------------------------------\n\n.. contents:: Table of contents\n :backlinks: top\n\nQuickstart\n----------\n\nRuns on Python 2.7.10 and on PyPy (not yet ported to Python 3 but the libraries are already compatible).\n\n- To generate a monitoring database (to later check if files were changed, but no possibility of repairing):\n\n``python rfigc.py -i \"your_folder\" -d \"dbhash.csv\" -g -f -l \"log.txt\"``\n\nNote: this also works for a single file, just replace \"your_folder\" by \"your_file.ext\".\n\n- To check if files were corrupted:\n\n``python rfigc.py -i \"your_folder\" -d \"dbhash.csv\" -l log.txt -s -e errors.csv``\n\n- To use this monitoring database to recover files names and directory layout after filescraping:\n\n``python rfigc.py -i \"your_folder\" -d \"dbhash.csv\" -l \"log.txt\" -o \"output_folder\" --filescraping_recovery``\n\n- To protect files headers with a file called ``hecc.txt``:\n\n``python header_ecc.py -i \"your_folder\" -d \"hecc.txt\" -l \"log.txt\" -g -f --ecc_algo 3``\n\n- To repair files headers and store the repaired files in ``output_folder``:\n\n``python header_ecc.py -i \"your_folder\" -d \"hecc.txt\" -o \"output_folder\" -l \"log.txt\" -c -v --ecc_algo 3``\n\n- To protect whole files with a file called ``ecc.txt``:\n\n``python structural_adaptive_ecc.py -i \"your_folder\" -d \"ecc.txt\" -l \"log.txt\" -g -f -v --ecc_algo 3``\n\n- To repair whole files:\n\n``python structural_adaptive_ecc.py -i \"your_folder\" -d \"ecc.txt\" -o \"output_folder\" -l \"log.txt\" -c -v --ecc_algo 3``\n\n- To repair an ecc file ``ecc.txt`` using an index file ``ecc.txt.idx`` (index file is generated automatically with ecc.txt):\n\n``python repair_ecc.py -i \"ecc.txt\" --index \"ecc.txt.idx\" -o \"ecc_repaired.txt\" -l \"log.txt\" -v -f``\n\n- To repair an ecc file ``ecc.txt`` without an index file (you can tweak the ``-t`` parameter from 0.0 to 1.0, 1.0 producing many false positives):\n\n``python repair_ecc.py -i \"ecc.txt\" -o \"ecc_repaired.txt\" -l \"log.txt\" -v -f -t 0.4``\n\n- To repair your files using multiple copies that you have stored on different mediums:\n\n``replication_repair.py -i \"path/to/dir1\" \"path/to/dir2\" \"path/to/dir3\" -o \"path/to/output\" --report \"rlog.csv\" -f -v``\n\n- If you have previously generated a rfigc database, you can use it to enhance the replication repair:\n\n``replication_repair.py -i \"path/to/dir1\" \"path/to/dir2\" \"path/to/dir3\" -o \"path/to/output\" -d \"dbhash.csv\" --report \"rlog.csv\" -f -v``\n\n- To run tests on your recovery tools, you can make a Makefile-like configuration file and use:\n\n``resiliency_tester.py -i \"your_folder\" -o \"test_folder\" -c \"resiliency_tester_config.txt\" -m 3 -l \"testlog.txt\" -f``\n\n- To get more options for any tool, use ``--help``.\n\n- To use the GUI with any tool, use ``--gui`` and do not supply any other argument, eg: ``python rfigc.py --gui``.\n\n- You can also use `PyPy `_ to hugely speedup the processing time of any tool here.\n\nThe problem of long term storage\n--------------------------------\n\nWhy are data corrupted with time? Entropy, my friend, entropy.\nEntropy refers to the universal tendency for systems to become\nless ordered over time. Corruption is exactly that: a disorder\nin bits order. In other words: *the Universe hates your data*.\n\nLong term storage is thus a very difficult topic: it's like fighting with\ndeath (in this case, the death of data). Indeed, because of entropy,\ndata will eventually fade away because of various silent errors such as\nbit rot. pyFileFixity aims to provide tools to detect any data\ncorruption, but also fight data corruption by providing repairing tools.\n\nThe only solution is to use a principle of engineering that is long\nknown and which makes bridges safe: add some **redundancy**.\n\nThere are only 2 ways to add redundancy:\n\n- the simple way to add redundancy is to **duplicate** the object (also\n called replication), but for data storage, this eats up a lot of storage\n and is not optimal.\n- the second way, and the best, optimal tools ever invented to recover\n from data corruption, are the **error correction codes** (forward\n error correction), which are a way to smartly produce redundant codes\n from your data so that you can later repair your data using these\n additional pieces of information (ie, an ECC generates n blocks for a\n file cut in k blocks (with k < n), and then the ecc code can rebuild\n the whole file with (at least) any k blocks among the total n blocks\n available). In other words, you can correct up to (n-k) erasures. But\n error correcting codes can also detect and repair automatically where\n the errors are (fully automatic data repair for you !), but at the\n cost that you can then only correct (n-k)/2 errors.\n\nError correction can seem a bit magical, but for a reasonable intuition,\nit can be seen as a way to average the corruption error rate: on\naverage, a bit will still have the same chance to be corrupted, but\nsince you have more bits to represent the same data, you lower the\noverall chance to lose this bit.\n\nThe problem is that most theoretical and pratical works on error\ncorrecting codes has been done almost exclusively on channel\ntransmission (such as 4G, internet, etc.), but not on data storage,\nwhich is very different for one reason: whereas in a channel we are in a\nspatial scheme (both the sender and the receiver are different entities\nin space but working at the same timescale), in data storage this is a\ntemporal scheme: the sender was you storing the data on your medium at\ntime t, and the receiver is again you but now retrieving the data at\ntime t+x. Thus, the sender does not exist anymore, thus you cannot ask\nthe sender to send again some data if it's too much corrupted: in data\nstorage, if a data is corrupted, it's lost for good, whereas in channel theory,\nparts of the data can be submitted again if necessary.\n\nSome attempts were made to translate channel theory and error correcting\ncodes theory to data storage, the first being Reed-Solomon which spawned\nthe RAID schema. Then CIRC (Cross-interleaved Reed-Solomon coding) was\ndevised for use on optical discs to recover from scratches, which was\nnecessary for the technology to be usable for consumers. Since then, new\nless-optimal but a lot faster algorithms such as LDPC, turbo-codes and\nfountain codes such as RaptorQ were invented (or rediscovered), but they\nare still marginally researched for data storage.\n\nThis project aims to, first, implement easy tools to evaluate strategies\n(filetamper.py) and file fixity (ie, detect if there are corruptions),\nand then the goal is to provide an open and easy framework to use\ndifferent kinds of error correction codes to protect and repair files.\n\nAlso, the ecc file specification is made to be simple and resilient to\ncorruption, so that you can process it by your own means if you want to,\nwithout having to study for hours how the code works (contrary to PAR2\nformat).\n\nWhy not just use RAID ?\n-----------------------\n\nRAID is clearly insufficient for long-term data storage, and in fact it\nwas primarily meant as a cheap way to get more storage (RAID0) or more\navailability (RAID1) of data, not for archiving data, even on a medium\ntimescale:\n\n- RAID 0 is just using multiple disks just like a single one, to extend\n the available storage. Let's skip this one.\n- RAID 1 is mirroring one disk with a bit-by-bit copy of another disk.\n That's completely useless for long term storage: if either disk\n fails, or if both disks are partially corrupted, you can't know what\n are the correct data and which aren't. As an old saying goes: \"Never\n take 2 compasses: either take 3 or 1, because if both compasses show\n different directions, you will never know which one is correct, nor\n if both are wrong.\" That's the principle of Triplication.\n- RAID 5 is based on the triplication idea: you have n disks (but least\n 3), and if one fails you can recover n-1 disks (resilient to only 1\n disk failure, not more).\n- RAID 6 is an extension of RAID 5 which is closer to error-correction\n since you can correct n-k disks. However, most (all?) currently\n commercially available RAID6 devices only implements recovery for at\n most n-2 (2 disks failures).\n- In any case, RAID cannot detect silent errors automatically, thus you\n either have to regularly scan, or you risk to lose some of your data\n permanently, and it's far more common than you can expect (eg, with\n RAID5, it is enough to have 2 silent errors on two disks on the same\n bit for the bit to be unrecoverable). That's why a limit of only 1 or\n 2 disks failures is just not enough.\n\nOn the opposite, ECC can correct n-k disks (or files). You can configure\nn and k however you want, so that for example you can set k = n/2, which\nmeans that you can recover all your files from only half of them! (once\nthey are encoded with an ecc file of course).\n\nThere also are new generation RAID solutions, mainly software based,\nsuch as SnapRAID or ZFS, which allow you to configure a virtual RAID\nwith the value n-k that you want. This is just like an ecc file (but a\nbit less flexible, since it's not a file but a disk mapping, so that you\ncan't just copy it around or upload it to a cloud backup hosting). In\naddition to recover (n-k) disks, they can also be configured to recover\nfrom partial, sectors failures inside the disk and not just the whole\ndisk (for a more detailed explanation, see Plank, James S., Mario Blaum,\nand James L. Hafner. \"SD codes: erasure codes designed for how storage\nsystems really fail.\" FAST. 2013.).\n\nThe other reason RAID is not adapted to long-term storage, is that it\nsupposes you store your data on hard-drives exclusively. Hard drives\naren't a good storage medium for the long term, for two reasons:\n\n| 1- they need a regular plug to keep the internal magnetic disks\n electrified (else the data will just fade away when there's no\n residual electricity).\n| 2- the reading instrument is directly included and merged with the\n data (this is the green electronic board you see from the outside, and\n the internal head). This is good for quick consumer use (don't need to\n buy another instrument: the HDD can just be plugged and it works), but\n it's very bad for long term storage, because the reading instrument is\n bound to fail, and a lot faster than the data can fade away: this\n means that even if your magnetic disks inside your HDD still holds\n your data, if the controller board or the head doesn't work anymore,\n your data is just lost. And a head (and a controller board) are almost\n impossible to replace, even by professionals, because the pieces are\n VERY hard to find (different for each HDD production line) and each\n HDD has some small physical defects, thus it's impossible to reproduce\n that too (because the head is so close to the magnetic disk that if\n you try to do that manually you'll probably fail).\n\nIn the end, it's a lot better to just separate the storage medium of\ndata, with the reading instrument. The medium I advise is optical disks\n(whether it's BluRay, DVD, CD or whatever), because the reading\ninstrument is separate, and the technology (laser reflecting on bumps\nand/or pits) is kind of universal, so that even if the technology is\nlost one day (deprecated by newer technologies, so that you can't find\nthe reading instrument anymore because it's not sold anymore), you can\nprobably emulate a laser using some software to read your optical disk,\njust like what the CAMiLEON project did to recover data from the\nLaserDiscs of the BBC Domesday Project (see Wikipedia).\n\nApplications included\n---------------------\n\nThe project currently include the following pure-python applications:\n\n- rfigc.py, a hash auditing tool, similar to md5deep/hashdeep, to\n compute a database of your files along with their metadata, so that\n later you can check if they were changed/corrupted.\n\n- header\\_ecc.py, an error correction code using Reed-Solomon\n generator/corrector for files headers. The idea is to supplement\n other more common redundancy tools such as PAR2 (which is quite\n reliable), by adding more resiliency only on the critical parts of\n the files: their headers. Using this script, you can significantly\n higher the chance of recovering headers, which will allow you to at\n least open the files.\n\n- structural\\_adaptive\\_ecc.py, a variable error correction rate\n encoder (kind of a generalization of header\\_ecc.py). This script\n allows to generate an ecc file for the whole content of your files,\n not just the header part, using a variable resilience rate: the\n header part will be the most protected, then the rest of each file\n will be progressively encoded with a smaller and smaller resilience\n rate. The assumption is that important information is stored first,\n and then data becomes less and less informative (and thus important,\n because the end of the file describes less important details). This\n assumption is very true for all compressed kinds of formats, such as\n JPG, ZIP, Word, ODT, etc...\n\n- repair\\_ecc.py, a script to repair the structure (ie, the entry and\n fields markers/separators) of an ecc file generated by header\\_ecc.py\n or structural\\_adaptive\\_ecc.py. The goal is to enhance the\n resilience of ecc files against corruption by ensuring that their\n structures can be repaired (up to a certain point which is very high\n if you use an index backup file, which is a companion file that is\n generated along an ecc file).\n\n- filetamper.py is a quickly made file corrupter, it will erase or\n change characters in the specified file. This is useful for testing\n your various protecting strategies and file formats (eg: is PAR2\n really resilient against corruption? Are zip archives still partially\n extractable after corruption or are rar archives better? etc.). Do\n not underestimate the usefulness of this tool, as you should always\n check the resiliency of your file formats and of your file protection\n strategies before relying on them.\n\n- easy\\_profiler.py is just a quick and simple profiling tool to get\n you started quickly on what should be optimized to get more speed, if\n you want to contribute to the project feel free to propose a pull\n request! (Cython and other optimizations are welcome as long as they\n are cross-platform and that an alternative pure-python implementation\n is also available).\n\n- replication\\_repair.py takes advantage of your multiple copies\n (replications) of your data over several storage mediums to recover\n your data in case it gets corrupted. The goal is to take advantage of\n the storage of your archived files into multiple locations: you will\n necessarily make replications, so why not use them for repair?\n Indeed, it's good practice to keep several identical copies of your data\n on several storage mediums, but in case a corruption happens,\n usually you will just drop the corrupted copies and keep the intacts ones.\n However, if all copies are partially corrupted, you're stuck. This script\n aims to take advantage of these multiple copies to recover your data,\n without generating a prior ecc file. It works simply by reading through all\n your different copies of your data, and it casts a majority vote over each\n byte: the one that is the most often occuring will be kept. In engineering,\n this is a very common strategy used for very reliable systems such as\n space rockets, and is called \"triple-modular redundancy\", because you need\n at least 3 copies of your data for the majority vote to work (but the more the\n better).\n\n- resiliency\\_tester.py allows you to test the robustness of the\n corruption correction of the scripts provided here (or any other\n command-line app). You just have to copy the files you want to test inside a\n folder, and then the script will copy the files into a test tree, then it\n will automatically corrupt the files randomly (you can change the parameters\n like block burst and others), then it will run the file repair command-lines\n you supply and finally some stats about the repairing power will be\n generated. This allows you to easily and objectively compare different set\n of parameters, or even different file repair solutions, on the very data\n that matters to you, so that you can pick the best option for you.\n\nNote that all tools are primarily made for command-line usage (type\nscript.py --help to get extended info about the accepted arguments), but\nyou can also use rfigc.py and header\\_ecc.py with a GUI by using the\n--gui argument (must be the first and only one argument supplied). The\nGUI is provided as-is and minimal work will be done to maintain it (the\nfocus will stay on functionality rather than ergonomy).\n\nIMPORTANT: it is CRITICAL that you use the same parameters for\ncorrecting mode as when you generated the database/ecc files (this is\ntrue for all scripts in this bundle). Of course, some options must be\nchanged: -g must become -c to correct, and --update is a particular\ncase. This works this way on purpose for mainly two reasons: first\nbecause it is very hard to autodetect the parameters from a database\nfile alone and it would produce lots of false positives, and secondly\n(the primary reason) is that storing parameters inside the database file\nis highly unresilient against corruption (if this part of the database\nis tampered, the whole becomes unreadable, while if they are stored\noutside or in your own memory, the database file is always accessible).\nThus, it is advised to write down the parameters you used to generate\nyour database directly on the storage media you will store your database\nfile on (eg: if it's an optical disk, write the parameters on the cover\nor directly on the disk using a marker), or better memorize them by\nheart. If you forget them, don't panic, the parameters are always stored\nas comments in the header of the generated ecc files, but you should try\nto store them outside of the ecc files anyway.\n\nFor users: what's the advantage of pyFileFixity?\n------------------------------------------------\n\nPros:\n\n- Open application and open specifications under the MIT license (you\n can do whatever you want with it and tailor it to your needs if you\n want to, or add better decoding procedures in the future as science\n progress so that you can better recover your data from your already\n generated ecc file).\n- Highly reliable file fixity watcher: rfigc.py will tell you without\n any ambiguity using several attributes if your files have been\n corrupted or not, and can even check for images if the header is\n valid (ie: if the file can still be opened).\n- Readable ecc file format (compared to PAR2 and most other similar\n specifications).\n- Highly resilient ecc file format against corruption (not only are\n your data protected by ecc, the ecc file is protected too against\n critical spots, both because there is no header so that each track is\n independent and if one track is corrupted beyond repair then other\n ecc tracks can still be read, and a .idx file will be generated to\n repair the structure of the ecc file to recover all tracks).\n- Very safe and conservative approach: the recovery process checks that\n the recovery was successful before committing a repaired block.\n- Partial recovery allowed (even if a file cannot be completely\n recovered, the parts that can will be repaired and then the rest that\n can't be repaired will be recopied from the corrupted version).\n- Support directory processing: you can encode an ecc file for a whole\n directory of files (with any number of sub-directories and depth).\n- No limit on the number of files, and it can recursively protect files\n in a directory tree.\n- Variable resiliency rate and header-only resilience, ensuring that\n you can always open your files even if partially corrupted (the\n structure of your files will be saved, so that you can use other\n softwares to repair beyond if this set of script is not sufficient to\n totally repair).\n- Support for erasures (null bytes) and even errors-and-erasures, which\n literally doubles the repair capabilities. To my knowledge, this is\n the only freely available parity software that supports erasures.\n- Display the predicted total ecc file size given your parameters,\n and the total time it will take to encode/decode.\n- No external library needed, only native Python 2.7.x (but with PyPy\n it will be way faster!).\n- Opensourced under the very permissive MIT licence, do whatever you\n want!\n\nCons:\n\n- Cannot protect meta-data, such as folders paths. The paths are\n stored, but cannot be recovered (yet? feel free to contribute if you\n know how). Only files are protected. Thus if your OS or your storage\n medium crashes and truncate a whole directory tree, the directory\n tree can't be repaired using the ecc file, and thus you can't access\n the files neither. However, you can use file scraping to extract the\n files even if the directory tree is lost, and then use RFIGC.py to\n reorganize your files correctly. There are alternatives, see the\n chapters below: you can either package all your files in a single\n archive using DAR or ZIP (thus the ecc will also protect meta-data), or see\n DVDisaster as an alternative solution, which is an ecc generator with\n support for directory trees meta-data (but only on optical disks).\n- Can only repair errors and erasures (characters that are replaced by\n another character), not deletion nor insertion of characters. However\n this should not happen with any storage medium (truncation can occur\n if the file bounds is misdetected, in this case pyFileFixity can\n partially repair the known parts of the file, but cannot recover the\n rest past the truncation, except if you used a resiliency rate of at\n least 0.5, in which case any message block can be recreated with only\n using the ecc file).\n- Cannot recreate a missing file from other available files (except you\n have set a resilience\\_rate at least 0.5), contrary to Parchives\n (PAR1/PAR2). Thus, you can only repair a file if you still have it on\n your filesystem. If it's missing, pyFileFixity cannot do anything\n (yet, this will be implemented in the future).\n\nNote that the tools were meant for data archival (protect files that you\nwon't modify anymore), not for system's files watching nor to protect\nall the files on your computer. To do this, you can use a filesystem\nthat directly integrate error correction code capacity, such as ZFS.\n\nRecursive/Relative Files Integrity Generator and Checker in Python (aka RFIGC)\n------------------------------------------------------------------------------\n\nRecursively generate or check the integrity of files by MD5 and SHA1\nhashes, size, modification date or by data structure integrity (only for\nimages).\n\nThis script is originally meant to be used for data archival, by\nallowing an easy way to check for silent file corruption. Thus, this\nscript uses relative paths so that you can easily compute and check the\nsame redundant data copied on different mediums (hard drives, optical\ndiscs, etc.). This script is not meant for system files corruption\nnotification, but is more meant to be used from times-to-times to check\nup on your data archives integrity (if you need this kind of application,\nsee `avpreserve's fixity `_).\n\nThis script was made for Python 2.7.6, but it should be easily adaptable\nto run on Python 3.x.\n\nExample usage\n~~~~~~~~~~~~~\n\n- To generate the database (only needed once):\n\n``python rfigc.py -i \"your_folder\" -d \"dbhash.csv\" -g``\n\n- To check:\n\n``python rfigc.py -i \"your_folder\" -d \"dbhash.csv\" -l log.txt -s``\n\n- To update your database by appending new files:\n\n``python rfigc.py -i \"your_folder\" -d \"dbhash.csv\" -u -a``\n\n- To update your database by appending new files AND removing\n inexistent files:\n\n``python rfigc.py -i \"your_folder\" -d \"dbhash.csv\" -u -a -r``\n\nNote that by default, the script is by default in check mode, to avoid\nwrong manipulations. It will also alert you if you generate over an\nalready existing database file.\n\nArguments\n~~~~~~~~~\n\n::\n\n -h, --help show a help message and exit\n -i /path/to/root/folder, --input /path/to/root/folder\n Path to the root folder from where the scanning will occ\n ur.\n -d /some/folder/databasefile.csv, --database /some/folder/databasefile.csv\n Path to the csv file containing the hash informations.\n -l /some/folder/filename.log, --log /some/folder/filename.log\n Path to the log file. (Output will be piped to both the\n stdout and the log file)\n -s, --structure_check\n Check images structures for corruption?\n -e /some/folder/errorsfile.csv, --errors_file /some/folder/errorsfile.csv\n Path to the error file, where errors at checking will be\n stored in CSV for further processing by other softwares (such as file repair so\n ftwares).\n -m, --disable_modification_date_checking\n Disable modification date checking.\n --skip_missing Skip missing files when checking (useful if you split yo\n ur files into several mediums, for example on optical discs with limited capacit\n y).\n -g, --generate Generate the database? (omit this parameter to check ins\n tead of generating).\n -f, --force Force overwriting the database file even if it already e\n xists (if --generate).\n -u, --update Update database (you must also specify --append or --rem\n ove).\n -a, --append Append new files (if --update).\n -r, --remove Remove missing files (if --update).\n \n --filescraping_recovery Given a folder of unorganized files, compare to the database and restore the filename and directory structure into the output folder.\n -o, --output Path to the output folder where to output the files reorganized after --recover_from_filescraping.\n\nHeader Error Correction Code script\n-----------------------------------\n\nThis script was made to be used in combination with other more common\nfile redundancy generators (such as PAR2, I advise MultiPar). This is an\nadditional layer of protection for your files: by using a higher\nresiliency rate on the headers of your files, you ensure that you will\nbe probably able to open them in the future, avoiding the \"critical\nspots\", also called \"fracture-critical\" in redundancy engineering (where\nif you modify just one bit, your whole file may become unreadable,\nusually bits residing in the headers - in other words, a single blow\nmakes the whole thing collapse, just like non-redundant bridges).\n\nAn interesting benefit of this approach is that it has a low storage\n(and computational) overhead that scales linearly to the number of\nfiles, whatever their size is: for example, if we have a set of 40k\nfiles for a total size of 60 GB, with a resiliency\\_rate of 30% and\nheader\\_size of 1KB (we limit to the first 1K bytes/characters = our\nfile header), then, without counting the hash per block and other\nmeta-data, the final ECC file will be about 2 \\* resiliency\\_rate \\*\nnumber\\_of\\_files \\* header\\_size = 24.5 MB. This size can be lower if\nthere are many files smaller than 1KB. This is a pretty low storage\noverhead to backup the headers of such a big number of files.\n\nThe script is pure-python as are its dependencies: it is thus completely\ncross-platform and open source. However, this imply that it is quite\nslow, but PyPy v2.5.0 was successfully tested against the script without\nany modification, and a speed increase of more 100x could be observed,\nso that you can expect a rate of more than 1MB/s, which is quite fast.\n\nStructural Adaptive Error Correction Encoder\n--------------------------------------------\n\nThis script implements a variable error correction rate encoder: each\nfile is ecc encoded using a variable resiliency rate -- using a high\nconstant resiliency rate for the header part (resiliency rate stage 1,\nhigh), then a variable resiliency rate is applied to the rest of the\nfile's content, with a higher rate near the beginning of the file\n(resiliency rate stage 2, medium) which progressively decreases until\nthe end of file (resiliency rate stage 3, the lowest).\n\nThe idea is that the critical parts of files usually are placed at the\ntop, and data becomes less and less critical along the file. What is\nmeant by critical is both the critical spots (eg: if you tamper only one\ncharacter of a file's header you have good chances of losing your entire\nfile, ie, you cannot even open it) and critically encoded information\n(eg: archive formats usually encode compressed symbols as they go along\nthe file, which means that the first occurrence is encoded, and then the\narchive simply writes a reference to the symbol. Thus, the first\noccurrence is encoded at the top, and subsequent encoding of this same\ndata pattern will just be one symbol, and thus it matters less as long\nas the original symbol is correctly encoded and its information\npreserved, we can always try to restore the reference symbols later).\nMoreover, really redundant data will be placed at the top because they\ncan be reused a lot, while data that cannot be too much compressed will\nbe placed later, and thus, corruption of this less compressed data is a\nlot less critical because only a few characters will be changed in the\nuncompressed file (since the data is less compressed, a character change\non the not-so-much compressed data won't have very significant impact on\nthe uncompressed data).\n\nThis variable error correction rate should allow to protect more the\ncritical parts of a file (the header and the beginning of a file, for\nexample in compressed file formats such as zip or jpg this is where the\nmost importantly strings are encoded) for the same amount of storage as\na standard constant error correction rate.\n\nOf course, you can set the resiliency rate for each stage to the values\nyou want, so that you can even do the opposite: setting a higher\nresiliency rate for stage 3 than stage 2 will produce an ecc that is\ngreater towards the end of the contents of your files.\n\nFurthermore, the currently designed format of the ecc file would allow\ntwo things that are not available in all current file ecc generators\nsuch as PAR2:\n\n1. it allows to partially repair a file, even if not all\nthe blocks can be corrected (in PAR2, a file is repaired only if all\nblocks can be repaired, which is a shame because there are still other\nblocks that could be repaired and thus produce a less corrupted file) ;\n\n2. the ecc file format is quite simple and readable, easy to process by\nany script, which would allow other softwares to also work on it (and it\nwas also done in this way to be more resilient against error\ncorruptions, so that even if an entry is corrupted, other entries are\nindependent and can maybe be used, thus the ecc is very error tolerant.\nThis idea was implemented in repair\\_ecc.py but it could be extended,\nespecially if you know the pattern of the corruption).\n\nThe script structural-adaptive-ecc.py implements this idea, which can be\nseen as an extension of header-ecc.py (and in fact the idea was the\nother way around: structural-adaptive-ecc.py was conceived first but was\ntoo complicated, then header-ecc.py was implemented as a working\nlessened implementation only for headers, and then\nstructural-adaptive-ecc.py was finished using header-ecc.py code\nprogress). It works, it was a quite well tested for my own needs on\ndatasets of hundred of GB, but it's not foolproof so make sure you test\nthe script by yourself to see if it's robust enough for your needs (any\nfeedback about this would be greatly appreciated!).\n\nECC Algorithms\n--------------\n\nYou can specify different ecc algorithms using the ``--ecc_algo`` switch.\n\nFor the moment, only Reed-Solomon is implemented, but it's universal\nso you can modify its parameters in lib/eccman.py.\n\nTwo Reed-Solomon codecs are available, they are functionally equivalent\nand thoroughly unit tested.\n\n- ``--ecc_algo 1``: use the first Reed-Solomon codec in galois field 2^8 of root 3 with fcr=1.\n This is the slowest implementation (but also the most easy code to understand).\n- ``--ecc_algo 2``: same as algo 1 but with a faster functions.\n- ``--ecc_algo 3``: use the second codec, which is the fastest.\n The generated ECC will be compatible with algo 1 and 2.\n- ``--ecc_algo 4``: also use the second, fastest RS codec, but\n with different parameters (US FAA ADSB UAT RS FEC norm),\n thus the generated ECC won't be compatible with algo 1 to 3.\n But do not be scared, the ECC will work just the same.\n\nCython implementation\n---------------------\n\nThis section describes how to use the Cython implementation. However,\nyou should first try PyPy, as it did give 10x to 100x speedup over\nCython in our case.\n\nA speedy Cython implementation of the Reed-Solomon library is included.\nIt should provide C-speed for all scripts (as long as you use\n--ecc\\_algo 1 or 2, not 3 nor 4). It is not needed, since a pure-python\nimplementation is used by default, but it can be useful if you want to\nencode big datasets of several hundred of GB.\n\nIf you want to build the C/Cython implementation, do the following:\n\n1- Install a C compiler for your platform. On Linux, gcc should already\nbe installed. On Windows, you need to use the Visual Studio C compiler\n(not MinGW nor Cygwin gcc, they won't work). You can use the \"Microsoft\nVisual C++ Compiler for Python 2.7\", and follow these instructions to\nmake it work if you have Python < 2.7.10:\n\nhttps://github.com/cython/cython/wiki/CythonExtensionsOnWindows\n\n2- cd to this folder (where pyFileFixity resides), and execute the\nfollowing command:\n\n``python setup.py build_ext --inplace --compiler=msvc``\n\nIf everything goes alright, the C compiler will compile the .c files\n(that were pre-generated by Cython) and you can then use PyFileFixity\nscripts just as usual and you should see a huge speedup. Else, if it\ndoesn't work, you might need to generate .c files using Cython for your\nplatform (because the pre-generated .c files may be incompatible with\nyour platform). To do that, you just need to install Cython, which is an\neasy task with nowadays Python distributions such as Anaconda: download\n32-bit Anaconda installer (on Windows you should avoid the 64-bit, it\nmay produce weird issues with Cython), then after install, open the\nAnaconda Command Prompt and execute: ``conda install cython``. This will\ninstall all the necessary stuff along the cython library. Then you can\nsimply execute again the command\n``python setup.py build_ext --inplace --compiler=msvc`` and it will this\ntime rebuild from scratch, by autodetecting that you have Cython\ninstalled, the setup.py script will automatically generate .c files from\n.pyx files and then .pyd files (binaries) from .c files.\n\nIf you get issues, you can see the following post on how to install\nCython:\n\nhttps://github.com/cython/cython/wiki/InstallingOnWindows\n\n3- You can now launch pyFileFixity like usual, it should automatically\ndetect the C/Cython compiled files and use that to speedup processing.\n\nNote about speed: Also, use a smaller --max\\_block\\_size to greatly\nspeedup the operations! That's the trick used to compute very quickly RS\nECC on optical discs. You give up a bit of resiliency of course (because\nblocks are smaller, thus you protect a smaller number of characters per\nECC. In the end, this should not change much about real resiliency, but\nin case you get a big bit error burst on a contiguous block, you may\nlose a whole block at once. That's why using RS255 is better, but it's\nvery time consuming. However, the resiliency ratios still hold, so for\nany other case of bit-flipping with average-sized bursts, this should\nnot be a problem as long as the size of the bursts is smaller than an\necc block.)\n\nIn case of a catastrophic event\n-------------------------------\n\nTODO: write more here\n\nIn case of a catastrophic event of your data due to the failure of your\nstorage media (eg: your hard drive crashed), then follow the following\nsteps:\n\n1- use dd\\_rescue to make a full bit-per-bit verbatim copy of your drive\nbefore it dies. The nice thing with dd\\_rescue is that the copy is\nexact, and also that it can retries or skip in case of bad sectors (it\nwon't crash on your suddenly at half the process).\n\n2- Use testdisk to restore partition or to copy files based on partition\nfilesystem informations.\n\n3- If you could not recover your files, you can try file scraping using\n`photorec `_ or\n`plaso `_ other similar tools as\na last resort to extract data based only from files content (no filename,\noften uncorrect filetype, file boundaries may be wrong so some data\nmay be cut off, etc.).\n\n4- If you used pyFileFixity before the failure of your storage media,\nyou can then use your pre-computed databases to check that files are\nintact (rfigc.py) and if they aren't, you can recover them (using\nheader\\_ecc.py and structural\\_adaptive\\_ecc.py). It can also help if\nyou recovered your files via data scraping, because your files will be\ntotally unorganized, but you can use a previously generated database\nfile to recover the full names and directory tree structure using\nrfigc.py --filescraping\\_recover.\n\nAlso, you can try to fix some of your files using specialized repairing\ntools (but remember that such tool cannot guarantee you the same\nrecovering capacity as an error correction code - and in addition, error\ncorrection code can tell you when it has recovered successfully). For\nexample:\n\n- for tar files, you can use `fixtar `_.\n Similar tools (but older): `tarfix `_\n and `tar-repair `_.\n- for RAID mounting and recovery, you can use \"Raid faster - recover\n better\" (rfrb) tool by Sabine Seufert and Christian Zoubek:\n https://github.com/lrq3000/rfrb\n- if your unicode strings were mangled (ie, you see weird symbols),\n try this script that will automatically demangle them:\n https://github.com/LuminosoInsight/python-ftfy\n- to repair tabular (2D) data such as .csv, try\n `Carpenter `_.\n\nProtecting directory tree meta-data\n-----------------------------------\n\nOne main current limitation of pyFileFixity is that it cannot protect\nthe directory tree meta-data. This means that in the worst case, if a\nsilent error happens on the inode pointing to the root directory that\nyou protected with an ecc, the whole directory will vanish, and all the\nfiles inside too. In less worst cases, sub-directories can vanish, but\nit's still pretty bad, and since the ecc file doesn't store any\ninformation about inodes, you can't recover the full path.\n\nThe inability to store these meta-data is because of two choices in the\ndesign: 1- portability: we want the ecc file to work even if we move the\nroot directory to another place or another storage medium (and of\ncourse, the inode would change), 2- cross-platform compatibility:\nthere's no way to get and store directory meta-data for all platforms,\nbut of course we could implement specific instructions for each main\nplatform, so this point is not really a problem.\n\nTo workaround this issue (directory meta-data are critical spots), other\nsoftwares use a one-time storage medium (ie, writing your data along\nwith generating and writing the ecc). This way, they can access at\nthe bit level the inode info, and they are guaranted that the inodes\nwon't ever change. This is the approach taken by DVDisaster: by using\noptical mediums, it can compute inodes that will be permanent, and thus\nalso encode that info in the ecc file. Another approach is to create a\nvirtual filesystem specifically to store just your files, so that you\nmanage the inode yourself, and you can then copy the whole filesystem\naround (which is really just a file, just like a zip file - which can\nalso be considered as a mini virtual file system in fact) like\n`rsbep `_.\n\nHere the portability principle of pyFileFixity prevents this approach.\nBut you can mimic this workaround on your hard drive for pyFileFixity to\nwork: you just need to package all your files into one file. This way,\nyou sort of create a virtual file system: inside the archive, files and\ndirectories have meta-data just like in a filesystem, but from the\noutside it's just one file, composed of bytes that we can just encode to\ngenerate an ecc file - in other words, we removed the inodes portability\nproblem, since this meta-data is stored relatively inside the archive,\nthe archive manage it, and we can just encode this info like any other\nstream of data! The usual way to make an archive from several files is\nto use TAR, but this will generate a solid archive which will prevent\npartial recovery. An alternative is to use DAR, which is a non-solid\narchive version of TAR, with lots of other features too. If you also\nwant to compress, you can just use ZIP (with DEFLATE algorithm) your\nfiles (this also generates a non-solid archive). You can then use\npyFileFixity to generate an ecc file on your DAR or ZIP archive, which\nwill then protect both your files just like before and the directories\nmeta-data too now.\n\nTools like pyFileFixity (or which can be used as complements)\n-------------------------------------------------------------\n\nHere are some tools with a similar philosophy to pyFileFixity, which you\ncan use if they better fit your needs, either as a replacement of\npyFileFixity or as a complement (pyFileFixity can always be used to\ngenerate an ecc file):\n\n- `DAR (Disk ARchive) `__: similar to tar\n but non-solid thus allows for partial recovery and per-file access,\n plus it saves the directory tree meta-data -- see catalog isolation\n -- plus it can handle error correction natively using PAR2 and\n encryption. Also supports incremental backup, thus it's a very nice\n versatile tool. Crossplatform and opensource.\n- `DVDisaster `__: error correction at the bit\n level for optical mediums (CD, DVD and BD / BluRay Discs). Very good,\n it also protects directory tree meta-data and is resilient to\n corruption (v2 still has some critical spots but v3 won't have any).\n- rsbep tool that is part of dvbackup package in Debian: allows to\n generate an ecc of a stream of bytes. Great to pipe to dar and/or gz\n for your backups, if you're on unix or using cygwin.\n- `rsbep modification by Thanassis\n Tsiodras `__:\n enhanced rsbep to avoid critical spots and faster speed. Also\n includes a \"freeze\" script to encode your files into a virtual\n filesystem (using Python/FUSE) so that even meta-data such as\n directory tree are fully protected by the ecc. Great script, but not\n maintained, it needs some intensive testing by someone knowledgeable\n to guarantee this script is reliable enough for production.\n- Parchive (PAR1, PAR2, MultiPar): well known error correction file\n generator. The big advantage of Parchives is that an ecc block\n depends on multiple files: this allows to completely reconstruct a\n missing file from scratch using files that are still available. Works\n good for most people, but most available Parchive generators are not\n satisfiable for me because 1- they do not allow to generate an ecc\n for a directory tree recursively (except MultiPar, and even if it is\n allowed in the PAR2 specs), 2- they can be very slow to generate\n (even with multiprocessor extensions, because the galois field is\n over 2^16 instead of 2^8, which is very costly), 3- the spec is not\n very resilient to errors and tampering over the ecc file, as it\n assumes the ecc file won't be corrupted (I also tested, it's still a\n bit resilient, but it could be a lot more with some tweaking of the\n spec), 4- it doesn't allow for partial recovery (recovering blocks\n that we can and pass the others that are unrecoverable): with PAR2, a\n file can be restored fully or it cannot be at all.\n- Zip (with DEFLATE algorithm, using 7-Zip or other tools): allows to\n create non-solid archives which are readable by most computers\n (ubiquitous algorithm). Non-solid archive means that a zip file can\n still unzip correct files even if it is corrupted, because files are\n encoded in blocks, and thus even if some blocks are corrupted, the\n decoding can happen. A `fast implementation with enhanced compression\n is available in pure Go `__\n (good for long storage).\n- TestDisk: for file scraping, when nothing else worked.\n- dd\\_rescue: for disk scraping (allows to forcefully read a whole disk\n at the bit level and copy everything it can, passing bad sector with\n options to retry them later on after a first full pass over the\n correct sectors).\n- ZFS: a file system which includes ecc correction directly. The whole\n filesystem, including directory tree meta-data, are protected. If you\n want ecc protection on your computer for all your files, this is the\n way to go.\n- Encryption: technically, you can encrypt your files without losing\n too much redundancy, as long as you use an encryption scheme that is\n block-based such as DES: if one block gets corrupted, it won't be\n decryptable, but the rest of the files' encrypted blocks should be\n decryptable without any problem. So encrypting with such algorithms\n leads to similar files as non-solid archives such as deflate zip. Of\n course, for very long term storage, it's better to avoid encryption\n and compression (because you raise the information contained in a\n single block of data, thus if you lose one block, you lose more\n data), but if it's really necessary to you, you can still maintain\n high chances of recovering your files by using block-based\n encryption/compression (note: block-based encryption can\n be seen as the equivalent of non-solid archives for compression,\n because the data is compressed/encrypted in independent blocks,\n thus allowing partial uncompression/decryption).\n- `SnapRAID `__\n- `par2ools `__: a set of\n additional tools to manage par2 archives\n- `Checkm `__: a tool similar\n to rfigc.py\n- `BagIt `__ with two python\n implementations `here `__ and\n `here `__: this is a file\n packaging format for sharing and storing archives for long term\n preservation, it just formalizes a few common procedures and meta\n data that are usually added to files for long term archival (such as\n MD5 digest).\n- `RSArmor `__ a tool based on\n Reed-Solomon to encode binary data files into hexadecimal, so that\n you can print the characters on paper. May be interesting for small\n datasets (below 100 MB).\n- `Ent `__ a tool to analyze the\n entropy of your files. Can be very interesting to optimize the error\n correction algorithm, or your compression tools.\n- `HashFS `_ is a non-redundant,\n duplication free filesystem, in Python. **Data deduplication** is very\n important for large scale long term storage: since you want your data\n to be redundant, this means you will use an additional storage space\n for your redundant copies that will be proportional to your original data.\n Having duplicated data will consume more storage and more processing\n time, for no benefit. That's why it's a good idea to deduplicate your data\n prior to create redundant copies: this will be faster and save you money.\n Deduplication can either be done manually (by using duplicates removers)\n or systematically and automatically using specific filesystems such as\n zfs (with deduplication enabled) or hashfs.\n- Paper as a storage medium: paper is not a great storage medium,\n because it has low storage density (ie, you can only store at most \n about 100 KB) and it can also degrade just like other storage mediums,\n but you cannot check that automatically since it's not digital. However,\n if you are interested, here are a few softwares that do that:\n `Paper key `_,\n `Paperbak `_,\n `Optar `_,\n `dpaper `_,\n `QR Backup `_,\n `QR Backup (another) `_,\n `QR Backup (again another) `_,\n `QR Backup (again) `_,\n `and finally a related paper `_.\n- AVPreserve tools, most notably `fixity `_ \n to monitor for file changes (similarly to rfigc, but actively as a daemon)\n and `interstitial `_ to detect\n interstitial errors in audio digitization workflows (great to ensure you\n correctly digitized a whole audio file into WAV without any error).\n\nFAQ\n---\n\n- Can I compress my data files and my ecc file?\n\nAs a rule of thumb, you should ALWAYS keep your ecc file in clear\ntext, so under no compression nor encryption. This is because in case\nthe ecc file gets corrupted, if compressed/encrypted, the\ndecompression/decrypting of the corrupted parts may completely flaw\nthe whole structure of the ecc file.\n\nYour data files, that you want to protect, *should* remain in clear\ntext, but you may choose to compress them if it drastically reduces\nthe size of your files, and if you raise the resilience rate of your\necc file (so compression may be a good option if you have an\nopportunity to trade the file size reduction for more ecc file\nresilience). Also, make sure to choose a non-solid compression\nalgorithm like DEFLATE (zip) so that you can still decode correct\nparts even if some are corrupted (else with a solid archive, if one\nbyte is corrupted, the whole archive may become unreadable).\n\nHowever, in the case that you compress your files, you should generate\nthe ecc file only *after* compression, so that the ecc file applies to\nthe compressed archive instead of the uncompressed files, else you\nrisk being unable to correct your files because the uncompression of\ncorrupted parts may output gibberish, and length extended corrupted\nparts (and if the size is different, Reed-Solomon will just freak\nout).\n\n- Can I encrypt my data files and my ecc file ?\n\nNEVER encrypt your ecc file, this is totally useless and\ncounterproductive.\n\nYou can encrypt your data files, but choose a non-solid algorithm\n(like AES if I'm not mistaken) so that corrupted parts do not prevent\nthe decoding of subsequent correct parts. Of course, you're lowering a\nbit your chances of recovering your data files by encrypting them (the\nbest chance to keep data for the long term is to keep them in clear\ntext), but if it's really necessary, using a non-solid encrypting\nscheme is a good compromise.\n\nYou can generate an ecc file on your encrypted data files, thus\n*after* encryption, and keep the ecc file in clear text (never encrypt\nnor compress it). This is not a security risk at all since the ecc\nfile does not give any information on the content inside your\nencrypted files, but rather just redundant info to correct corrupted\nbytes (however if you generate the ecc file on the data files before\nencryption, then it's clearly a security risk, and someone could\nrecover your data without your permission).\n\n- What medium should I use to store my data?\n\nThe details are long and a bit complicated (I may write a complete article\nabout it in the future), but the tl;dr answer is that you should use *optical disks*,\nbecause it decouples the storage medium and the reading hardware\n(eg, at the opposite we have hard drives, which contains both the reading\nhardware and the storage medium, so if one fails, you lose both)\nand because it's most likely future-proof (you only need a laser, which\nis universal, the laser's parameters can always be tweaked).\n\nFrom scientific studies, it seems that, at the time of writing this (2015),\nBluRay HTL disks are the most resilient against environmental degradation.\nTo raise the duration, you can also put optical disks in completely opaque boxes\n(to avoid light degradation) and in addition you can put any storage medium\n(not only optical disks, but also hard drives and anything really) in\n*completely* air-tight and water-tight bags or box and put in a fridge or a freezer.\nThis is a law of nature: lower the temperature, lower will be the entropy, in other\nwords lower will be the degradation over time. It works the same with digital data.\n\n- What file formats are the most recoverable?\n\nIt's difficult to advise a specific format. What we can do is advise the characteristics\nof a good file format:\n\n * future-proof (should be readable in the future).\n * non-solid (ie, divised into indepedent blocks, so that a corruption to one block doesn't cause a problem to the decoding of other blocks).\n * open source implementation available.\n * minimize corruption impact (ie, how much of the file becomes unreadable with a partial corruption? Only the partially corrupted area, or other valid parts too?).\n * No magic bytes or header importance (ie, corrupting the header won't prevent opening the file).\n\nThere are a few studies about the most resilient file formats, such as:\n\n * `\"Just one bit in a million: On the effects of data corruption in files\" by Volker Heydegger `_.\n * `\"Analysing the impact of file formats on data integrity\" by Volker Heydegger `_.\n * `\"A guide to formats\", by The UK national archives `_ (you want to look at the Recoverability entry in each table).\n\n- If you have any question about Reed-Solomon codes, the best place to\n ask is probably here (with the incredible Dilip Sarwate):\n http://www.dsprelated.com/groups/comp.dsp/1.php?searchfor=reed%20solomon\n\n.. |Example| image:: https://raw.githubusercontent.com/lrq3000/pyFileFixity/master/tux-example.jpg\n :scale: 60 %\n :alt: Image corruption and repair example\n.. |Build Status| image:: https://travis-ci.org/lrq3000/pyFileFixity.svg?branch=master\n :target: https://travis-ci.org/lrq3000/pyFileFixity\n.. |Coverage Status| image:: https://codecov.io/github/lrq3000/pyFileFixity/coverage.svg?branch=master\n :target: https://codecov.io/github/lrq3000/pyFileFixity?branch=master\n.. |PyPi Status| image:: https://img.shields.io/pypi/v/pyFileFixity.svg\n :target: https://pypi.python.org/pypi/pyFileFixity\n.. |PyPi Downloads| image:: https://img.shields.io/pypi/dm/pyFileFixity.svg\n :target: https://pypi.python.org/pypi/pyFileFixity",
"description_content_type": null,
"docs_url": null,
"download_url": "UNKNOWN",
"downloads": {
"last_day": -1,
"last_month": -1,
"last_week": -1
},
"home_page": "https://github.com/lrq3000/pyFileFixity",
"keywords": "file repair monitor change reed-solomon error correction",
"license": "MIT License",
"maintainer": null,
"maintainer_email": null,
"name": "pyFileFixity",
"package_url": "https://pypi.org/project/pyFileFixity/",
"platform": "any",
"project_url": "https://pypi.org/project/pyFileFixity/",
"project_urls": {
"Download": "UNKNOWN",
"Homepage": "https://github.com/lrq3000/pyFileFixity"
},
"release_url": "https://pypi.org/project/pyFileFixity/2.3.1/",
"requires_dist": null,
"requires_python": null,
"summary": "Helping file fixity (long term storage of data) via redundant error correcting codes and hash auditing.",
"version": "2.3.1"
},
"last_serial": 2081098,
"releases": {
"2.0.0b1": [
{
"comment_text": "",
"digests": {
"md5": "272ffb5ffe44239b8a1a7384907b2eae",
"sha256": "e1fb76095ebe0b31b5763211214c84dd01d2ed8836d5809837f65b4657d6251e"
},
"downloads": -1,
"filename": "pyFileFixity-2.0.0b1-cp27-none-win32.whl",
"has_sig": false,
"md5_digest": "272ffb5ffe44239b8a1a7384907b2eae",
"packagetype": "bdist_wheel",
"python_version": "cp27",
"requires_python": null,
"size": 325779,
"upload_time": "2015-10-19T13:50:19",
"url": "https://files.pythonhosted.org/packages/3f/0e/d2539b9e8c6c0457f83d4a03ee73382d8ad41c1061c32972a9166b1ccf0f/pyFileFixity-2.0.0b1-cp27-none-win32.whl"
},
{
"comment_text": "",
"digests": {
"md5": "29145685dd84b97efafb38f1fd902798",
"sha256": "fded74dd6dbbe66908f96fb5d881448047d1abfc2c59fc2fdeb151ab20b67250"
},
"downloads": -1,
"filename": "pyFileFixity-2.0.0b1.tar.gz",
"has_sig": false,
"md5_digest": "29145685dd84b97efafb38f1fd902798",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 855616,
"upload_time": "2015-10-19T13:50:40",
"url": "https://files.pythonhosted.org/packages/86/20/d3b570140285f8cbde4f39ae8be76238ef1b67aa905c73f278ead17e4d43/pyFileFixity-2.0.0b1.tar.gz"
},
{
"comment_text": "",
"digests": {
"md5": "3279eb42fe4640f898559383eaab587f",
"sha256": "fb40c04a2267f21e14112da7618511b81cd94d40cd48b8afea3fab424f631fc5"
},
"downloads": -1,
"filename": "pyFileFixity-2.0.0b1.win32-py2.7.exe",
"has_sig": false,
"md5_digest": "3279eb42fe4640f898559383eaab587f",
"packagetype": "bdist_wininst",
"python_version": "2.7",
"requires_python": null,
"size": 558193,
"upload_time": "2015-10-19T13:50:58",
"url": "https://files.pythonhosted.org/packages/d8/e6/aab6085ac90ef762a05111dd3b420d8c583793293a48472a497c99f42149/pyFileFixity-2.0.0b1.win32-py2.7.exe"
},
{
"comment_text": "",
"digests": {
"md5": "682a11c9dabaa55bc19d725cbd6e1466",
"sha256": "8208edcf3304ed55f8f0a7d9bd8366092c3f1ab47339be125babc55349e1b322"
},
"downloads": -1,
"filename": "pyFileFixity-2.0.0b1.zip",
"has_sig": false,
"md5_digest": "682a11c9dabaa55bc19d725cbd6e1466",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 959139,
"upload_time": "2015-10-19T13:51:26",
"url": "https://files.pythonhosted.org/packages/9b/af/f4e42f5c51ffd6b9843d98a1af61be1d0a5c5b9e526c96e86e3d489270fe/pyFileFixity-2.0.0b1.zip"
}
],
"2.0.0b2": [
{
"comment_text": "",
"digests": {
"md5": "c1056355cd98eddcc1c5fad05d52a765",
"sha256": "ae3e6d08bae428fdf6acdeed05d2adaa850a04595e2313ed72f464c6d5f7d406"
},
"downloads": -1,
"filename": "pyFileFixity-2.0.0b2-cp27-none-win32.whl",
"has_sig": false,
"md5_digest": "c1056355cd98eddcc1c5fad05d52a765",
"packagetype": "bdist_wheel",
"python_version": "cp27",
"requires_python": null,
"size": 337070,
"upload_time": "2015-11-16T04:37:03",
"url": "https://files.pythonhosted.org/packages/29/ab/f37f95fa01b52741e64181dbf6e49eef3e6cf1e1f7d31758fc00f5b7bd98/pyFileFixity-2.0.0b2-cp27-none-win32.whl"
},
{
"comment_text": "",
"digests": {
"md5": "484fca080c7bce240b3e6a8c9514a25f",
"sha256": "289bae41eb5fd1d3603645c65fd16b0d72e8241ab590f55c8084657c5820b43d"
},
"downloads": -1,
"filename": "pyFileFixity-2.0.0b2.tar.gz",
"has_sig": false,
"md5_digest": "484fca080c7bce240b3e6a8c9514a25f",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 887715,
"upload_time": "2015-11-16T04:37:34",
"url": "https://files.pythonhosted.org/packages/c6/fc/a0c5656d0fa30f6b90fcb1ed8df448e6d4b0d2f13bb2b986a6c5020faed8/pyFileFixity-2.0.0b2.tar.gz"
},
{
"comment_text": "",
"digests": {
"md5": "442be9826a9333e2314fd03919c14d07",
"sha256": "e76fdbe26aca75ffd74b8a7d82403273f21a544ee3b7343e73fc04622a8cabb1"
},
"downloads": -1,
"filename": "pyFileFixity-2.0.0b2.win32-py2.7.exe",
"has_sig": false,
"md5_digest": "442be9826a9333e2314fd03919c14d07",
"packagetype": "bdist_wininst",
"python_version": "2.7",
"requires_python": null,
"size": 571135,
"upload_time": "2015-11-16T04:37:52",
"url": "https://files.pythonhosted.org/packages/ad/67/26f33c056e36e6f0f5b90620964f9eb0fb857df6d5787ace18044995d5cb/pyFileFixity-2.0.0b2.win32-py2.7.exe"
},
{
"comment_text": "",
"digests": {
"md5": "f015c3287296830404cd520c556395c3",
"sha256": "a4a2c1187f9cc2b7fce4bd4e9f7360101d9a66e68dd184f64f6957f1688ac85f"
},
"downloads": -1,
"filename": "pyFileFixity-2.0.0b2.zip",
"has_sig": false,
"md5_digest": "f015c3287296830404cd520c556395c3",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 993946,
"upload_time": "2015-11-16T04:38:09",
"url": "https://files.pythonhosted.org/packages/a3/83/2a2ec5708fae78d40eedfba9d7d876f943994db0bd5676297647bb971c6d/pyFileFixity-2.0.0b2.zip"
}
],
"2.0.1": [
{
"comment_text": "",
"digests": {
"md5": "0e5e0a587c12facaa9f64eba45d4b928",
"sha256": "5b0b408babb9a04d52444dcb55866699289edbadf1f8160d441756d96a75730a"
},
"downloads": -1,
"filename": "pyFileFixity-2.0.1-cp27-none-win32.whl",
"has_sig": false,
"md5_digest": "0e5e0a587c12facaa9f64eba45d4b928",
"packagetype": "bdist_wheel",
"python_version": "cp27",
"requires_python": null,
"size": 345940,
"upload_time": "2015-12-06T22:27:47",
"url": "https://files.pythonhosted.org/packages/bf/01/712b0f26559d656f5bdbe3f4c289e696bc454bdcdaaf1ba1b2e8a41d6560/pyFileFixity-2.0.1-cp27-none-win32.whl"
},
{
"comment_text": "",
"digests": {
"md5": "f92119d43212afcf1cd8d81b2f3c5905",
"sha256": "dd8f005ab38eab8797a6dc8dae435ac231081309cf4bdaeadb9e39595a0faf4b"
},
"downloads": -1,
"filename": "pyFileFixity-2.0.1.tar.gz",
"has_sig": false,
"md5_digest": "f92119d43212afcf1cd8d81b2f3c5905",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 941868,
"upload_time": "2015-12-06T22:28:05",
"url": "https://files.pythonhosted.org/packages/79/b2/ee6cbe41f760ed908dbc9eba13d6b825fe200cd561d0d294de1c52a03cff/pyFileFixity-2.0.1.tar.gz"
},
{
"comment_text": "",
"digests": {
"md5": "b37f1fccb08fcc0aefb9070702957ab2",
"sha256": "0c1b5821437fc1975b414a6953cebed8e53e50b41e82e0140cdc05ecdadd864f"
},
"downloads": -1,
"filename": "pyFileFixity-2.0.1.win32-py2.7.exe",
"has_sig": false,
"md5_digest": "b37f1fccb08fcc0aefb9070702957ab2",
"packagetype": "bdist_wininst",
"python_version": "2.7",
"requires_python": null,
"size": 580874,
"upload_time": "2015-12-06T22:28:21",
"url": "https://files.pythonhosted.org/packages/f5/e6/0cf3f77b39c7be5f0b85d592a1e9e7aea0ca9551d7dfcceb0240c1054794/pyFileFixity-2.0.1.win32-py2.7.exe"
},
{
"comment_text": "",
"digests": {
"md5": "6e59c1b289ceaa9f024d1a6a2d5d752a",
"sha256": "e3fc73f5b0dbef3ce994f1014f664162ad1c50180746ab8d1d9135091f32f9c7"
},
"downloads": -1,
"filename": "pyFileFixity-2.0.1.zip",
"has_sig": false,
"md5_digest": "6e59c1b289ceaa9f024d1a6a2d5d752a",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 1067698,
"upload_time": "2015-12-06T22:28:38",
"url": "https://files.pythonhosted.org/packages/8f/3f/cc9a80251af8f7c9f10b64872020f392724608e3444feba94dc33ffc5094/pyFileFixity-2.0.1.zip"
}
],
"2.0.3": [
{
"comment_text": "",
"digests": {
"md5": "9f048ecf6fe4bae495f81b474885863c",
"sha256": "b544c970e8af4926095720c14069049513e23d10c5287a9b48bad21a81cdbceb"
},
"downloads": -1,
"filename": "pyFileFixity-2.0.3-cp27-none-win32.whl",
"has_sig": false,
"md5_digest": "9f048ecf6fe4bae495f81b474885863c",
"packagetype": "bdist_wheel",
"python_version": "cp27",
"requires_python": null,
"size": 345951,
"upload_time": "2015-12-06T23:13:03",
"url": "https://files.pythonhosted.org/packages/c6/60/0e136bc24e6871a4dfcfbfe4231f3691edcb6b4cc627466fb8a045694b22/pyFileFixity-2.0.3-cp27-none-win32.whl"
},
{
"comment_text": "",
"digests": {
"md5": "a2d0bfb242c3929be1325cab81d39587",
"sha256": "07b2939f637587cc29988a4c0c41e2564af5795ca877e2f89a340603e5058983"
},
"downloads": -1,
"filename": "pyFileFixity-2.0.3.tar.gz",
"has_sig": false,
"md5_digest": "a2d0bfb242c3929be1325cab81d39587",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 1174900,
"upload_time": "2015-12-06T23:13:20",
"url": "https://files.pythonhosted.org/packages/d3/51/b8d19f3421537dd73f1718882930881a2277963cadc6f388a16694207fd1/pyFileFixity-2.0.3.tar.gz"
},
{
"comment_text": "",
"digests": {
"md5": "30515ea1de928b600c559c2fdab340e1",
"sha256": "4088ab7092fc9e8612576c78c6cf08a8c5beb5fcd368b33888205d34bfeeccf8"
},
"downloads": -1,
"filename": "pyFileFixity-2.0.3.win32-py2.7.exe",
"has_sig": false,
"md5_digest": "30515ea1de928b600c559c2fdab340e1",
"packagetype": "bdist_wininst",
"python_version": "2.7",
"requires_python": null,
"size": 581108,
"upload_time": "2015-12-06T23:13:36",
"url": "https://files.pythonhosted.org/packages/bf/e6/736363df9bb15013c34765928165932d0bf4ca5a9cf98b29c6c9680e5b65/pyFileFixity-2.0.3.win32-py2.7.exe"
},
{
"comment_text": "",
"digests": {
"md5": "f9ccf92af4b916828ec9a944aba62c30",
"sha256": "b417cab7ba434dd0a593c655fdfeac8c471bdc1b06003ab09411815685c2e927"
},
"downloads": -1,
"filename": "pyFileFixity-2.0.3.zip",
"has_sig": false,
"md5_digest": "f9ccf92af4b916828ec9a944aba62c30",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 1329902,
"upload_time": "2015-12-06T23:13:55",
"url": "https://files.pythonhosted.org/packages/c8/c9/5a71885c8dea908f85b0ec86a4af6b9d501f96ca6f23d72f7a11a3d89242/pyFileFixity-2.0.3.zip"
}
],
"2.3.0": [
{
"comment_text": "",
"digests": {
"md5": "efa62cd5375e6f883ccff324144d06fc",
"sha256": "44b4a228c1c1037034564a2298f85f4242c8bfb213159090f4be563a9fd4a848"
},
"downloads": -1,
"filename": "pyFileFixity-2.3.0-cp27-none-win32.whl",
"has_sig": false,
"md5_digest": "efa62cd5375e6f883ccff324144d06fc",
"packagetype": "bdist_wheel",
"python_version": "cp27",
"requires_python": null,
"size": 1508145,
"upload_time": "2015-12-07T22:58:47",
"url": "https://files.pythonhosted.org/packages/9c/bd/750ec609067d4199e32c71eb66e9c38c0262b1de1a84e5b0a03746bfb012/pyFileFixity-2.3.0-cp27-none-win32.whl"
},
{
"comment_text": "",
"digests": {
"md5": "d9b900c6c80cce10b88d8953fa855f7d",
"sha256": "184769ac1e9fbf4a653770a37e7b793b24ef77115c314cde6071ba34804bc3fa"
},
"downloads": -1,
"filename": "pyFileFixity-2.3.0.tar.gz",
"has_sig": false,
"md5_digest": "d9b900c6c80cce10b88d8953fa855f7d",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 1180454,
"upload_time": "2015-12-07T22:59:06",
"url": "https://files.pythonhosted.org/packages/be/a6/8b486aa9a998e430cbc3a12d473b1c227d272d8ea2d282c934018492e0d7/pyFileFixity-2.3.0.tar.gz"
},
{
"comment_text": "",
"digests": {
"md5": "186cae8546d07220d9e04d0b5fdf33d8",
"sha256": "9d0a09b4a80271cd40b7a5b0afbf54612301872cd45f470b7e3f9a06ef0cafcd"
},
"downloads": -1,
"filename": "pyFileFixity-2.3.0.win32-py2.7.exe",
"has_sig": false,
"md5_digest": "186cae8546d07220d9e04d0b5fdf33d8",
"packagetype": "bdist_wininst",
"python_version": "2.7",
"requires_python": null,
"size": 1737065,
"upload_time": "2015-12-07T22:59:28",
"url": "https://files.pythonhosted.org/packages/a1/14/90d752f95564f77ed3ce9f35d9cab24dab115d54ea6b9ee0374114144be5/pyFileFixity-2.3.0.win32-py2.7.exe"
},
{
"comment_text": "",
"digests": {
"md5": "aaf7cfeeabd1a76a599aac3eff78d67f",
"sha256": "3c2f06a479a953b5f79069bd6edd5daa3fcc4685365df1f458a4ecb14ff207df"
},
"downloads": -1,
"filename": "pyFileFixity-2.3.0.zip",
"has_sig": false,
"md5_digest": "aaf7cfeeabd1a76a599aac3eff78d67f",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 1344161,
"upload_time": "2015-12-07T22:59:50",
"url": "https://files.pythonhosted.org/packages/f9/f8/df8c0bb2dc3e68eab003733b39f6d43d81caa47205bcad0e2af07be790b8/pyFileFixity-2.3.0.zip"
}
],
"2.3.1": [
{
"comment_text": "",
"digests": {
"md5": "6f4ffd6ed2423fe8005fb2e60dfcbd2c",
"sha256": "ed4539f3b7530675154d6e5cd21b93644779f652f3ff1793cc367bf0e0ae6b8d"
},
"downloads": -1,
"filename": "pyFileFixity-2.3.1-cp27-none-win32.whl",
"has_sig": false,
"md5_digest": "6f4ffd6ed2423fe8005fb2e60dfcbd2c",
"packagetype": "bdist_wheel",
"python_version": "cp27",
"requires_python": null,
"size": 1508154,
"upload_time": "2015-12-07T23:37:24",
"url": "https://files.pythonhosted.org/packages/1c/ee/9c38d55528571ace26fd6df5bcd127fa0905e1eed38ae8378dfbc33d9f5f/pyFileFixity-2.3.1-cp27-none-win32.whl"
},
{
"comment_text": "",
"digests": {
"md5": "935d20acdbb8e694edca81dc93ef0102",
"sha256": "bb262f33f08b7dd0ef1d51e9806d9c3a5996bb351b8c9ab5052c166a781fd3b3"
},
"downloads": -1,
"filename": "pyFileFixity-2.3.1.tar.gz",
"has_sig": false,
"md5_digest": "935d20acdbb8e694edca81dc93ef0102",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 1180397,
"upload_time": "2015-12-07T23:37:42",
"url": "https://files.pythonhosted.org/packages/60/59/b1fd3a8e965d481f05d3fd08f050878a9dfcc783729ee4d622d6f287b9c5/pyFileFixity-2.3.1.tar.gz"
},
{
"comment_text": "",
"digests": {
"md5": "f372f117cb47671151ea8bf9e9120df5",
"sha256": "7c96dabff1c6ca0d983a1831446cad5b68b8e205959993e24049cca6eeb65287"
},
"downloads": -1,
"filename": "pyFileFixity-2.3.1.win32-py2.7.exe",
"has_sig": false,
"md5_digest": "f372f117cb47671151ea8bf9e9120df5",
"packagetype": "bdist_wininst",
"python_version": "2.7",
"requires_python": null,
"size": 1737068,
"upload_time": "2015-12-07T23:38:04",
"url": "https://files.pythonhosted.org/packages/a0/7b/dc0bf9e6452ddf667e48eda98313e2dbe6afa628d9487b5bf400efaf40d2/pyFileFixity-2.3.1.win32-py2.7.exe"
},
{
"comment_text": "",
"digests": {
"md5": "c80c00cf70fce3034b3c482b0bf05178",
"sha256": "9bf0cadc6446a49efdedcfb2d2f3c162865a138cf914f43c86c8563a46a9d0f2"
},
"downloads": -1,
"filename": "pyFileFixity-2.3.1.zip",
"has_sig": false,
"md5_digest": "c80c00cf70fce3034b3c482b0bf05178",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 1344160,
"upload_time": "2015-12-07T23:38:26",
"url": "https://files.pythonhosted.org/packages/31/43/f4d96170d4b4e77b32ba130cd2b51211ae957b3ca8ce2c9a055e4ca180af/pyFileFixity-2.3.1.zip"
}
]
},
"urls": [
{
"comment_text": "",
"digests": {
"md5": "6f4ffd6ed2423fe8005fb2e60dfcbd2c",
"sha256": "ed4539f3b7530675154d6e5cd21b93644779f652f3ff1793cc367bf0e0ae6b8d"
},
"downloads": -1,
"filename": "pyFileFixity-2.3.1-cp27-none-win32.whl",
"has_sig": false,
"md5_digest": "6f4ffd6ed2423fe8005fb2e60dfcbd2c",
"packagetype": "bdist_wheel",
"python_version": "cp27",
"requires_python": null,
"size": 1508154,
"upload_time": "2015-12-07T23:37:24",
"url": "https://files.pythonhosted.org/packages/1c/ee/9c38d55528571ace26fd6df5bcd127fa0905e1eed38ae8378dfbc33d9f5f/pyFileFixity-2.3.1-cp27-none-win32.whl"
},
{
"comment_text": "",
"digests": {
"md5": "935d20acdbb8e694edca81dc93ef0102",
"sha256": "bb262f33f08b7dd0ef1d51e9806d9c3a5996bb351b8c9ab5052c166a781fd3b3"
},
"downloads": -1,
"filename": "pyFileFixity-2.3.1.tar.gz",
"has_sig": false,
"md5_digest": "935d20acdbb8e694edca81dc93ef0102",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 1180397,
"upload_time": "2015-12-07T23:37:42",
"url": "https://files.pythonhosted.org/packages/60/59/b1fd3a8e965d481f05d3fd08f050878a9dfcc783729ee4d622d6f287b9c5/pyFileFixity-2.3.1.tar.gz"
},
{
"comment_text": "",
"digests": {
"md5": "f372f117cb47671151ea8bf9e9120df5",
"sha256": "7c96dabff1c6ca0d983a1831446cad5b68b8e205959993e24049cca6eeb65287"
},
"downloads": -1,
"filename": "pyFileFixity-2.3.1.win32-py2.7.exe",
"has_sig": false,
"md5_digest": "f372f117cb47671151ea8bf9e9120df5",
"packagetype": "bdist_wininst",
"python_version": "2.7",
"requires_python": null,
"size": 1737068,
"upload_time": "2015-12-07T23:38:04",
"url": "https://files.pythonhosted.org/packages/a0/7b/dc0bf9e6452ddf667e48eda98313e2dbe6afa628d9487b5bf400efaf40d2/pyFileFixity-2.3.1.win32-py2.7.exe"
},
{
"comment_text": "",
"digests": {
"md5": "c80c00cf70fce3034b3c482b0bf05178",
"sha256": "9bf0cadc6446a49efdedcfb2d2f3c162865a138cf914f43c86c8563a46a9d0f2"
},
"downloads": -1,
"filename": "pyFileFixity-2.3.1.zip",
"has_sig": false,
"md5_digest": "c80c00cf70fce3034b3c482b0bf05178",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 1344160,
"upload_time": "2015-12-07T23:38:26",
"url": "https://files.pythonhosted.org/packages/31/43/f4d96170d4b4e77b32ba130cd2b51211ae957b3ca8ce2c9a055e4ca180af/pyFileFixity-2.3.1.zip"
}
]
}