{ "info": { "author": "Rob Speer", "author_email": "rspeer@mit.edu", "bugtrack_url": null, "classifiers": [], "description": "reductio (ad bigdatum)\n======================\nBecause mapping and reducing isn't supposed to be hard.\n\nWhat is reductio?\n-----------------\nReductio is a minimalistic map-reduce framework for Python. It runs on top of\nFabric and setuptools, which you might already use to get your code onto other\nmachines.\n\nIt has no database. It has no distributed filesystem. It uses no server other\nthan sshd. Because of this, it has essentially no memory requirement!\n\nReductio is designed for disk-bound big data tasks, which many of them are. If\nthe pieces you need to map-reduce fit entirely in the RAM of your worker\ncomputers, you are paying a huge premium for that. And if they don't, a system\nthat tries to buffer things in RAM is going to be wasting all its effort. At\nsome point, you won't see your data again unless you write it to the damn disk,\nand that's what is going to take most of the time.\n\nI created reductio out of absolute necessity, so I could start crunching my\ndata. You might notice its documentation is practically nonexistent at the\nmoment. \nWhat Reductio does\n------------------\nReductio extends Fabric (http://fabfile.org). It is meant to support the\nfollowing approximate process:\n\n- Set up the appropriate Python environment on all your worker machines,\n including required packages, a place to keep the data, and an up-to-date\n version of the code you want to run.\n- Tell each worker machine how to contact all the other machines, so they can\n send their data onward (\"scatter\") when they're done with it. \n- Give each worker machine some Python functions to run over all the data.\n These will often be \"maps\" or \"reduces\".\n- Collect results and group the things that belong together using Unix's\n extremely optimized `sort` command.\n\nExample\n-------\n`reductio/example/fabfile.py` is in example of how to count all the letter\nbigrams (pairs of adjacent letters) in the ubiquitous \"words\" file, and\naggregate them into a table of frequencies.\n\nRunning tasks\n-------------\nIf you have a task defined as the function `do_stuff` in `mymodule.py`, but\nfirst you want to run `setup` to make sure code and other things are up to\ndate, you would run them both with this command:\n\n fab -f mymodule setup do_stuff\n\n\nWhy not Hadoop?\n---------------\nFace it, if you knew how to configure Hadoop and get your code to run with it,\nyou'd be doing so right now.\n\nAlso, Hadoop is written for Java programmers, and Python is distinctly a\nsecond-class citizen in its world. Hadoop seems to think all Python code\ntakes the form of standalone scripts with no dependencies, which perhaps\nsays something about what Java programmers think of Python.\n\nReductio recognizes that none of this is going to work unless you have the\nright Python setup, so it builds on tools that Python programmers already use\nto deploy their code.\n\nWhy not Disco?\n--------------\nI approve of the Disco project (http://discoproject.org) and its goal of\ncreating a map-reduce ecosystem designed around Python, but I find it too\ncomplex and too \"magical\" at the moment.\n\nIt makes it difficult to understand what is going on in its internals, and yet\nyou have to understand its internals when something goes wrong or when you want\nto do something the designers didn't expect.", "description_content_type": null, "docs_url": null, "download_url": "UNKNOWN", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "http://github.com/rspeer/reductio", "keywords": "mapreduce distributed fabric", "license": "AGPLv3", "maintainer": null, "maintainer_email": null, "name": "reductio", "package_url": "https://pypi.org/project/reductio/", "platform": "UNKNOWN", "project_url": "https://pypi.org/project/reductio/", "project_urls": { "Download": "UNKNOWN", "Homepage": "http://github.com/rspeer/reductio" }, "release_url": "https://pypi.org/project/reductio/0.2.1/", "requires_dist": null, "requires_python": null, "summary": "Map/Reduce for Fabric", "version": "0.2.1" }, "last_serial": 798738, "releases": { "0.1": [ { "comment_text": "", "digests": { "md5": "f64825b14de0f649a8d72542f011801d", "sha256": "fbdf5fdc460eacd3b758c8d81b6afeb7337548cca33ec5bbf8542499df14f0c2" }, "downloads": -1, "filename": "reductio-0.1.tar.gz", "has_sig": false, "md5_digest": "f64825b14de0f649a8d72542f011801d", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 6591, "upload_time": "2011-11-23T10:07:17", "url": "https://files.pythonhosted.org/packages/61/7f/ecd3446fe940fba189bea8cade04b3f07d811c89785ed6b2e7e6b16bb81c/reductio-0.1.tar.gz" } ], "0.1.1": [ { "comment_text": "", "digests": { "md5": "2073b7fbb3c304dabf145a19c30d5b0f", "sha256": "0cfb838429c7fc2fc0bfb2be7bbb3091ce32a944363794876945fb9e469bf033" }, "downloads": -1, "filename": "reductio-0.1.1.tar.gz", "has_sig": false, "md5_digest": "2073b7fbb3c304dabf145a19c30d5b0f", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 6679, "upload_time": "2011-11-23T22:06:58", "url": "https://files.pythonhosted.org/packages/a3/a0/52fea4b04bff851217fc993d3b76d54e8432dd626dccef861e2c63bbc0e5/reductio-0.1.1.tar.gz" } ], "0.1.2": [ { "comment_text": "", "digests": { "md5": "e782eb572104b6c6d4806384c3e87f7d", "sha256": "48eeca53d749f2900780104c27eb2a5a2df0f4d0a54fb172c67a170989b98ba6" }, "downloads": -1, "filename": "reductio-0.1.2.tar.gz", "has_sig": false, "md5_digest": "e782eb572104b6c6d4806384c3e87f7d", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 6943, "upload_time": "2011-11-24T00:34:16", "url": "https://files.pythonhosted.org/packages/4f/a3/41b45a177131483b68f36824c5d368fcd54d173879fe47e10da431068685/reductio-0.1.2.tar.gz" } ], "0.1.3": [ { "comment_text": "", "digests": { "md5": "afe78f19183d5b56af684a4d416e7693", "sha256": "c3da146b0d063ec97483b75ee8e389d4d6c747d522fc83c02efb39c2c4d03587" }, "downloads": -1, "filename": "reductio-0.1.3.tar.gz", "has_sig": false, "md5_digest": "afe78f19183d5b56af684a4d416e7693", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 6958, "upload_time": "2011-11-24T00:44:26", "url": "https://files.pythonhosted.org/packages/9a/53/1780016631a782384162eb2f2ef1f51b9abb72116c1f33270c16adaea578/reductio-0.1.3.tar.gz" } ], "0.1.4": [ { "comment_text": "", "digests": { "md5": "cf4d3c79f7bff6b41e1eb72455235e8f", "sha256": "31ea296c72c72adaaa8a306804ec4537c14b9ef675670c137117d4c0326517ed" }, "downloads": -1, "filename": "reductio-0.1.4.tar.gz", "has_sig": false, "md5_digest": "cf4d3c79f7bff6b41e1eb72455235e8f", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 6962, "upload_time": "2011-11-24T00:46:30", "url": "https://files.pythonhosted.org/packages/33/f2/9115a0ca56e062c8536e4d02104c55e2c0519ef07fb50d231cb041137c41/reductio-0.1.4.tar.gz" } ], "0.2.1": [ { "comment_text": "", "digests": { "md5": "8ffc3a36cf79907c0b2882fc943bfb18", "sha256": "1a6bdc8a6755992e5f0922b09ae21791faa999ef37acd845ff74072ab7c68f9a" }, "downloads": -1, "filename": "reductio-0.2.1.tar.gz", "has_sig": false, "md5_digest": "8ffc3a36cf79907c0b2882fc943bfb18", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 9485, "upload_time": "2011-12-05T22:36:48", "url": "https://files.pythonhosted.org/packages/29/8b/9309c14e637469ddf0d147ca51af0ee7a7856993d16b485bb428866d8853/reductio-0.2.1.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "8ffc3a36cf79907c0b2882fc943bfb18", "sha256": "1a6bdc8a6755992e5f0922b09ae21791faa999ef37acd845ff74072ab7c68f9a" }, "downloads": -1, "filename": "reductio-0.2.1.tar.gz", "has_sig": false, "md5_digest": "8ffc3a36cf79907c0b2882fc943bfb18", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 9485, "upload_time": "2011-12-05T22:36:48", "url": "https://files.pythonhosted.org/packages/29/8b/9309c14e637469ddf0d147ca51af0ee7a7856993d16b485bb428866d8853/reductio-0.2.1.tar.gz" } ] }