{ "info": { "author": "UNKNOWN", "author_email": "UNKNOWN", "bugtrack_url": null, "classifiers": [ "Development Status :: 3 - Alpha", "Environment :: Console", "License :: OSI Approved :: MIT License", "Programming Language :: Python", "Topic :: Scientific/Engineering", "Topic :: Scientific/Engineering :: Bio-Informatics", "Topic :: Scientific/Engineering :: Information Analysis", "Topic :: Software Development :: Build Tools" ], "description": "==================\nAbout psicons.core\n==================\n\n.. |psicons| replace:: *psicons*\n\n\nBackground\n----------\n\nScientific analysis can be problematic:\n\n* It may involve multiple steps, each using the results of the previous stage. \n Making a mistake often means repeating the whole series for safety.\n\n* Sometimes analysis chains have to be repeated on different datasets.\n Sometimes, even within a single analysis, the same manipulation or test has\n to be repeated with slightly different parameters.\n\n* Even immediately after the fact, it's easy to forget what was done. 9 months\n later when responding to a referee's report, it may be impossible.\n\n* Collaborators, clients or bosses may demand accountability.\n\n* With long but routine tasks, it's easy to get bored and make mistakes.\n\nIt this light, |psicons| is a quick and dirty hack to subvert the scons build \nsystem for scientific analysis. Every stage of analysis is a command-line call \nto a script or executable that takes *inputs* and produces *outputs*. When \nscons is called, dependencies between outputs and inputs are tested and only \nthose stages are run that are necessary to update outputs. In addition, the \nexact sequence of analyses is recorded by the build file. \n\nIn summary, |psicons| provides:\n\n* Repeatability: running a build file, reruns the same analysis\n\n* Reproducibility: the build file (and custom scripts) document the steps of \n the analysis\n* Minimization of effort: if inputs or analysis steps are changed, only the \n necessary (dependent) steps of the analysis are rerun\n \n* Mistake-resistant: errors don't derail analysis due to reproducibility (\"what \n did I do\") and minimization of effort (only dependent steps are repeated)\n\n* Programmability: analysis tasks may be constructed with programatically\n (\"repeat the analysis across this parameter range\")\n\n\nStatus\n------\n\n*psicons* is very much a hack-and-see project, having been produced in the \naftermath of the 2009 H1N1 pandemic from the need for complex processing of \nlarge amounts of sequence data using ad-hoc scripts and formats. It worked \nwell in that limited role, but is still an early release exploring the \napproach. Functionality is limited and the API may change. There are other \nmore developed (and more specialised) alternatives. Comment is invited.\n\n\nInstallation\n------------\n\nThe simplest way to install *psicons* is via ``easy_install`` [setuptools]_ or \nan equivalent program::\n\n\t% easy_install psicons.core\n\nAlternatively the tarball can be downloaded, unpacked and ``setup.py`` run::\n\n\t% tar zxvf psicons-core.tgz\n\t% cd psicons-core\n\t% python set.py install\n\nIt should work with just about any version of Python.\n\n*psicons* requires that scons is installed, which is where things get tricky.\nScons by default installs itself in a sandboxed way with multiple versions\nliving side-by-side and thus being non-importable. Of course, |psicons| needs\nto use the scons library, so a conventional installation must be forced. \nDownload the scons tarball, unpack it and install it like thus::\n\n\t% python setup.py install --standard-lib\n\n\nUsing psicons\n-------------\n\nA full API is included in the source distribution.\n\n\nExamples\n~~~~~~~~\n\nPsicons works just like scons. In fact, it is scons. More details are available \nelsewhere but briefly, you run scons like this::\n\n\t# look for a build file called \"Sconstruct\" by default\n\t% scons \n\t# looks for a named build file\n\t% scons -f mybuildfile\n\nThis causes scons to execute the build file, which is just a Python script, \ndefining a series of tasks or *commands*::\n\n\t# an scons build file\n\t# some necessary administration - set up the build environment\n\tenv = Environment()\n\t\n\t# compile two libraries and then combine into one program\n\tfirst_libs = env.Object ('hello.c', CCFLAGS='-DHELLO')\n\tsecond_libs = env.Object ('goodbye.c', CCFLAGS='-DGOODBYE')\n\tenv.Program (first_libs + second_libs)\n\nThe first time this file is executed, the first two commands build libraries, \nwhile the third combines the libraries into a single executable. Dependencies \nbetween the steps are automatically tracked: should one of the original source \nfiles be changed (e.g. hello.c), when the file is rerun only the steps \n\"downstream\" of it (e.g. recompilation of the first library, and the final \nlinking) are rerun. \n\nScons has a large number of commands for all sorts of software builds. Psicons \nadds two new commands, so that local scripts or external programs can be used \nto in a build. In this way, complex multi-step analyses can be constructed from \na series of interdependent commands, that \"build\" intermediate data and final \nresults::\n\n\tfrom psicons.core import *\n\tenv = Environment()\n\t\n\t# call a local script\n\tIN_DATA = 'jg_08-10_2010.csv'\n\tCLEAN_DATA = 'jg_08-10_2010-cleaned.csv'\n\tmake_clean_data = Script (env, 'clean_seqs.py',\n\t\targs = ['--save-as', CLEAN_DATA],\n\t\tinfiles = [IN_DATA],\n\t\toutput = CLEAN_DATA,\n\t)\n\t\n\t# call an external command\n\tEPI_DATA = 'jg-types.txt'\n\tRESULT_DATA = 'results.tab'\n\ttype_data = External (env, 'treemaker',\n\t\targs = ['--save-as', RESULT_DATA],\n\t\tinfiles = [CLEAN_DATA, EPI_DATA],\n\t\toutput = [RESULT_DATA],\n\t)\n\nThe interfaces of these two commands are similar:\n\n* what is being called?\n* what inputs does it use (depend on)?\n* what outputs does it produce?\n\nWhen scons is run on this build file, it calls the script 'clean_seqs.py' on \n`IN_DATA` to produce `CLEAN_DATA`. Then the external program `treemaker` is \ncalled on `CLEAN_DATA` and `EPI_DATA` to produce `RESULT_DATA`. Should \n`EPI_DATA` be edited, when scons is called again, only the second external \nstep will be run again as the first step and it's results is still up to date. \nThus:\n\n* Analyses may be run (and rerun) easily\n\n* If data changes (or scripts change - bug fixes), only the necessary steps are \n rerun\n \n* The actions taken are recorded in the build file\n\nTo ease renaming intermediate or output files in a rational way, *psicons* \noffers a few utility functions for interpolating file names from parameters. To \nillustrate::\n\n\t# generate a new string from a template\n\t>>> d = {'foo': '123', 'bar': '456'}\n\t>>> interpolate ('ab{foo}cd{bar}ef', d)\n\t'ab123cd456ef'\n\t# name new file name from old by adding suffix to name \n\t>>> interpolate_from_path ('mydata.csv', '{stem}-cleaned{ext}')\n\t'mydata-cleaned.csv'\n\n\nLimitations\n-----------\n\nCertainly, far, far more complicated reproducibility tools are out there (see `\nhere `__) but \nmany are based around certain disciplines (e.g. geophysics, computational math), \nrequire working through web interfaces or using very standard sets of analysis \ntools. *psicons* is written from the point of view of a bioinformatician doing \nsequence and phylogenetic analysis, working on the commandline using a lot of \ncustom scripts and an endlessly changing lineup of supplied tools. As sometimes \nhappens, other tools didn't fit, so I wrote one that did. \n\nAs with many quick hack tools, documentation is currently a bit thin.\n\nThe need for a modified scons installation is a blemish. Future versions of\n|psicons| may need to directly incorporate scons for ease of installation.\n\nClearly, a set of standard tools for extracting, transforming and plotting \ndata would be a powerful addition to *psicons*. This doesn't exist as yet.\n\nThe process of installing a new tool or module for use by SCons is fiddly\n[scons-tools]_, involves copying libraries in a tool directory, registering \ntheir use in build files, voids the ease of using ``easy_install`` and makes \ndevelopment a pain. Thus, the additional \"commands\" are real scons commands, so\nmuch as functions that generate commands. But they are easy to use.\n\n\nCredit\n------\n\nThanks to the architects of Scons, of course.\n\nWhile this project was started before encountering Madagascar [madagascar]_, it \nhas inevitably shaped development. It's a remarkably powerful system, although \nill-suited to my current purposes. You should check it out.\n\nOnly when writing this document did I become aware of sconstools [sconstools]_,\nwhich seems to be following exactly the same direction as |psicons|.\n\n\nReferences\n----------\n\n.. [psicons-home] `psicons home page `__\n\n.. [psicons-pypi] `psicons on PyPi `__\n\n.. [setuptools] `Setuptools & easy_install `__\n\n.. [psicons-github] `psicons on github `__\n\n.. [scons] `Scons `__\n\n.. [madagascar] `Madagscar and Scons for reproducibility `__\n\n.. [scons-custom] `Where To Put Your Custom Builders and Tools `__\n\n.. [sconstools] `Sconstools `__\n\n\n=========\nChangelog\n=========\n\n0.2dev (29072011)\n-----------------\n\n- First widely released version\n- Added documentation\n\n\n0.1dev (unreleased)\n-------------------\n\n- Initial release", "description_content_type": null, "docs_url": null, "download_url": "UNKNOWN", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "http://svn.plone.org/svn/collective/", "keywords": "reproducibility documentable", "license": "GPL", "maintainer": null, "maintainer_email": null, "name": "psicons.core", "package_url": "https://pypi.org/project/psicons.core/", "platform": "UNKNOWN", "project_url": "https://pypi.org/project/psicons.core/", "project_urls": { "Download": "UNKNOWN", "Homepage": "http://svn.plone.org/svn/collective/" }, "release_url": "https://pypi.org/project/psicons.core/0.1/", "requires_dist": null, "requires_python": null, "summary": "A tool for documentable and reproducible analysis and research", "version": "0.1" }, "last_serial": 796758, "releases": { "0.1": [ { "comment_text": "", "digests": { "md5": "3a47f0adeb56d8b0c2a7932b3aaee7b2", "sha256": "2982cf5e93cea47c65d1022a9dda47217046c50a4cdda241a827d56554d38310" }, "downloads": -1, "filename": "psicons.core-0.1.tar.gz", "has_sig": false, "md5_digest": "3a47f0adeb56d8b0c2a7932b3aaee7b2", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 52226, "upload_time": "2011-08-01T19:07:17", "url": "https://files.pythonhosted.org/packages/94/1b/0ecfa7bce6024b51753518ea04f02ece0bb91d84bc9cb29ddb8d1c8847f7/psicons.core-0.1.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "3a47f0adeb56d8b0c2a7932b3aaee7b2", "sha256": "2982cf5e93cea47c65d1022a9dda47217046c50a4cdda241a827d56554d38310" }, "downloads": -1, "filename": "psicons.core-0.1.tar.gz", "has_sig": false, "md5_digest": "3a47f0adeb56d8b0c2a7932b3aaee7b2", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 52226, "upload_time": "2011-08-01T19:07:17", "url": "https://files.pythonhosted.org/packages/94/1b/0ecfa7bce6024b51753518ea04f02ece0bb91d84bc9cb29ddb8d1c8847f7/psicons.core-0.1.tar.gz" } ] }