{
    "info": {
        "author": "Jeff Hammel",
        "author_email": "jhammel@mozilla.com",
        "bugtrack_url": null,
        "classifiers": [],
        "description": "The Story of Toolbox\n====================\n\nToolbox is fundamentally a document-oriented approach to resource\nindexing.  A \"tool\" consists three mandatory string fields -- name,\ndescription, and URL -- that are generic to the large class of problems\nof web resources, as well as classifiers, such as author, usage, type,\netc. A tool may have an arbitrary number of classifier fields as\nneeded.  Each classifier consists of a set of values with which a tool\nis tagged. This gives toolbox the flexibility to fit a large number of\ndata models, such as PYPI, DOAP, and others.\n\n\nRunning Toolbox\n---------------\n\nYou can download and run the toolbox software yourself:\nhttp://github.com/k0s/toolbox\n\nTo serve in baseline mode, install the software and run::\n\n paster serve paste.ini\n\nThis will serve the handlers and static content using the paste\n(http://pythonpaste.org) webserver using ``README.txt`` as the\n``/about`` page and serving the data in ``sample``.\n\nThe dispatcher (``toolbox.dispatcher:Dispatcher``) is the central (WSGI)\nwebapp that designates per-request to a number of handlers (from\n``handlers.py``).  The dispatcher has a few options:\n\n* about: path to a restructured text file to serve at ``/about``\n* model_type: name of the backend to use (memory_cache, file_cache, or couch)\n* template_dir: extra directory to look for templates\n\nThese may be configured in the ``paste.ini`` file in the\n``[app:toolbox]`` section by prepending with the namespace\n``toolbox.``. It is advisable that you copy the example ``paste.ini``\nfile for your own usage needs.  Additional ``toolbox.``-namespaced\narguments will be passed to the model.  For instance, to specify the\ndirectory for the ``file_cache`` model, the provided ``paste.ini`` uses\n``toolbox.directory = %(here)s/sample``.\n\n\nArchitecture\n------------\n\nToolbox uses a fairly simple architecture with a single abstract data\nmodel allowing an arbitrary number of implementations to be constructed::\n\n Interfaces            Implementations\n\n +----+              +-+-----+\n |HTTP|              | |files|\n +----+---\\  +-----+ | +-----+\n           |-|model|-+-+-----+\n +------+-/  +-----+ | |couch|\n |script|            | +-----+\n +------+            +-+------+\n                     | |memory|\n                     | +------+\n                     +-+---+\n                       |...|\n                       +---+\n\nToolbox was originally intended to use a directory of files, one per project,\nas the backend. These were originally intended to be HTML files as the\nabove model may be clearly mapped as HTML::\n\n <div class=\"project\"><h1><a href=\"{{url}}\">{{name}}</a></h1>\n <p class=\"description\">{{description}}</p>\n {{for field in fields}}\n  <ul class=\"{{field}}\">\n  {{for value in values[field]}}\n   <li>{{value}}</li>\n  {{endfor}}\n {{endfor}}\n </div>\n\nThis microformat approach allows not only easy editing of the HTML\ndocuments, but the documents may be indepently served and displayed\nwithout the toolbox server-side. \n\nThe HTML microformat was never implemented (though, since the model\nbackend is pluggable, it easily could be). Instead, the original\nimplementation used JSON blobs stored in one file per tool. This\napproach loses the displayable aspect, though since JSON is a defined\nformat with several good tools for exploring and manipulating the data\nperhaps this disavantage is offset.\n\nA couch backend was also written.\n\n      +------------+-----------+------------+\n      |Displayable?|File-based?|Concurrency?|\n+-----+------------+-----------+------------+\n|HTML |Yes         |Yes        |No          |\n+-----+------------+-----------+------------+\n|JSON |Not really  |Yes        |No          |\n+-----+------------+-----------+------------+\n|Couch|No          |No         |Yes?        |\n+-----+------------+-----------+------------+\n\nThe concurrency issue with file-based documennt backends may be\novercome by using locked files.  Ideally, this is accomplished at the\nfilesystem level.  If your filesystem does not promote this\nfunctionality, it may be introduced programmatically.  A rough cartoon\nof a good implementation is as follows:\n\n1. A worker thread is spawned to write the data asynchronously. The\ndata is sent to the worker thread.\n\n2. The worker checks for the presence of a lockfile (herein further\ndetailed). If the lockfile exists and is owned by an active process,\nthe worker waits until said process is done with it. (For a more\nrobust implementation, the worker sends a request to write the file to\nsome controller.)\n\n3. The worker owns a lockfile based on its PID in some directory\nparallel to the directory root under consideration (for example,\n``/tmp/toolbox/lock/${PID}-${filename}.lck``).\n\n4. The worker writes to the file.\n\n5. The worker removes the lock\n\nThe toolbox web service uses a dispatcher->handler framework.  The\nhandlers are loosely pluggable (they are assigned in the dispatcher),\nbut could (and probably should) be made completely pluggable.  That\nsaid, the toolbox web system features an integration of templates,\nstatic resources (javascript, css, images), and handlers, so true\npluggability is further away than just supporting pluggable handlers\nin the dispatcher.\n\nDeployment, however, may be tailored as desired.  Any of the given\ntemplates may be overridden via passing a ``template_dir`` parameter\nwith a path to a directory that have templates of the appropriate\nnames as found in toolbox's ``templates`` directory. \n\nLikewise, the static files (css, js, etc.) are served using ``paste``'s \n``StaticURLParser`` out of toolbox's ``static`` directory. (See\ntoolbox's ``factory.py``.) Notably this is *not* done using the WSGI\napp itself.  Doing it with middleware allows the deployment to be\ncustomizable by writing your own factory.  For example, instead of\nusing the ``paste`` webserver and the included ``paste.ini``, you\ncould use nginx or apache and ``mod_wsgi`` with a factory file\ninvoking ``Dispatcher`` with the desired arguments and serving the\nstatic files with an arbitrary static file server.\n\nIt is common sense, if rarely followed, that deployment should be\nsimple.  If you want to get toolbox running on your desktop and/or for\ntesting, you should be able to do this easily (see the ``INSTALL.sh``\nfor a simple installation using ``bash``; you'll probably want to\nperform these steps by hand for any sort of real-world deployment).\nIf you want a highly customized deployment, then this will require\nmore expertise and manual setup.\n\nThe template data and the JSON are closely tied together.  This has the\ndistinct advantage of avoiding data translation steps and avoiding\ncode duplication.\n\nToolbox uses several light-footprint libraries:\n\n* webob for Request/Response handling: http://pythonpaste.org/webob/\n\n* tempita for (HTML) templates: http://pythonpaste.org/tempita/\n\n* whoosh for search.  This pure-python implementation of full-text\n  search is relatively fast (for python) and should scale decently to\n  the target scale of toolbox (1000s or 10000s of tools). While not as\n  fast as lucene, whoosh is easy to deploy and has a good API and\n  preserves toolbox as a deployable software product versus an\n  instance that requires the expert configuration, maintainence, and\n  tuning of several disparate software products that is both\n  non-automatable (cannot be installed with a script) and\n  time-consuming. http://packages.python.org/Whoosh/\n\n* jQuery: jQuery is the best JavaScript library and everyone\n  should use it. http://jquery.com/\n\n* jeditable for AJAXy editing: http://www.appelsiini.net/projects/jeditable\n\n* jquery-token for autocomplete: http://loopj.com/jquery-tokeninput/\n\n* less for dynamic stylesheets: http://lesscss.org/\n\n\nUser Interaction\n----------------\n\nA user will typically interact with Toolbox through the AJAX web\ninterface.  The server side returns relatively simple (HTML) markup,\nbut structured in such a way that JavaScript may be utilized to\npromote rich interaction.  The simple HTML + complex JS manifests\nseveral things:\n\n1. The document is a document. The tools HTML presented to the user (with\nthe current objectionable exception of the per-project Delete button)\nis a document form of the data. It can be clearly and easily\ntranslated to data (for e.g. import/export) or simply marked up using\n(e.g.) JS to add functionality. By keeping concerns seperate\n(presentation layer vs. interaction layer) a self-evident clarity is\nmaintained.\n\n2. Computation is shifted client-side. Often, an otherwise lightweight\nwebapp loses considerable performance rendering complex templates. By\nkeeping the templates light-weight and doing control presentation and\nhandling in JS, high performance is preserved.\n\n\nWhat Toolbox Doesn't Do\n-----------------------\n\n* versioning: toolbox exposes editing towards a canonical document.\n  It doesn't do versioning.  A model instance may do whatever\n  versioning it desires, and since the models are pluggable, it would\n  be relatively painless to subclass e.g. the file-based model and\n  have a post-save hook which does an e.g. ``hg commit``. Customized\n  templates could be used to display this information.\n\n* authentication: the information presented by toolbox is freely\n  readable and editable. This is by intention, as by going to a \"wiki\"\n  model and presenting a easy to use, context-switching-free interface\n  curation is encouraged (ignoring the possibly imaginary problem of\n  wiki-spam). Access-level auth could be implemented using WSGI\n  middleware (e.g. repoze.who or bitsyauth) or through a front end\n  \"webserver\" integration layer such as Apache or nginx. Finer grained\n  control of the presentation layer could be realized by using custom\n  templates.\n\n\nWhat Toolbox Would Like To Do\n-----------------------------\n\nUltimately, toolbox should be as federated as possible.  The basic\narchitecture of toolbox as a web service + supporting scripts makes\nthis feasible and more self-contained than most proposed federated\nservices.  The basic federated model has proved, in practice,\ndifficult to achieve through purely the (HTTP) client-server model, as\nwithout complete federation and adherence to protocol offline cron\njobs should be utilized to pull external data sources. If a webservice\nonly desires to talk to others of its own type and are willing to keep\na queue of requests for when hosts are offline, entire HTTP federation\nmay be implemented with only a configuration-specified discovery\nservice to find the nodes.\n\n\nEvolution\n---------\n\nOften, a piece software is presented as a state out of context (that\nis minus the evolution which led it to be and led it to look further\nout towards beyond the horizon).  While this is an interesting special\neffect for an art project, software being communication this\nis only conducive to software in the darkest of black-box approaches.\n\n\"Beers are like web frameworks: if they're not micro, you don't know\nwhat you're talking about.\" - hipsterhacker\n\nFor sites that fit the architecture of a given framework, it may be\nadvisable to make use of them.  However, for most webapp/webservice\ncategories which have a finite scope and definitive intent, it is\noften easier, more maintainable, and more legible to build a complete\nHTTP->WSGI->app architecture than to try to hammer a framework into\nfitting your problem or redefining the problem to fit the framework.\nThis approach was used for toolbox.\n\nThe GenshiView template, http://k0s.org/hg/GenshiView, was invoked to\ngenerate a basic dispatcher->handler system.  The cruft was removed,\nleaving only the basic structure and the TempitaHandler since tempita\nis lightweight and it was envisioned that filesystem tempita templates\n(MakeItSo!) would be used elsewhere in the project.  The basic\nhandlers (projects views, field-sorted view, new, etc.) were written\nand soon a usable interface was constructed.\n\nA ``sample`` directory was created to hold the JSON blobs. Because\nthis was done early on, two goals were achieved: \n\n1. the software could be dogfooded immediately using actual applicable\ndata. This helped expose a number of issues concerning the data format\nright away.\n\n2. There was a place to put tools before the project reached a\ndeployable state (previously, a few had lived in a static state using\na rough sketch of the HTML microformat discussed above on\nk0s.org). Since the main point of toolbox is to record Mozilla tools,\nthe wealth of references mentioned in passing could be put somewhere,\ninstead of passed by and forgotten.  One wishes that they do not miss\nthe train while purchasing a ticket.\n\nThe original intent, when the file-based JSON blob approach was to be\nthe deployed backend, was to have two repositories: one for the code\nand one for the JSON blobs.  When this approach was scrapped, the\nfile-based JSON blobs were relegated to the ``sample`` directory, with\nthe intent to be to import them into e.g. a couch database on actual\ndeployment (using an import script). The samples could then be used\nfor testing.\n\nThe model has a single \"setter\" function, ``def update``, used for\nboth creating and updating projects.  Due to this and due to the fact\nthe model was ABC/pluggable from the beginning, a converter ``export``\nfunction could be trivially written at the ABC-level::\n\n    def export(self, other):\n        \"\"\"export the current model to another model instance\"\"\"\n        for project in self.get():\n            other.update(project)\n\nThis with an accompanying CLI utility was used to migrate from JSON\nblob files in the ``sample`` directory to the couch instance.  This\nparticular methodology as applied to an unexpected problem (the\nunanticipated switch from JSON blobs to couch) is a good example of\nthe power of using a problem to drive the software forward (in this\ncase, creation of a universal export function and associated command\nline utility). The alternative, a one-off manual data migration, would\nhave been just as time consuming, would not be repeatable, would not\nhave extended toolbox, and may have (like many one-offs do) infected\nthe code base with associated semi-permanant vestiges.  In general,\nproblems should be used to drive innovation.  This can only be done if\nthe software is kept in a reasonably good state.  Otherwise\nconsiderable (though probably worthwhile) refactoring should be done\nprior to feature extension which will become cost-prohibitive in\ntime-critical situations where a one-off is (more) likely to be employed.\n\n\nUse Cases\n---------\n\nThe target use-case is software tools for Mozilla, or, more generally,\na software index.  For this case, the default fields uses are given in\nthe paste.ini file: usage, author, type, language. More fields may be\nadded to the running instance in the future.\n\nHowever, the classifier classification can be used for a wide variety\nof web-locatable resources.  A few examples:\n\n* songs: artist, album, genre, instruments\n* de.li.cio.us: type, media, author, site\n\n\nResources\n---------\n\n* http://readthedocs.org/",
        "description_content_type": null,
        "docs_url": null,
        "download_url": "UNKNOWN",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "http://k0s.org/hg/toolbox/",
        "keywords": null,
        "license": "MPL",
        "maintainer": null,
        "maintainer_email": null,
        "name": "toolk0s",
        "package_url": "https://pypi.org/project/toolk0s/",
        "platform": "UNKNOWN",
        "project_url": "https://pypi.org/project/toolk0s/",
        "project_urls": {
            "Download": "UNKNOWN",
            "Homepage": "http://k0s.org/hg/toolbox/"
        },
        "release_url": "https://pypi.org/project/toolk0s/0.4/",
        "requires_dist": null,
        "requires_python": null,
        "summary": "a place to list links + tools",
        "version": "0.4"
    },
    "last_serial": 2529048,
    "releases": {
        "0.4": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "83f4d6fe2626f3f56624b99a8ec9beab",
                    "sha256": "6767d984832f157a6a0d6dcd615ab9efaeffd2590c3c86d17c1c24d2b02d1ac2"
                },
                "downloads": -1,
                "filename": "toolk0s-0.4.tar.gz",
                "has_sig": false,
                "md5_digest": "83f4d6fe2626f3f56624b99a8ec9beab",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 29021,
                "upload_time": "2016-12-19T20:28:34",
                "url": "https://files.pythonhosted.org/packages/74/b4/a23adfc35c5e1830680da0bef14a2ead4942820b419e47760d0e5cbc44b7/toolk0s-0.4.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "83f4d6fe2626f3f56624b99a8ec9beab",
                "sha256": "6767d984832f157a6a0d6dcd615ab9efaeffd2590c3c86d17c1c24d2b02d1ac2"
            },
            "downloads": -1,
            "filename": "toolk0s-0.4.tar.gz",
            "has_sig": false,
            "md5_digest": "83f4d6fe2626f3f56624b99a8ec9beab",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 29021,
            "upload_time": "2016-12-19T20:28:34",
            "url": "https://files.pythonhosted.org/packages/74/b4/a23adfc35c5e1830680da0bef14a2ead4942820b419e47760d0e5cbc44b7/toolk0s-0.4.tar.gz"
        }
    ]
}