{
"info": {
"author": "Thomas Aglassinger",
"author_email": "roskakori@users.sourceforge.net",
"bugtrack_url": null,
"classifiers": [
"Development Status :: 4 - Beta",
"Environment :: Plugins",
"Intended Audience :: Developers",
"License :: OSI Approved :: GNU Library or Lesser General Public License (LGPL)",
"Natural Language :: English",
"Operating System :: OS Independent",
"Programming Language :: Python :: 2.5",
"Programming Language :: Python :: 2.6",
"Programming Language :: Python :: 2.7",
"Topic :: Software Development :: Libraries"
],
"description": "Proconex is a module to simplify the implementation of the producer/consumer\nidiom. In addition to simple implementations based on Python's ``Queue.Queue``,\nproconex also takes care of exceptions raised during producing or consuming\nitems and ensures that all the work shuts down in a clean manner without\nleaving zombie threads.\n\nExample Usage\n=============\n\nIn order to use proconex, we need a few preparations.\n\nFirst, set up Python's logging:\n\n>>> import logging\n>>> logging.basicConfig(level=logging.INFO)\n\nIn case you want to use the `with` statement to clean up and still use Python\n2.5, you need to import it:\n\n>>> from __future__ import with_statement\n\nAnd finally, we of course need to import proconex itself:\n\n>>> import proconex\n\nHere is a simple producer that reads lines from a file:\n\n>>> class LineProducer(proconex.Producer):\n... def __init__(self, fileToReadPath):\n... super(LineProducer, self).__init__()\n... self._fileToReadPath = fileToReadPath\n... def items(self):\n... with open(self._fileToReadPath, 'rb') as fileToRead:\n... for lineNumber, line in enumerate(fileToRead, start=1):\n... yield (lineNumber, line.rstrip('\\n\\r'))\n\nThe constructor can take any parameters you need to set up the producer. In\nthis case, all we need is the path to the file to read, ``fileToReadPath``.\nThe constructor simply stores the value in an attribute for later reference.\n\nThe function ``items()`` typically is implemented as generator and yields the\nproduced items one after another until there are no more items to produce. In\nthis case, we just return the file line by line as a tuple of line number and\nline contents without trailing newlines.\n\nNext, we need a consumer. Here is a simple one that processes the lines read\nby the producer above and prints its number and text:\n\n>>> class LineConsumer(proconex.Consumer):\n... def consume(self, item):\n... lineNumber, line = item\n... if \"self\" in line:\n... print u\"line %d: %s\" % (lineNumber, line)\n\nWith classes for producer and consumer defined, we can create a producer and a\nlist of consumers:\n\n>>> producer = LineProducer(__file__)\n>>> consumers = [LineConsumer(\"consumer#%d\" % consumerId)\n... for consumerId in xrange(3)]\n\nTo actually start the production process, we need a worker to control the\nproducer and consumers:\n\n>>> with proconex.Worker(producer, consumers) as lineWorker:\n... lineWorker.work() # doctest: +ELLIPSIS\nline ...\n\nThe with statement makes sure that all threads are terminated once the worker\nfinished or failed. Alternatively you can use ``try ... except ... finally``\nto handle errors and cleanup:\n\n>>> producer = LineProducer(__file__)\n>>> consumers = [LineConsumer(\"consumer#%d\" % consumerId)\n... for consumerId in xrange(3)]\n>>> lineWorker = proconex.Worker(producer, consumers)\n>>> try:\n... lineWorker.work()\n... except Exception, error:\n... print error\n... finally:\n... lineWorker.close() # doctest: +ELLIPSIS\nline ...\n\nAdditionally to ``Worker`` there also is a ``Converter``, which not only\nproduces and consumes items but also yields converted items. While\n``Converter``s use the same ``Producer``s as ``Worker``s, they require\ndifferent consumers based on ``ConvertingConsumer``. Such a consumer has a\n``addItem()`` which ``consume()`` should use to add the converted item.\n\nHere is an example for a consumer that converts consumed integer numbers to\ntheir square value:\n\n>>> class SquareConvertingIntegerConsumer(proconex.ConvertingConsumer):\n... def consume(self, item):\n... self.addItem(item * item)\n\nA fitting producer for integer numbers between 0 and 4 is:\n\n>>> class IntegerProducer(proconex.Producer):\n... def items(self):\n... for item in xrange(5):\n... yield item\n\nCombining these in a converter, we get:\n\n>>> with proconex.Converter(IntegerProducer(\"producer\"),\n... SquareConvertingIntegerConsumer(\"consumer\")) as converter:\n... for item in converter.items():\n... print item\n0\n1\n4\n9\n16\n\nLimitations\n===========\n\nWhen using proconex, there are a few things you should be aware of:\n\n* Due to Python's Global Interpreter Lock (GIL), at least one of producer and\n consumer should be I/O bound in order to allow thread switches.\n* The code contains a few polling loops because ``Queue`` does\n not support canceling `get()` and `put()`. The polling does not drain the\n CPU because it uses a timeout when waiting for events to happen. Still,\n there is room for improvement and contributions are welcome.\n* The only way to recover from errors during production is to restart the\n whole process from the beginning.\n\nIf you need more flexibility and control than proconex offers, try\n`celery `_.\n\nSource code\n===========\n\nProconex is distributed under the GNU Lesser General Public License, version 3\nor later.\n\nThe source code is available from .\n\nVersion history\n===============\n\nVersion 0.4, 2012-04-14\n\n* Fixed occasional premature termination of ``Converter`` which could lead to\n the consumers ignoring the last few items put on the queue by the producer.\n\nVersion 0.3, 2012-01-06\n\n* Added ``Converter`` class, which is similar to ``Worker`` but expects\n consumers to yield results the caller can process.\n* Changed exceptions raised by producer and consumer to preserve their stack\n trace when passed to the ``Worker`` or ``Converter``.\n\nVersion 0.2, 2012-01-04\n\n* Added support for multiple producers.\n* Added limit for queue size. By default it is twice the number of consumers.\n\nVersion 0.1, 2012-01-03\n\n* Initial public release.",
"description_content_type": null,
"docs_url": null,
"download_url": "UNKNOWN",
"downloads": {
"last_day": -1,
"last_month": -1,
"last_week": -1
},
"home_page": "http://pypi.python.org/pypi/proconex/",
"keywords": "xml output stream large big huge namespace unicode memory footprint",
"license": "GNU Lesser General Public License 3 or later",
"maintainer": null,
"maintainer_email": null,
"name": "proconex",
"package_url": "https://pypi.org/project/proconex/",
"platform": "UNKNOWN",
"project_url": "https://pypi.org/project/proconex/",
"project_urls": {
"Download": "UNKNOWN",
"Homepage": "http://pypi.python.org/pypi/proconex/"
},
"release_url": "https://pypi.org/project/proconex/0.4/",
"requires_dist": null,
"requires_python": null,
"summary": "producer/consumer with exception handling",
"version": "0.4"
},
"last_serial": 796717,
"releases": {
"0.1": [
{
"comment_text": "",
"digests": {
"md5": "c3084e30af1418ec124e3265da719a70",
"sha256": "4a4e7a04cefa27369e18b60647b69b11da7fa0ff559ef6d3ba10fa3bf446e994"
},
"downloads": -1,
"filename": "proconex-0.1.zip",
"has_sig": false,
"md5_digest": "c3084e30af1418ec124e3265da719a70",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 11836,
"upload_time": "2012-01-03T14:26:09",
"url": "https://files.pythonhosted.org/packages/05/65/d08a1e0a5c0fdc84e57cb437a51e4669168d2e1e966cd87cc89c15067fb1/proconex-0.1.zip"
}
],
"0.2": [
{
"comment_text": "",
"digests": {
"md5": "93a0eae6ce12dfd17b281fbdcd4dcf7f",
"sha256": "5e6c34d2815170404c3a50d659f6db4280204b36a1901b56897135f78a67fde8"
},
"downloads": -1,
"filename": "proconex-0.2.zip",
"has_sig": false,
"md5_digest": "93a0eae6ce12dfd17b281fbdcd4dcf7f",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 12926,
"upload_time": "2012-01-04T17:58:53",
"url": "https://files.pythonhosted.org/packages/d2/cb/b3994d237afae1f7cb399060cd8287d62d2634d68b74423f58d1bf85d677/proconex-0.2.zip"
}
],
"0.3": [
{
"comment_text": "",
"digests": {
"md5": "a0fa03897c03a8679d17c4e53aba8f40",
"sha256": "d8b076b0b660c66b0a3cbe0dc139afd8b243ddbc0fd7ee889c1140e09b1737d4"
},
"downloads": -1,
"filename": "proconex-0.3.zip",
"has_sig": false,
"md5_digest": "a0fa03897c03a8679d17c4e53aba8f40",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 14742,
"upload_time": "2012-01-06T18:14:24",
"url": "https://files.pythonhosted.org/packages/e1/67/895e6a0c2500bee075a50ad40f28d0459d944b96dfde9531b253da28db62/proconex-0.3.zip"
}
],
"0.4": [
{
"comment_text": "",
"digests": {
"md5": "50a1c1ca5cdc8fe2a68a9249dcbeff79",
"sha256": "efe0c5bb5fa01110661f5920d5edbb17d466081e1e83645f7d949571d13b4854"
},
"downloads": -1,
"filename": "proconex-0.4.zip",
"has_sig": false,
"md5_digest": "50a1c1ca5cdc8fe2a68a9249dcbeff79",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 14947,
"upload_time": "2012-04-14T19:14:35",
"url": "https://files.pythonhosted.org/packages/c7/98/b1f004743e51875e1a0962bbcebf75ce2e407bb3866e54099131b17de8e2/proconex-0.4.zip"
}
]
},
"urls": [
{
"comment_text": "",
"digests": {
"md5": "50a1c1ca5cdc8fe2a68a9249dcbeff79",
"sha256": "efe0c5bb5fa01110661f5920d5edbb17d466081e1e83645f7d949571d13b4854"
},
"downloads": -1,
"filename": "proconex-0.4.zip",
"has_sig": false,
"md5_digest": "50a1c1ca5cdc8fe2a68a9249dcbeff79",
"packagetype": "sdist",
"python_version": "source",
"requires_python": null,
"size": 14947,
"upload_time": "2012-04-14T19:14:35",
"url": "https://files.pythonhosted.org/packages/c7/98/b1f004743e51875e1a0962bbcebf75ce2e407bb3866e54099131b17de8e2/proconex-0.4.zip"
}
]
}