{
    "info": {
        "author": "Jim Fulton",
        "author_email": "jim@zope.com",
        "bugtrack_url": null,
        "classifiers": [],
        "description": "*************\nCatalog Queue\n*************\n\nA catalog queue provides a queue for catalog indexing. The basic idea\nis to queue catalog operations so:\n\n- Operations can be batched for greater efficiency\n\n- Application requests don't have to wait for indexing to be done\n\nThe benefits of queueing are especially significant when text indexes\nare used.\n\n.. contents::\n\nDetailed Documentation\n**********************\n\nUsing Queues\n============\n\nA queue is created by instantiating a\nzc.catalogqueue.queue.CatalogQueue object:\n\n    >>> import zc.catalogqueue.queue\n    >>> queue = zc.catalogqueue.queue.CatalogQueue()\n\nWe can pass a queue size.  It should be a prime number. The default is\n1009, which is a bit large.\n\n    >>> queue = zc.catalogqueue.queue.CatalogQueue(11)\n\nTypically, queues are registered as\nzc.catalogqueue.interfaces.ICatalogQueue utilities.\n\n    >>> import zope.interface, pprint\n    >>> pprint.pprint(sorted(zope.interface.providedBy(queue)), width=1)\n    [<InterfaceClass zc.catalogqueue.interfaces.ICatalogQueue>,\n     <InterfaceClass persistent.interfaces.IPersistent>]\n\nThere are some bits of information that the queue maintains regarding its own\nprocessing state.  The time of last processing and the total number of\ncataloging events processed are available.  Since this queue hasn't been\nprocessed yet, these have some initial values:\n\n    >>> print queue.lastProcessedTime\n    None\n    >>> queue.totalProcessed\n    0\n\nThe length of the queue provides access to the number of pending cataloging\nevents:\n\n    >>> len(queue)\n    0\n\nQueues are used in 2 ways.  As content are modified, we call add,\nupdate, and remove methods on the queue:\n\n    >>> queue.add(1)\n    >>> queue.update(1)\n    >>> queue.remove(1)\n\n    >>> queue.update(2)\n    >>> queue.update(2)\n\n    >>> queue.add(3)\n    >>> queue.update(3)\n    >>> queue.add(3)\n    Traceback (most recent call last):\n    ...\n    TypeError: Attempt to add an object that is already in the catalog\n\n    >>> queue.update(4)\n    >>> queue.update(4)\n    >>> queue.update(4)\n\n    >>> queue.remove(5)\n    >>> queue.update(5)\n    Traceback (most recent call last):\n    ...\n    TypeError: Attempt to change an object that has been removed\n\n    >>> queue.update(0)\n    >>> queue.update(0)\n\nAt this point, we've added several events, but haven't processed the queue, so\nwe expect ``lastProcessedTime``, ``totalProcessed`` to be unchanged, but the\nqueue length to reflect the pending tasks:\n\n    >>> print queue.lastProcessedTime\n    None\n    >>> queue.totalProcessed\n    0\n    >>> len(queue)\n    6\n\nPeriodically, we call process on the queue.  We need to pass an ids\nobject and a collection of injection (catalog) objects:\n\n    >>> class Ids:\n    ...     def queryObject(self, id, default=None):\n    ...         if not id:\n    ...             return default\n    ...         return \"object %s\" % id\n\n    >>> class Injection:\n    ...     def __init__(self, name):\n    ...         self.name = name\n    ...     def index_doc(self, docid, value):\n    ...         print self.name, 'indexing', docid, value\n    ...     def unindex_doc(self, docid):\n    ...         print self.name, 'unindexing', docid\n\n    >>> queue.process(Ids(), [Injection('cat1'), Injection('cat2')], 10)\n    cat1 unindexing 1\n    cat2 unindexing 1\n    cat1 indexing 2 object 2\n    cat2 indexing 2 object 2\n    cat1 indexing 3 object 3\n    cat2 indexing 3 object 3\n    cat1 indexing 4 object 4\n    cat2 indexing 4 object 4\n    cat1 unindexing 5\n    cat2 unindexing 5\n    6\n\nThere are a number of things to note about this example:\n\n- Each object is processed only once.\n\n- What happens depends on the last event.\n\n- Object 0 wasn't indexed because queryObject returned None.  We\n  ignore events for objects that have been removed from the intid\n  utility.\n\n- The number of objects processed is returned.\n\nThe processing information has been updated on the queue:\n\n    >>> queue.lastProcessedTime  # doctest: +ELLIPSIS\n    datetime.datetime(... tzinfo=<UTC>)\n    >>> queue.totalProcessed\n    6\n\n    >>> previous_time = queue.lastProcessedTime\n\nThe length of the queue now indicates that no further events are pending:\n\n    >>> len(queue)\n    0\n\nIf we process the queue without additional events, we'll just get 0\nback:\n\n    >>> queue.process(Ids(), [Injection('cat1'), Injection('cat2')], 10)\n    0\n\nThe historical processing information is updated:\n\n    >>> queue.lastProcessedTime  # doctest: +ELLIPSIS\n    datetime.datetime(... tzinfo=<UTC>)\n    >>> queue.lastProcessedTime > previous_time\n    True\n    >>> queue.totalProcessed\n    6\n\n    >>> len(queue)\n    0\n\nOf course, the limit argument limits how many events we process:\n\n    >>> for i in range(10):\n    ...     queue.update(i)\n    >>> len(queue)\n    10\n    >>> queue.process(Ids(), [Injection('cat1')], 5)\n    cat1 indexing 1 object 1\n    cat1 indexing 2 object 2\n    cat1 indexing 3 object 3\n    cat1 indexing 4 object 4\n    5\n    >>> queue.totalProcessed\n    11\n    >>> len(queue)\n    5\n\n    >>> queue.process(Ids(), [Injection('cat1')], 5)\n    cat1 indexing 5 object 5\n    cat1 indexing 6 object 6\n    cat1 indexing 7 object 7\n    cat1 indexing 8 object 8\n    cat1 indexing 9 object 9\n    5\n    >>> queue.totalProcessed\n    16\n    >>> len(queue)\n    0\n\n(Remember that 0 isn't processed because it can't be found.)\n\nWhen an object can't be found, a warning is logged:\n\n    >>> import zope.testing.loggingsupport\n    >>> handler = zope.testing.loggingsupport.InstalledHandler('zc')\n    >>> queue.update(0)\n    >>> queue.process(Ids(), [Injection('cat1')], 5)\n    1\n\n    >>> print handler\n    zc.catalogqueue.queue WARNING\n      Couldn't find object for 0\n\n    >>> handler.uninstall()\n\nEdgecase\n========\n\nIf a \"old\" state has two 'ADDED' events, and the committed state processes the\nqueue, and the \"new\" state modifies one of the objects marked for addition, the\ncode marks the other for removal.\n\n    >>> from zc.catalogqueue.CatalogEventQueue import (\n    ...     CatalogEventQueue, ADDED, REMOVED, CHANGED, CHANGED_ADDED)\n    >>> cq = CatalogEventQueue()\n\n    >>> def resolve(old, committed, new):\n    ...     return sorted(cq._p_resolveConflict(\n    ...        {'_conflict_policy': 0, '_data': old},\n    ...        {'_conflict_policy': 0, '_data': committed},\n    ...        {'_conflict_policy': 0, '_data': new}\n    ...        )['_data'].items())\n\n    >>> resolve({1: (1, ADDED), 2: (1, ADDED)}, {},\n    ...         {1: (1, ADDED), 2: (3, REMOVED), 3: (1, ADDED)})\n    [(2, (3, 0)), (3, (1, 1))]\n\nDownload\n********",
        "description_content_type": null,
        "docs_url": null,
        "download_url": "UNKNOWN",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "UNKNOWN",
        "keywords": null,
        "license": "ZPL 2.1",
        "maintainer": null,
        "maintainer_email": null,
        "name": "zc.catalogqueue",
        "package_url": "https://pypi.org/project/zc.catalogqueue/",
        "platform": "UNKNOWN",
        "project_url": "https://pypi.org/project/zc.catalogqueue/",
        "project_urls": {
            "Download": "UNKNOWN",
            "Homepage": "UNKNOWN"
        },
        "release_url": "https://pypi.org/project/zc.catalogqueue/0.3.1/",
        "requires_dist": null,
        "requires_python": null,
        "summary": "UNKNOWN",
        "version": "0.3.1"
    },
    "last_serial": 802164,
    "releases": {
        "0.3.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "45569557c9524101f0fa56cf0596accf",
                    "sha256": "320b20e2a0a067ff87f7a0e65b2201676b0b2c1d43d870f5e72fc8de0cdca36f"
                },
                "downloads": -1,
                "filename": "zc.catalogqueue-0.3.0.tar.gz",
                "has_sig": false,
                "md5_digest": "45569557c9524101f0fa56cf0596accf",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 10684,
                "upload_time": "2009-09-03T23:09:36",
                "url": "https://files.pythonhosted.org/packages/01/61/a4217489563e239af4283b7fa07aea8cf9b8f2ccb22825bc21060ab87602/zc.catalogqueue-0.3.0.tar.gz"
            }
        ],
        "0.3.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "6d0547af880b5c701af5988060e681c0",
                    "sha256": "4ba1396ab4ed808134ba70404c03d285fee2c80bf5975949585753b17f78d4f8"
                },
                "downloads": -1,
                "filename": "zc.catalogqueue-0.3.1.tar.gz",
                "has_sig": false,
                "md5_digest": "6d0547af880b5c701af5988060e681c0",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 11952,
                "upload_time": "2009-10-29T19:14:24",
                "url": "https://files.pythonhosted.org/packages/bb/b6/251fe51bf4cd9954d39fb6bc9b5f6f11018bfefb2493dd675039f251f3d4/zc.catalogqueue-0.3.1.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "6d0547af880b5c701af5988060e681c0",
                "sha256": "4ba1396ab4ed808134ba70404c03d285fee2c80bf5975949585753b17f78d4f8"
            },
            "downloads": -1,
            "filename": "zc.catalogqueue-0.3.1.tar.gz",
            "has_sig": false,
            "md5_digest": "6d0547af880b5c701af5988060e681c0",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 11952,
            "upload_time": "2009-10-29T19:14:24",
            "url": "https://files.pythonhosted.org/packages/bb/b6/251fe51bf4cd9954d39fb6bc9b5f6f11018bfefb2493dd675039f251f3d4/zc.catalogqueue-0.3.1.tar.gz"
        }
    ]
}