The queue module contains the queues that zc.async clients use to
deposit jobs, and the collections of dispatchers and their agents.

The dispatchers expect to find queues in a mapping off the root of the
database in a key given in zc.async.interfaces.KEY, so we'll follow that
pattern, even though it doesn't matter too much for our examples
[#setUp]_.

    >>> import zc.async.queue
    >>> import zc.async.interfaces
    >>> container = root[zc.async.interfaces.KEY] = zc.async.queue.Queues()

Now we can add a queue.  The collection sets the ``parent`` and ``name``
attributes.

    >>> queue = container[''] = zc.async.queue.Queue()
    >>> queue.name
    ''
    >>> queue.parent is container
    True
    >>> import transaction
    >>> transaction.commit()

[#queues_collection]_ As shown in the README.txt of this package (or see
zc.async.adapters.defaultQueueAdapter), the queue with the name '' will
typically be registered as an adapter to persistent objects that
provides zc.async.interfaces.IQueue [#verify]_.

The queue doesn't have any jobs yet.

    >>> len(queue)
    0
    >>> bool(queue)
    False
    >>> list(queue)
    []

It also doesn't have any regsitered dispatchers.

    >>> len(queue.dispatchers)
    0
    >>> bool(queue.dispatchers)
    False
    >>> list(queue.dispatchers)
    []

We'll look at the queue as a collection of jobs; then we'll look at the
``dispatcher`` agents collection; and then we'll look at how the two
interact.

Queues and Jobs
===============

As described in the README, we can put jobs to be performed now in the queue,
and jobs to be performed later. The collection can be introspected, and then
agents can ``claim`` a job or simply remove it with ``pull`` or ``remove``.
We'll examine the differences between these calls below.

We'll start by adding a job to be performed as soon as possible.  Note
that we'll use a testing tool that lets us control the current time
generated by datetime.datetime.now with a `zc.async.testing.set_now`
callable.  This code also expects that a UUID will be registered as a
zc.async.interfaces.IUUID utility with the empty name ('').

    >>> from zc.async.instanceuuid import UUID

    >>> import zc.async.testing
    >>> zc.async.testing.setUpDatetime()

    >>> def mock_work():
    ...     return 42
    ...
    >>> job = queue.put(mock_work)
    >>> len(queue)
    1
    >>> list(queue) == [job]
    True
    >>> queue[0] is job
    True
    >>> bool(queue)
    True
    >>> job.parent is queue
    True
    >>> transaction.commit()

A job added without any special calls gets a `begin_after` attribute
of now.

    >>> import datetime
    >>> import pytz
    >>> now = datetime.datetime.now(pytz.UTC)
    >>> now
    datetime.datetime(2006, 8, 10, 15, 44, 22, 211, tzinfo=<UTC>)
    >>> job.begin_after == now
    True

A ``begin_by`` attribute is a duration (datetime.timedelta) or None, and
defaults to None.

    >>> job.begin_by == None
    True

If ``begin_by`` were an hour, that would mean that it must be completed an hour
after the ``begin_after`` datetime, or else the system will fail it.

Now let's add a job to be performed later, using ``begin_after``.

This means that it's immediately ready to be performed: we can ``claim`` it.
This is the API that agents call on a queue to get a job.

    >>> job is queue.claim()
    True
    >>> job.parent is None
    True

Now the queue is empty.

    >>> len(queue)
    0
    >>> list(queue)
    []

You can specify a begin_after date when you make the call.  Then the job
isn't due immediately.

    >>> import operator
    >>> import zc.async.job
    >>> job2 = queue.put(
    ...     zc.async.job.Job(operator.mul, 7, 6),
    ...     datetime.datetime(2006, 8, 10, 16, tzinfo=pytz.UTC))
    ...
    >>> len(queue)
    1
    >>> job2.begin_after
    datetime.datetime(2006, 8, 10, 16, 0, tzinfo=<UTC>)
    >>> queue.claim() is None
    True

When the time passes, it is available to be claimed.

    >>> zc.async.testing.set_now(
    ...     datetime.datetime(2006, 8, 10, 16, tzinfo=pytz.UTC))
    >>> job2 is queue.claim()
    True
    >>> len(queue)
    0

Jobs are ordered by their begin_after dates for all operations, including
claiming and iterating.

    >>> job3 = queue.put(
    ...     zc.async.job.Job(operator.mul, 14, 3),
    ...     datetime.datetime(2006, 8, 10, 16, 2, tzinfo=pytz.UTC))
    >>> job4 = queue.put(
    ...     zc.async.job.Job(operator.mul, 21, 2),
    ...     datetime.datetime(2006, 8, 10, 16, 1, tzinfo=pytz.UTC))
    >>> job5 = queue.put(
    ...     zc.async.job.Job(operator.mul, 42, 1),
    ...     datetime.datetime(2006, 8, 10, 16, 0, tzinfo=pytz.UTC))

    >>> list(queue) == [job5, job4, job3]
    True
    >>> queue[2] is job3
    True
    >>> queue[1] is job4
    True
    >>> queue[0] is job5
    True

    >>> job5 is queue.claim()
    True
    >>> len(queue)
    2

The ``pull`` method is a way to remove jobs without the connotation of
"claiming" them.  This has several implications that we'll dig into later.
For now, we'll just use it to pull a job, and then return it.

    >>> job4 is queue.pull()
    True
    >>> job4 is queue.put(job4)
    True
    >>> list(queue) == [job4, job3]
    True

The ``pull`` method takes the first item off the queue, or you can pass an
index.

The ``remove`` method lets you remove a specific job from the queue.

    >>> queue.remove(job3)
    >>> list(queue) == [job4]
    True

    >>> queue.remove(job4)
    >>> list(queue)
    []

    >>> queue.remove(job4) # doctest: +ELLIPSIS
    Traceback (most recent call last):
    ...
    LookupError: ('item not in queue', <zc.async.job.Job...>)

    >>> job3 is queue.put(job3)
    True
    >>> job4 is queue.put(job4)
    True
    >>> list(queue) == [job4, job3]
    True

Let's add another job without an explicit due date.

    >>> job6 = queue.put(
    ...     zc.async.job.Job(operator.mod, 85, 43))
    >>> list(queue) == [job6, job4, job3]
    True

Pre-dating (before now) is equivalent to not passing a datetime.

    >>> job7 = queue.put(
    ...     zc.async.job.Job(operator.and_, 43, 106),
    ...     begin_after=datetime.datetime(2006, 8, 10, 15, 35, tzinfo=pytz.UTC))
    ...
    >>> list(queue) == [job6, job7, job4, job3]
    True

Other timezones are normalized to UTC.

    >>> job8 = queue.put(
    ...     zc.async.job.Job(operator.or_, 40, 10),
    ...     pytz.timezone('EST').localize(
    ...         datetime.datetime(2006, 8, 10, 11, 30)))
    ...
    >>> job8.begin_after
    datetime.datetime(2006, 8, 10, 16, 30, tzinfo=<UTC>)

Naive timezones are not allowed.

    >>> queue.put(mock_work, datetime.datetime(2006, 8, 10, 16, 15))
    Traceback (most recent call last):
    ...
    ValueError: cannot use timezone-naive values

``claim``
---------

Above we have mentioned ``claim`` and ``pull``.  The semantics of ``pull``
are the same as zc.queue: it pulls the first item off the queue, unless you
specify an index.

    >>> first = queue[0]
    >>> first is queue.pull(0)
    True
    >>> last = queue[-1]
    >>> last is queue.pull(-1)
    True
    >>> first is queue.put(first)
    True
    >>> last is queue.put(last)
    True

``claim`` is similar in that it removes an item from the queue.  However,
it has several different behaviors:

- It only will give jobs for which the begin_after value is >= now.

- If begin_after + begin_by >= now, a job that makes the original job fail
  is used instead.

- If a job is not failing and has one or more ``quota_names`` and the
  associated quotas are filled with jobs not in the CALLBACKS or COMPLETED
  status then it will not be returned.

- It does not take an index argument.

- It does take a ``filter`` argument, which takes a job and returns a boolean
  True if the job can be accepted.

- If no results are available, it returns None, or a default you pass in,
  rather than raising IndexError.

The ``claim`` method is intended to be the primary interface for agents
interacting with the queue.

Let's examine the behavior of ``claim`` with some examples.  We've already
seen the most basic usage: it has returned the first item in the queue.

Right now the queue has five jobs.  Only two are ready to be started at this
time because of the begin_after values.

    >>> list(queue) == [job7, job6, job4, job3, job8]
    True
    >>> [j for j in queue
    ...  if j.begin_after <= datetime.datetime.now(pytz.UTC)] == [job7, job6]
    True
    >>> queue.claim() is job7
    True
    >>> queue.claim() is job6
    True
    >>> print queue.claim()
    None

Now let's set the time to the begin_after of the last job, but then use a
filter that only accepts jobs that do the ``operator.or_`` job (job8).

    >>> zc.async.testing.set_now(queue[-1].begin_after)
    >>> [j for j in queue
    ...  if j.begin_after <= datetime.datetime.now(pytz.UTC)] == [
    ...  job4, job3, job8]
    True
    >>> def only_or(job):
    ...     return job.callable is operator.or_
    ...
    >>> queue.claim(only_or) is job8
    True
    >>> print queue.claim(only_or)
    None

These filters, as used by agents, allow control over what jobs happen for a
given dispatcher, which typically equates to a given process.

The quotas allow control over what jobs are happening globally for a
given queue.  They are limits, not goals: if you create a quota that can
have a maximum of 1 active job, this will limit jobs that identify
themselves with this quota name to be performed only one at a time, or
serialized.  (To be clear, unlike some uses of the word "quota, "it will
not cause a *preference* for jobs that identify themselves with this
name.)

Let's use some quotas.

Our current jobs are not a part of any quotas.  We'll try to add some
quota_names.

    >>> job4.quota_names
    ()
    >>> job3.quota_names
    ()
    >>> job4.quota_names = ('content catalog',)
    Traceback (most recent call last):
    ...
    ValueError: ('unknown quota name', 'content catalog')

The same kind of error happens if we try to put a job with unknown quota
names in a queue.

    >>> job4 is queue.pull()
    True
    >>> print job4.parent
    None
    >>> job4.status == zc.async.interfaces.NEW
    True
    >>> job4.quota_names = ('content catalog',)
    >>> queue.put(job4)
    Traceback (most recent call last):
    ...
    ValueError: ('unknown quota name', 'content catalog')

Note that the attribute on the job is quota_names: it expects an iterable
of strings, not a string.  The code tries to help catch type errors, at a
trivial level, by bailing out on strings:

    >>> job4.quota_names = ''
    Traceback (most recent call last):
    ...
    TypeError: provide an iterable of names
    >>> job4.quota_names
    ('content catalog',)

We need to add the quota to the queue to be able to add it.

    >>> queue.quotas.create('content catalog', 1)
    >>> quota = queue.quotas['content catalog']
    >>> quota.name
    'content catalog'
    >>> quota.parent is queue.quotas
    True
    >>> quota.size
    1
    >>> list(quota)
    []

Now we can add job4.  We'll make job3 specify the same quota while it is in
the queue.

    >>> job4 is queue.put(job4)
    True
    >>> job3.quota_names = ('content catalog',)

Now I can claim job4 and put in a (stub) agent.  Until job4 has moved to the
CALLBACKS or COMPLETED status, I will be unable to claim job4.

    >>> import zope.interface
    >>> import persistent.list
    >>> import BTrees
    >>> class Completed(persistent.Persistent):
    ...     def __init__(self):
    ...         self._data = BTrees.family64.IO.BTree()
    ...     def add(self, value):
    ...         key = zc.async.utils.dt_to_long(
    ...             datetime.datetime.now(pytz.UTC)) + 15
    ...         while key in self._data:
    ...             key -= 1
    ...         value.key = key
    ...         self._data[key] = value
    ...     def first(self):
    ...         return self._data[self._data.minKey()]
    ...     def __iter__(self):
    ...         return self._data.values()
    ...
    >>> class StubAgent(persistent.list.PersistentList):
    ...     zope.interface.implements(zc.async.interfaces.IAgent)
    ...     parent = name = None
    ...     size = 3
    ...     def __init__(self):
    ...         self.completed = Completed()
    ...         persistent.list.PersistentList.__init__(self)
    ...     @property
    ...     def queue(self):
    ...         return self.parent.parent
    ...     def claimJob(self, filter=None):
    ...         if len(self) < self.size:
    ...             job = self.queue.claim(filter=filter)
    ...             if job is not None:
    ...                 job.parent = self
    ...                 self.append(job)
    ...                 return job
    ...     def pull(self, index=0):
    ...         res = self.pop(index)
    ...         res.parent = None
    ...         return res
    ...     def remove(self, item):
    ...         self.pull(self.index(item))
    ...     def jobCompleted(self, job):
    ...         persistent.list.PersistentList.remove(self, job)
    ...         self.completed.add(job)
    ...     def reschedule(self, item, when):
    ...         now = datetime.datetime.utcnow()
    ...         if isinstance(when, datetime.datetime):
    ...             self.remove(job)
    ...             if when.tzinfo is not None:
    ...                 when = when.astimezone(pytz.UTC).replace(tzinfo=None)
    ...             if when <= now:
    ...                 self.queue.disclaim(item)
    ...             else:
    ...                 self.queue.put(item, begin_after=when)
    ...         elif isinstance(when, datetime.timedelta):
    ...             self.remove(job)
    ...             if when <= datetime.timedelta():
    ...                 self.queue.disclaim(item)
    ...             else:
    ...                 self.queue.put(item, begin_after=now+when)
    ...         else:
    ...             raise TypeError('``when`` must be datetime or timedelta')
    ...

    >>> job4 is queue.claim()
    True
    >>> job4.parent = StubAgent()
    >>> job4.parent.append(job4)
    >>> job4.status == zc.async.interfaces.ASSIGNED
    True
    >>> list(quota) == [job4]
    True
    >>> [j.quota_names for j in queue]
    [('content catalog',)]
    >>> print queue.claim()
    None
    >>> job4()
    42
    >>> job4.status == zc.async.interfaces.COMPLETED
    True
    >>> job3 is queue.claim()
    True

The final characteristic of ``claim`` to review is that jobs that have
timed out are returned in wrappers that fail the original job.

    >>> job9_from_outer_space = queue.put(mock_work)
    >>> job9_from_outer_space.begin_by = datetime.timedelta(hours=1)
    >>> zc.async.testing.set_now(
    ...     datetime.datetime.now(pytz.UTC) +
    ...     job9_from_outer_space.begin_by + datetime.timedelta(seconds=1))
    >>> job9 = queue.claim()
    >>> job9 is job9_from_outer_space
    False
    >>> stub = root['stub'] = StubAgent()
    >>> job9.parent = stub
    >>> stub.append(job9)
    >>> transaction.commit()
    >>> job9()
    >>> job9_from_outer_space.status == zc.async.interfaces.COMPLETED
    True
    >>> print job9_from_outer_space.result.getTraceback()
    ... # doctest: +NORMALIZE_WHITESPACE
    Traceback (most recent call last):
    Failure: zc.async.interfaces.TimeoutError:
    <BLANKLINE>

Dispatchers
===========

When a queue is installed, dispatchers register and activate themselves.
Dispatchers typically get their UUID from the instanceuuid module in
this package, but we will generate our own here.

First we'll register dispatcher using the instance UUID we introduced near the
beginning of this document.

    >>> UUID in queue.dispatchers
    False
    >>> queue.dispatchers.register(UUID)
    >>> UUID in queue.dispatchers
    True

The registration fired off an event.  This may be used by subscribers to
create some agents, if desired.

    >>> from zope.component import eventtesting
    >>> import zc.async.interfaces
    >>> evs = eventtesting.getEvents(
    ...     zc.async.interfaces.IDispatcherRegistered)
    >>> evs # doctest: +ELLIPSIS
    [<zc.async.interfaces.DispatcherRegistered object at ...>]

We can get the dispatcher's collection of agents (an IDispatcherAgents)
now from the ``dispatchers`` collection.  This is the object attached to
the event seen above.

    >>> verifyObject(zc.async.interfaces.IDispatchers, queue.dispatchers)
    True
    >>> da = queue.dispatchers[UUID]
    >>> verifyObject(zc.async.interfaces.IDispatcherAgents, da)
    True
    >>> da.UUID == UUID
    True
    >>> da.parent is queue
    True

    >>> evs[0].object is da
    True

[#check_dispatchers_mapping]_ The object is not activated, and has not
been pinged.

    >>> print da.activated
    None
    >>> print da.last_ping.value
    None

When the object's ``last_ping.value`` + ``ping_interval`` is greater than now,
a new ``last_ping.value`` should be recorded, as we'll see below.  If the
``last_ping.value`` (or ``activated``, if more recent) +
``ping_death_interval`` is older than now, the dispatcher is considered to
be ``dead``.

    >>> da.ping_interval
    datetime.timedelta(0, 30)
    >>> da.ping_death_interval
    datetime.timedelta(0, 60)
    >>> da.dead
    False

``activate`` activates the dispatcher.  A ``ping`` is not expected until the
dispatcher is activated, though, to handle unusual circumstances such as the
dispatcher being wrongfully declared dead ("I'm not dead yet!").

    >>> bool(da.activated)
    False
    >>> queue.dispatchers.ping(UUID)
    >>> bool(da.activated)
    True
    >>> print event_logs # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
    z...
    zc.async.events CRITICAL
      Dispatcher UUID('...') not activated prior to ping. This can indicate
      that the dispatcher's ping_death_interval is set too short, or that some
      transactions in the system are taking too long to commit. Activating, to
      correct the current problem, but if the dispatcher was inappropriately
      viewed as ``dead`` and deactivated, you should investigate the cause
      (be sure to check that time is synced on all participating machines).
    >>> da.deactivate()
    >>> bool(da.activated)
    False

Now we'll activate the dispatcher the proper way.

    >>> import datetime
    >>> import pytz
    >>> now = datetime.datetime.now(pytz.UTC)
    >>> da.activate()
    >>> now <= da.activated <= datetime.datetime.now(pytz.UTC)
    True

It's still not dead. :-)

    >>> da.dead
    False

This also fired an event.

    >>> evs = eventtesting.getEvents(
    ...     zc.async.interfaces.IDispatcherActivated)
    >>> evs # doctest: +ELLIPSIS
    [<zc.async.interfaces.DispatcherActivated object at ...>]
    >>> evs[0].object is da
    True

Now, after activation, a dispatcher should iterate over agents and look for
jobs.  There are not any agents at the moment.

    >>> len(da)
    0

Agents are a pluggable part of the design.  The implementation in this
package is a reasonable default.  For this document, we'll use a simple
and very incomplete stub.  See other documents in this package for use
of the default.

In real usage, perhaps a subscriber to one of the events above will add
an agent, or a user will create one manually.  We'll add our stub to the
DispatcherAgents collection.

    >>> agent = da['main'] = StubAgent()
    >>> agent.name
    'main'
    >>> agent.parent is da
    True

Now, if we had a real dispatcher for our UUID, every few seconds it would poll
its agents for any new jobs.  It would also call ``ping`` on the
queue.dispatchers object.  Let's do an imaginary run.

    >>> import zc.twist
    >>> import twisted.python.failure
    >>> def getJob(agent):
    ...     try:
    ...         job = agent.claimJob()
    ...     except zc.twist.EXPLOSIVE_ERRORS:
    ...         raise
    ...     except:
    ...         agent.failure = zc.twist.sanitize(
    ...             twisted.python.failure.Failure())
    ...         # we'd log here too
    ...         job = None
    ...     return job
    >>> jobs_to_do = []
    >>> def doJobsStub(job):
    ...     jobs_to_do.append(job)
    ...
    >>> activated = set()
    >>> import ZODB.POSException
    >>> def pollStub(conn):
    ...     for queue in conn.root()['zc.async'].values():
    ...         if UUID not in queue.dispatchers:
    ...             queue.dispatchers.register(UUID)
    ...         da = queue.dispatchers[UUID]
    ...         if queue._p_oid not in activated:
    ...             if da.activated:
    ...                 if da.dead:
    ...                     da.deactivate()
    ...                 else:
    ...                     # log problem
    ...                     print "already activated: another process?"
    ...                     continue
    ...             da.activate()
    ...             activated.add(queue._p_oid)
    ...             # be sure to remove if transaction fails
    ...             try:
    ...                 transaction.commit()
    ...             except (SystemExit, KeyboardInterrupt):
    ...                 transaction.abort()
    ...                 raise
    ...             except:
    ...                 # log problem
    ...                 print "problem..."
    ...                 transaction.abort()
    ...                 activated.remove(queue._p_oid)
    ...                 continue
    ...         for agent in da.values():
    ...             job = getJob(agent)
    ...             while job is not None:
    ...                 doJobsStub(job)
    ...                 job = getJob(agent)
    ...         queue.dispatchers.ping(UUID)
    ...         try:
    ...             transaction.commit()
    ...         except (SystemExit, KeyboardInterrupt):
    ...             transaction.abort()
    ...             raise
    ...         except:
    ...             # log problem
    ...             print "problem..."
    ...             transaction.abort()
    ...             activated.remove(key)
    ...             continue
    ...

Running this now will generate an "already activated" warning, because we've
already manually activated the agent.  Normally this would only happen when
the same instance were started more than once simultaneously--a situation that
could be accomplished with ``zopectl start`` followed by ``zopectl debug`` for
instance.

    >>> pollStub(conn)
    already activated: another process?

So, we'll just put the queue's oid in ``activated``.

    >>> activated.add(queue._p_oid)

Now, when we poll, we get a ping.

    >>> before = datetime.datetime.now(pytz.UTC)
    >>> pollStub(conn)
    >>> before <= da.last_ping.value <= datetime.datetime.now(pytz.UTC)
    True

We don't have any jobs to claim yet.  Let's add one and do it again.
We'll use a test fixture, time_flies, to make the time change.

    >>> job10 = queue.put(mock_work)

    >>> def time_flies(seconds):
    ...     zc.async.testing.set_now(
    ...         datetime.datetime.now(pytz.UTC) +
    ...         datetime.timedelta(seconds=seconds))
    ...

    >>> last_ping = da.last_ping.value
    >>> time_flies(5)
    >>> pollStub(conn)
    >>> da.last_ping.value == last_ping
    True

    >>> len(jobs_to_do)
    1
    >>> print queue.claim()
    None

The ping_time won't change for at least another da.ping_interval from the
original ping.  We've already gone through 5 seconds.  We'll fly through
10 and then 15 for the rest.

    >>> time_flies(10)
    >>> pollStub(conn)
    >>> da.last_ping.value == last_ping
    True
    >>> time_flies(15)
    >>> pollStub(conn)
    >>> da.last_ping.value > last_ping
    True

Dead Dispatchers
----------------

What happens when a dispatcher dies?  If it is the only one, that's the
end: when it restarts it should clean out the old jobs in its agents and
then proceed.  But what if a queue has more than one simultaneous
dispatcher?  How do we know to clean out the dead dispatcher's jobs?

The ``ping`` method not only changes the ``last_ping.value`` but checks the
next sibling dispatcher, as defined by UUID, to make sure that it is not
dead. It uses the ``dead`` attribute, introduced above, to test whether
the sibling is alive.

We'll introduce another virtual dispatcher to show this behavior.

    >>> import uuid
    >>> alt_UUID = uuid.uuid1()
    >>> queue.dispatchers.register(alt_UUID)
    >>> alt_da = queue.dispatchers[alt_UUID]
    >>> alt_da.activate()
    >>> zc.async.testing.set_now(
    ...     datetime.datetime.now(pytz.UTC) +
    ...     alt_da.ping_death_interval + datetime.timedelta(seconds=1))
    >>> alt_da.dead
    True
    >>> bool(alt_da.activated)
    True
    >>> pollStub(conn)
    >>> bool(alt_da.activated)
    False

Let's do that again, with an agent in the new dispatcher and some jobs in the
agent. Assigned jobs will be reassigned; in-progress jobs will have a new task
that calls ``handleInterrupt``; callback jobs will resume their callback; and
completed jobs will be moved to the completed collection.

    >>> alt_agent = alt_da['main'] = StubAgent()
    >>> alt_agent.size = 4
    >>> alt_da.activate()
    >>> jobA = queue.put(mock_work)
    >>> jobB = queue.put(mock_work)
    >>> jobC = queue.put(mock_work)
    >>> jobD = queue.put(mock_work)
    >>> jobE = queue.put(mock_work)
    >>> jobA is alt_agent.claimJob()
    True
    >>> jobB is alt_agent.claimJob()
    True
    >>> jobC is alt_agent.claimJob()
    True
    >>> jobD is alt_agent.claimJob()
    True
    >>> print alt_agent.claimJob()
    None
    >>> len(alt_agent)
    4
    >>> len(queue)
    1
    >>> jobB._status_id = 1 # ACTIVE
    >>> jobC._status_id = 2 # CALLBACKS
    >>> jobD()
    42
    >>> jobD.status == zc.async.interfaces.COMPLETED
    True
    >>> queue.dispatchers.ping(alt_UUID)
    >>> zc.async.testing.set_now(
    ...     datetime.datetime.now(pytz.UTC) +
    ...     alt_da.ping_death_interval + datetime.timedelta(seconds=1))
    >>> alt_da.dead
    True
    >>> queue.dispatchers.ping(UUID)
    >>> bool(alt_da.activated)
    False
    >>> len(alt_agent)
    0
    >>> len(queue)
    4
    >>> queue[1] is jobA
    True
    >>> queue[2].callable == jobB.handleInterrupt
    True
    >>> queue[3].callable == jobC.resumeCallbacks
    True
    >>> alt_agent.completed.first() is jobD
    True

Activating a dispatcher also cleans out active jobs from its agents. Doing
it on activation is a fail-safe: normally, agents should not have jobs in
them after deactivation.

    >>> alt_agent.claimJob() is jobE
    True
    >>> len(alt_agent)
    1
    >>> alt_da.activate()
    >>> len(alt_agent)
    0
    >>> queue[0] is jobA
    True
    >>> queue[1].callable == jobB.handleInterrupt
    True
    >>> queue[2].callable == jobC.resumeCallbacks
    True
    >>> queue[3] is jobE
    True

As an aside, notice that the ``handleInterrupt`` and ``resumeCallbacks`` jobs
have custom error log levels, and custom retry policies.

    >>> import logging
    >>> queue[1].failure_log_level == logging.CRITICAL
    True
    >>> queue[2].failure_log_level == logging.CRITICAL
    True
    >>> queue[1].retry_policy_factory is zc.async.job.RetryCommonForever
    True
    >>> queue[2].retry_policy_factory is zc.async.job.RetryCommonForever
    True

If you have multiple workers, it is strongly suggested that you get the
associated servers connected to a shared time server.

Rescheduled Jobs
----------------

When a job is interrupted--it was active when it stopped--it asks its retry
policy whether to retry.  The policy can respond in one of three ways:

- yes, please retry right now;

- yes, please retry later; or

- no, please do not retry, and fail instead.

All three of these possibilities potentially affect the queue in different
ways, so we will look at examples.

First, let's look at the simple cases for all three.

The scenario above lets us look at what is probably the most common case: the
default retry policy.  Let's claim the ``handleInterrupt`` task.

    >>> len(queue)
    4
    >>> j = alt_agent.claimJob(
    ...     filter=lambda candidate:
    ...            candidate.callable.__name__ == 'handleInterrupt')
    ...
    >>> j.callable == jobB.handleInterrupt
    True
    >>> len(queue)
    3

The default retry policy's behavior is to retry nine times, for a total of ten
attempts, and to request that the original job be retried as soon as
possible--first in line. That behavior will become more interesting when we
look at interaction with quotas below. But now, performing this job will push
jobB back in the queue--again, notably, in very first place in the queue.

    >>> j()
    >>> len(queue)
    4
    >>> queue[0] is jobB
    True
    >>> jobB is alt_agent.claimJob()
    True
    >>> jobB()
    42
    >>> len(queue)
    3

That was an example of the retry policy saying "please retry right now":
returning ``True``. Now let's look at the other two possible responses.

"please retry later": with a datetime.

    >>> import persistent
    >>> import datetime
    >>> class StubRetryPolicy(persistent.Persistent):
    ...     zope.interface.implements(zc.async.interfaces.IRetryPolicy)
    ...     _reply = datetime.datetime(3000, 1, 1)
    ...     def __init__(self, job):
    ...         self.parent = self.__parent__ = job
    ...     def jobError(self, failure, data_cache):
    ...         return self._reply
    ...     def commitError(self, failure, data_cache):
    ...         return self._reply
    ...     def interrupted(self):
    ...         return self._reply
    ...     def updateData(self, data_cache):
    ...         pass
    ...

    >>> j = zc.async.job.Job(mock_work)
    >>> j._status_id = 1 # ACTIVE; hack internals for test
    >>> j.retry_policy_factory = StubRetryPolicy
    >>> j_wrap = queue.put(j.handleInterrupt)
    >>> len(queue)
    4
    >>> j_wrap is alt_agent.claimJob(
    ...     filter=lambda candidate:
    ...            candidate.callable.__name__ == 'handleInterrupt')
    ...
    True
    >>> len(queue)
    3
    >>> j_wrap()
    >>> len(queue)
    4
    >>> queue[3] is j
    True
    >>> j.begin_after
    datetime.datetime(3000, 1, 1, 0, 0, tzinfo=<UTC>)

...with a timedelta.

    >>> j = zc.async.job.Job(mock_work)
    >>> j._status_id = 1 # ACTIVE; hack internals for test
    >>> StubRetryPolicy._reply = datetime.timedelta(hours=1)
    >>> j.retry_policy_factory = StubRetryPolicy
    >>> j_wrap = queue.put(j.handleInterrupt)
    >>> len(queue)
    5
    >>> j_wrap is alt_agent.claimJob(
    ...     filter=lambda candidate:
    ...            candidate.callable.__name__ == 'handleInterrupt')
    ...
    True
    >>> len(queue)
    4
    >>> j_wrap()
    >>> len(queue)
    5
    >>> queue[3] is j
    True
    >>> j.begin_after
    datetime.datetime(2006, 8, 10, 18, 32, 33, tzinfo=<UTC>)

"please do not retry": does not affect the queue, so we will not show it here.

The trickiest aspects occur with quotas.

"please retry now": with a quota.  In this case, the job should retain its
precise place in the quota if the quota has a limit of one, and should remain
in the quota if the quota has a greater limit.

    >>> quota = queue.quotas['content catalog']
    >>> quota[0]()
    42
    >>> quota.clean()
    >>> len(quota)
    0
    >>> j = queue.put(mock_work)
    >>> j.quota_names = ('content catalog',)
    >>> j2 = queue.put(mock_work)
    >>> j2.quota_names = ('content catalog',)
    >>> transaction.commit()
    >>> len(queue)
    7
    >>> j is alt_agent.claimJob(
    ...     filter=lambda candidate:
    ...            'content catalog' in candidate.quota_names)
    ...
    True
    >>> list(quota) == [j]
    True

    >>> j._status_id = 1 # ACTIVE; fake beginning to call
    >>> alt_agent.remove(j)
    >>> j_wrap = queue.put(j.handleInterrupt)

The ``j`` job still claims the space in the quota, so ``j2`` can't be
claimed.

    >>> print alt_agent.claimJob(
    ...     filter=lambda candidate:
    ...            'content catalog' in candidate.quota_names)
    ...
    None

Now we'll get the ``handleInterrupt`` task.  The ``j`` job still cannot be
removed from the quota.

    >>> j_wrap is alt_agent.claimJob(
    ...     filter=lambda candidate:
    ...            candidate.callable.__name__ == 'handleInterrupt')
    ...
    True
    >>> len(quota)
    1
    >>> quota.clean()
    >>> list(quota) == [j]
    True

Now we'll perform the task.  The ``j`` job returns to the queue in a
``PENDING`` status, and the quota still cannot be cleared.

    >>> j_wrap()
    >>> len(queue)
    7
    >>> queue[0] is j
    True
    >>> len(quota)
    1
    >>> quota.clean()
    >>> list(quota) == [j]
    True

The agent can claim job ``j`` again, however.

    >>> j is alt_agent.claimJob(
    ...     filter=lambda candidate:
    ...            'content catalog' in candidate.quota_names)
    ...
    True

Still cannot clear the quota.

    >>> len(quota)
    1
    >>> quota.clean()
    >>> list(quota) == [j]
    True

Once we perform j, the quota can be cleared, and j2 can be claimed, as usual.

    >>> j()
    42

    >>> j2 is alt_agent.claimJob(
    ...     filter=lambda candidate:
    ...            'content catalog' in candidate.quota_names)
    ...
    True

"please retry later": with a quota.  In this case, other jobs can take the
place of the original job in the quota.

    >>> j3 = queue.put(mock_work)
    >>> j3.quota_names = ('content catalog',)
    >>> transaction.commit()
    >>> j2.retry_policy_factory = StubRetryPolicy
    >>> list(quota) == [j2]
    True

    >>> j2._status_id = 1 # ACTIVE; fake beginning to call
    >>> alt_agent.remove(j2)
    >>> j2_wrap = queue.put(j2.handleInterrupt)
    >>> j2_wrap is alt_agent.claimJob(
    ...     filter=lambda candidate:
    ...            candidate.callable.__name__ == 'handleInterrupt')
    ...
    True
    >>> len(queue)
    6
    >>> j2_wrap()
    >>> len(queue)
    7
    >>> queue[-2] is j2
    True
    >>> j2.status == zc.async.interfaces.PENDING
    True

    >>> j3 is alt_agent.claimJob(
    ...     filter=lambda candidate:
    ...            'content catalog' in candidate.quota_names)
    ...
    True
    >>> j3()
    42

A retry policy can also request rescheduling from a jobError or a commitError,
although the default retry policies do not.

jobError: "please reschedule now"

    >>> def raiseAnError():
    ...     raise RuntimeError
    ...
    >>> StubRetryPolicy._reply = datetime.timedelta()
    >>> j = queue.put(raiseAnError, retry_policy_factory=StubRetryPolicy)
    >>> len(queue)
    7
    >>> j is alt_agent.claimJob(
    ...     filter=lambda candidate:
    ...            candidate.callable.__name__ == 'raiseAnError')
    ...
    True
    >>> len(queue)
    6
    >>> j() is j
    True
    >>> len(queue)
    7
    >>> queue.claim() is j
    True

jobError: "please reschedule later"

    >>> StubRetryPolicy._reply = datetime.timedelta(hours=1)
    >>> j = queue.put(raiseAnError, retry_policy_factory=StubRetryPolicy)
    >>> len(queue)
    7
    >>> j is alt_agent.claimJob(
    ...     filter=lambda candidate:
    ...            candidate.callable.__name__ == 'raiseAnError')
    ...
    True
    >>> len(queue)
    6
    >>> j() is j
    True
    >>> len(queue)
    7
    >>> queue[-2] is j
    True
    >>> j.begin_after
    datetime.datetime(2006, 8, 10, 18, 32, 33, tzinfo=<UTC>)

commitError: "please reschedule now"

    >>> error_flag = False
    >>> old_commit = transaction.TransactionManager.commit
    >>> def commit(self):
    ...     global error_flag
    ...     if error_flag:
    ...         error_flag = False
    ...         raise ValueError()
    ...     old_commit(self)
    ...
    >>> transaction.TransactionManager.commit = commit
    >>> def causeCommitError():
    ...     global error_flag
    ...     error_flag = True
    ...
    >>> StubRetryPolicy._reply = datetime.timedelta()
    >>> j = queue.put(causeCommitError, retry_policy_factory=StubRetryPolicy)
    >>> len(queue)
    8
    >>> j is alt_agent.claimJob(
    ...     filter=lambda candidate:
    ...            candidate.callable.__name__ == 'causeCommitError')
    ...
    True
    >>> len(queue)
    7
    >>> j() is j
    True
    >>> len(queue)
    8
    >>> queue.claim() is j
    True

commitError: "please reschedule later"

    >>> StubRetryPolicy._reply = datetime.timedelta(hours=1)
    >>> j = queue.put(causeCommitError, retry_policy_factory=StubRetryPolicy)
    >>> len(queue)
    8
    >>> j is alt_agent.claimJob(
    ...     filter=lambda candidate:
    ...            candidate.callable.__name__ == 'causeCommitError')
    ...
    True
    >>> len(queue)
    7
    >>> j() is j
    True
    >>> len(queue)
    8
    >>> queue[-2] is j
    True
    >>> j.begin_after
    datetime.datetime(2006, 8, 10, 18, 32, 33, tzinfo=<UTC>)


    >>> transaction.TransactionManager.commit = old_commit

=========
Footnotes
=========

.. [#setUp] We'll actually create the state that the text needs here.

    >>> from ZODB.tests.util import DB
    >>> db = DB()
    >>> conn = db.open()
    >>> root = conn.root()
    >>> import zc.async.configure
    >>> zc.async.configure.base()

.. [#queues_collection] The queues collection is a simple mapping that only
    allows queues to be inserted.

    >>> len(container)
    1
    >>> list(container.keys())
    ['']
    >>> list(container)
    ['']
    >>> list(container.values()) == [queue]
    True
    >>> list(container.items()) == [('', queue)]
    True
    >>> container.get('') is queue
    True
    >>> container.get(2) is None
    True
    >>> container[''] is queue
    True
    >>> container['foo']
    Traceback (most recent call last):
    ...
    KeyError: 'foo'

    >>> container['foo'] = None
    Traceback (most recent call last):
    ...
    ValueError: value must be IQueue

    >>> del container['']
    >>> len(container)
    0
    >>> list(container)
    []
    >>> list(container.keys())
    []
    >>> list(container.items())
    []
    >>> list(container.values())
    []
    >>> container.get('') is None
    True
    >>> queue.name is None
    True
    >>> queue.parent is None
    True

    >>> container[''] = queue

.. [#verify] Verify queue interface.

    >>> from zope.interface.verify import verifyObject
    >>> verifyObject(zc.async.interfaces.IQueue, queue)
    True

.. [#check_dispatchers_mapping]

    >>> len(queue.dispatchers)
    1
    >>> list(queue.dispatchers.keys()) == [UUID]
    True
    >>> list(queue.dispatchers) == [UUID]
    True
    >>> list(queue.dispatchers.values()) == [da]
    True
    >>> list(queue.dispatchers.items()) == [(UUID, da)]
    True
    >>> queue.dispatchers.get(UUID) is da
    True
    >>> queue.dispatchers.get(2) is None
    True
    >>> queue.dispatchers[UUID] is da
    True
    >>> queue.dispatchers[2]
    Traceback (most recent call last):
    ...
    KeyError: 2
