{ "info": { "author": "Philip J Grabner, Cadit Health Inc", "author_email": "oss@cadit.com", "bugtrack_url": null, "classifiers": [ "Development Status :: 5 - Production/Stable", "Environment :: Console", "Environment :: Web Environment", "Framework :: Pyramid", "Intended Audience :: Developers", "License :: OSI Approved :: MIT License", "License :: Public Domain", "Natural Language :: English", "Operating System :: OS Independent", "Programming Language :: Python", "Topic :: Internet", "Topic :: Internet :: WWW/HTTP", "Topic :: Internet :: WWW/HTTP :: WSGI", "Topic :: Software Development", "Topic :: Software Development :: Libraries :: Application Frameworks" ], "description": "=================\npyramid_scheduler\n=================\n\nThe ``pyramid-scheduler`` package is a pyramid plugin that allows\nasynchronous and deferred task scheduling and management. It uses\nAPScheduler_ for actual task management and Kombu_ for messaging\nbetween processes.\n\n**IMPORTANT**: the pyramid-scheduler is very new, but fully\nfunctional. The following features are considered \"beta\", and\nshould probably not be used in production:\n\n* Multiple queues. For now, just use a single queue (and thus you are\n limited to a single background process).\n\n* Intuitive sequence of job events. Currently, the events that can be\n listened to (from pyramid_scheduler.api.Event, such as JOB_CREATED,\n JOB_EXECUTED, JOB_CANCELED, and JOB_REMOVED) do not reliably fire in\n an intuitive order. For example, you may get a JOB_REMOVED event\n before the JOB_EXECUTED event for a deferred job.\n\n\nTL;DR\n=====\n\nInstall:\n\n.. code-block:: bash\n\n $ pip install pyramid-scheduler\n\nUse:\n\n.. code-block:: python\n\n # ini file settings:\n # [app:main]\n # scheduler.combined = true | false ## should execution be in-process?\n # scheduler.queues = jobs ## space-separated list of queues\n # scheduler.broker.url = %(dburl)s ## the URL used for kombu messaging\n # ## other optional settings:\n # ## scheduler.housekeeping\n # ## scheduler.housekeeping.append\n # ## scheduler.jobstore.default.class\n # ## scheduler.misfire_grace_time\n\n # enabling the plugin adds a `scheduler` attribute to the registry\n def main(global_config, **settings):\n # ... (the usual pyramid startup calls) ...\n config.include('pyramid_scheduler)\n\n # create an asynchronous task\n def slow_process(name, id):\n # ...a slow asynchronous job...\n def handle_request_quickly(request):\n request.registry.scheduler.add_async_job(slow_process, args=('my-first-arg', 2))\n\n # schedule a deferred task for one hour from now\n def delayed_process():\n # ...something that should happen later...\n def handle_request_now(request):\n import time\n request.registry.scheduler.add_date_job(delayed_process, time.time() + 3600)\n\n # do something every 10 minutes\n def interval_process(reason=None):\n # ...gets executed every 10 minutes with an optional reason...\n def handle_request_often(request):\n request.registry.scheduler.add_date_job(interval_process, minutes=10)\n\n\nConcepts\n========\n\nThe pyramid-scheduler package refers to asynchronous or deferred tasks\nto be managed as \"jobs\" and these fall into the following categories:\n\n* Asynchronous jobs:\n\n These jobs are executed immediately, but asynchronously.\n\n* Deferred jobs, i.e. \"date\" jobs:\n\n Jobs that are scheduled to be performed at a specific time.\n\n* Interval jobs:\n\n Similar to deferred jobs, but that are then re-executed on an\n interval.\n\n* Cron jobs:\n\n Similar to interval jobs, but they use a scheduling definition\n that is similar to a unix \"cron\" definition.\n\nConceptually, there are two activities: the activity of creating the\njobs and the activity of executing the jobs. These can be performed by\nthe same process (with \"scheduler.combined=true\"), or they can be\nperformed by different processes (with \"scheduler.combined=false\").\n\nIf there are multiple processes that are creating jobs (for example,\nif your are running multiple servers or your WSGI configuration uses\nmultiple processes), then you **CANNOT** run pyramid-scheduler in\ncombined mode since then there will be multiple processes executing\nthe jobs leading to multiple executions.\n\nTypically, combined mode is used during development where a single\n`pserve` instance will be used. Then, in production mode, you will\nhave multiple servers and WSGI processes that generate jobs, that\nare then executed by a single background process (managed via the\n`pscheduler` daemon).\n\nPyramid-scheduler supports multiple \"queues\". The main reason to use\nseparate queues is that the `pscheduler` can be configured to only\nprocess jobs for specific queues, which means that multiple\npschedulers can work in parallel as long as they are listening to\ndifferent queues. (A later enhancement is planned to allow multiple\npschedulers to handle a single queue.)\n\n**IMPORTANT**: currently, the following limitations exist:\n\n* The callback handler provided as the first argument to the scheduler\n add_*_job() methods must be a normal module-level defined\n function. It *cannot* be a lambda function, an internal function, a\n method, or otherwise function that is not globally resolvable using\n standard dot-notation.\n\n* The `args` and `kwargs` parameters must all be completely\n pickle-able.\n\n* Deferred jobs that are scheduled to occur further in the past than\n `misfire_grace_time` will be silently dropped.\n\n* Jobs that take a `date` or `start_date` parameter can specify those\n values either as an epoch int or float or as a datetime object. If\n a datetime is provided, it **must** be timezone \"naive\" (see the\n documentation of datetime_).\n\nUnder the hood, pyramid-scheduler uses APScheduler to do the actual\nprocessing and scheduling. For messaging between the job creators and\nthe pscheduler background process, it uses Kombu messaging, which\nsupports a variety of transports including Redis and SQLAlchemy. This\npackage was developed as an alternative to celery, due to severe\nlimitations found in the celery API and shortcomings in the actual\nimplementation.\n\n\nInstallation\n============\n\nYou can manually install it by running:\n\n.. code-block:: bash\n\n $ pip install pyramid-scheduler\n\nHowever, a better approach is to use standard python distribution\nutilities, and add pyramid-scheduler as a dependency in your project's\n`install_requires` parameter in your ``setup.py``. Then run a ``python\nsetup.py develop``.\n\nThen, enable the package either in your INI file via:\n\n.. code-block:: text\n\n pyramid.includes = pyramid_scheduler\n\nor in code in your package's application initialization via:\n\n.. code-block:: python\n\n def main(global_config, **settings):\n # ...\n config.include('pyramid_scheduler')\n # ...\n\n\nConfiguration\n=============\n\nThe following configuration options (placed in the \"[app:main]\"\nsection of your INI file):\n\nTODO: add documentation\n\n* scheduler.combined\n* scheduler.queues\n* scheduler.delegator\n* scheduler.broker.url\n* scheduler.broker.serializer\n* scheduler.broker.compressor\n* scheduler.housekeeping\n* scheduler.housekeeping.append\n* scheduler.jobstore.default.class\n* scheduler.misfire_grace_time\n\n\nDebugging\n=========\n\nThe first step in debugging a pyramid-scheduler instance is to elevate\nthe logging, and that is easiest via the application\nconfiguration. Here, an example that increases logging to DEBUG level\nand sends the logs to STDERR:\n\n.. code-block:: ini\n\n [loggers]\n keys = scheduler, ...\n\n [handlers]\n keys = console, ...\n\n [formatters]\n keys = generic, ...\n\n [logger_scheduler]\n level = DEBUG\n handlers = console\n qualname = pyramid_scheduler\n\n [handler_console]\n class = StreamHandler\n args = (sys.stderr,)\n level = NOTSET\n formatter = generic\n\n [formatter_generic]\n format = %(levelname)-5.5s [%(name)s] %(message)s\n\n\nIf that does not expose the source of the problem, you can take some\nof the following actions:\n\nConfirm Communication\n---------------------\n\nYou can confirm that the task producers and consumers are\ncommunicating by sending a ``print-jobs`` message. First, check\nthe configurations by sending the message from a fake producer\nby using the ``pscheduler --message`` feature as follows:\n\n.. code-block:: bash\n\n $ pscheduler --message 'print-jobs' {CONFIG}.ini\n\n DEBUG [pyramid_scheduler.pscheduler] loading application from \"{CONFIG}.ini\"\n DEBUG [pyramid_scheduler.broker] sending message to messenger\n\nand you should see something like this in the pscheduler daemon logs\n(depending on what happens to STDOUT, you may only see the DEBUG\nmessages, not the actual Jobstore messages):\n\n.. code-block:: text\n\n DEBUG [pyramid_scheduler.broker] received message: \n DEBUG [pyramid_scheduler.scheduler] received message event: print-jobs\n Jobstore default:\n pyramid_scheduler_wrapper (trigger: cron[hour='0', minute='5'], next run at: 2014-12-03 00:05:00)\n Jobstore internal.transient.8524480f-26b4-4a69-8bcd-3bb180d0cf9e:\n housekeeping (trigger: interval[1 day, 0:00:00], next run at: 2014-12-03 14:45:26.696818)\n\nIf that does not work, you need to check the application\nconfigurations, both on the consumer and producer sides. You may also\nneed to debug the Kombu_ sub-system.\n\nTODO: add documentation\n\n\nTask Execution Process\n======================\n\nThere are several ways that the tasks in a queue can actually be\nexecuted. The preferred way, described here, is to use the\n`pscheduler` script which is intended to be run in daemon mode by a\ndaemon-running service, such as DJB's daemontools_ package.\n\nInstall daemontools (adjust for your package manager):\n\n.. code-block:: bash\n\n $ apt-get install daemontools\n\nThis should install & start the `svscan` monitoring against the\n``/etc/service`` directory. You can do a `ps` to confirm this, and if\nit is not running, read the daemontools docs. If it is scanning a\ndirectory other than ``/etc/service``, adjust the following examples\nappropriately.\n\nCreate & configure the logging subsystem by creating the following\nfile in ``/etc/service/pscheduler/log/run`` (where ``pscheduler`` can\nbe whatever you want). This example will store up to 100MiB of logs\nin the ``/var/log/pscheduler`` directory:\n\n.. code-block:: text\n\n #!/bin/sh\n exec multilog t s10485760 n10 /var/log/pscheduler\n\nCreate & configure the `pscheduler` service by creating the following\nfile in ``/etc/service/pscheduler/run`` (where ``pscheduler`` can be\nwhatever you want). This example will run `pscheduler` as the\n``www-data`` user (it is simplest if it runs as the same user as the\nappserver that is producing pyramid-scheduler tasks):\n\n.. code-block:: text\n\n #!/bin/sh\n exec env -i PATH=\"/bin:/usr/bin:$PATH\" \\\n setuidgid www-data \\\n /path/to/virtualenv/bin/pscheduler \\\n /path/to/config.ini \\\n 2>&1\n\nAnd ensure that both files are executable:\n\n.. code-block:: bash\n\n $ chmod 755 /etc/service/pscheduler/log/run\n $ chmod 755 /etc/service/pscheduler/run\n\n\n.. _APScheduler: https://pypi.python.org/pypi/APScheduler\n.. _Kombu: https://pypi.python.org/pypi/kombu\n.. _datetime: http://docs.python.org/2/library/datetime.html\n.. _daemontools: http://cr.yp.to/daemontools.html\n", "description_content_type": "", "docs_url": null, "download_url": "", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "http://github.com/cadithealth/pyramid_scheduler", "keywords": "web wsgi pyramid asynchronous task scheduling management scheduler", "license": "MIT (http://opensource.org/licenses/MIT)", "maintainer": "", "maintainer_email": "", "name": "pyramid_scheduler", "package_url": "https://pypi.org/project/pyramid_scheduler/", "platform": "", "project_url": "https://pypi.org/project/pyramid_scheduler/", "project_urls": { "Homepage": "http://github.com/cadithealth/pyramid_scheduler" }, "release_url": "https://pypi.org/project/pyramid_scheduler/0.3.4/", "requires_dist": null, "requires_python": "", "summary": "A pyramid plugin that allows asynchronous and deferred task scheduling and management.", "version": "0.3.4" }, "last_serial": 4901242, "releases": { "0.1.0": [], "0.2.0": [ { "comment_text": "", "digests": { "md5": "66dc9fbdba378d1839fe5caf1cc398f9", "sha256": "36aa035f4932cf469e15fa5bdeef9ebe0c30366c1a5940cad598c4025d2444e1" }, "downloads": -1, "filename": "pyramid_scheduler-0.2.0.tar.gz", "has_sig": false, "md5_digest": "66dc9fbdba378d1839fe5caf1cc398f9", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 14084, "upload_time": "2013-05-31T17:33:37", "url": "https://files.pythonhosted.org/packages/a0/43/3f638c00e51c776919ffe000369f3a8face7d89b3ea508375e8b39686013/pyramid_scheduler-0.2.0.tar.gz" } ], "0.2.1": [ { "comment_text": "", "digests": { "md5": "89dece86abda182ecb570436f1b8993a", "sha256": "2f5b32b5ff0599a68796b570b43f1d4979445e62df423287c7f04504a32cdbb7" }, "downloads": -1, "filename": "pyramid_scheduler-0.2.1.tar.gz", "has_sig": false, "md5_digest": "89dece86abda182ecb570436f1b8993a", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 14071, "upload_time": "2013-05-31T17:36:57", "url": "https://files.pythonhosted.org/packages/0d/c3/9a3b3a03d8db01ef27d5351f8697a7bcaf2f38f3e21c3b9eeecffa4193e1/pyramid_scheduler-0.2.1.tar.gz" } ], "0.2.2": [ { "comment_text": "", "digests": { "md5": "988ac129da3d3533fdd79eef4ff48186", "sha256": "19cc9987a5743bad1f6f2cee47ff8207b1026c39d53f729840e986f773234baf" }, "downloads": -1, "filename": "pyramid_scheduler-0.2.2.tar.gz", "has_sig": false, "md5_digest": "988ac129da3d3533fdd79eef4ff48186", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 14145, "upload_time": "2013-05-31T17:49:11", "url": "https://files.pythonhosted.org/packages/cd/29/e430add828ecd593a71b28bd704612e0be33018ecb4b4892427278084192/pyramid_scheduler-0.2.2.tar.gz" } ], "0.2.3": [ { "comment_text": "", "digests": { "md5": "4a84a455d0815a40b30a4cfaf716e0b4", "sha256": "c10022de3acc539eebe2884ff152540023c8bcf5daa0219f5338ed7665717508" }, "downloads": -1, "filename": "pyramid_scheduler-0.2.3.tar.gz", "has_sig": false, "md5_digest": "4a84a455d0815a40b30a4cfaf716e0b4", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 14187, "upload_time": "2013-08-19T20:10:33", "url": "https://files.pythonhosted.org/packages/1f/8d/5236fc4eece2ca825de9343ed3de207aa0f0ea5ef58a3ba3e6a89ede1d9a/pyramid_scheduler-0.2.3.tar.gz" } ], "0.2.4": [ { "comment_text": "", "digests": { "md5": "38b2bf13cf871e9da93038e1dcc66d8f", "sha256": "f69f8c8ab4ab2c08ee2a8041143cc0d9ef82724600aeeb6a7209f1e6ddb59591" }, "downloads": -1, "filename": "pyramid_scheduler-0.2.4.tar.gz", "has_sig": false, "md5_digest": "38b2bf13cf871e9da93038e1dcc66d8f", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 15742, "upload_time": "2013-11-06T19:24:19", "url": "https://files.pythonhosted.org/packages/30/6f/d19f2ab9d011083f79c1a968511571a997b4047a2bd0a188b0d338020bbc/pyramid_scheduler-0.2.4.tar.gz" } ], "0.2.5": [ { "comment_text": "", "digests": { "md5": "19a93c8f20634f146c76f2e40e9d383b", "sha256": "3245154657cbc0ea42a699173c11a9950e6d9ae086471bb809d6b50389a0f148" }, "downloads": -1, "filename": "pyramid_scheduler-0.2.5.tar.gz", "has_sig": false, "md5_digest": "19a93c8f20634f146c76f2e40e9d383b", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 17306, "upload_time": "2013-12-18T23:46:55", "url": "https://files.pythonhosted.org/packages/bf/5b/6faab560f9efc59c296e8efea7222b5d67cfda17a786a0cddbbb7d3cd8a9/pyramid_scheduler-0.2.5.tar.gz" } ], "0.3.0": [ { "comment_text": "", "digests": { "md5": "cac50fa6bc660b492ef448fc9dfe8ab7", "sha256": "e9a0d4161cec21bbc5d10ed65ee769ed421cf887e36bf4d9f6543bceb959c7dd" }, "downloads": -1, "filename": "pyramid_scheduler-0.3.0.tar.gz", "has_sig": false, "md5_digest": "cac50fa6bc660b492ef448fc9dfe8ab7", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 17658, "upload_time": "2014-04-12T16:43:11", "url": "https://files.pythonhosted.org/packages/0b/06/41da947d57690b2e589dc5d25fb5c2f689b33b7fcfc03693528a1d00b722/pyramid_scheduler-0.3.0.tar.gz" } ], "0.3.1": [ { "comment_text": "", "digests": { "md5": "5f2c2b6094b45882bb8c0d1279797d5a", "sha256": "72e9c1356aa3951265ce0e9091a1fab3e643506c35e8103be4de5a2a2f2239a7" }, "downloads": -1, "filename": "pyramid_scheduler-0.3.1.tar.gz", "has_sig": false, "md5_digest": "5f2c2b6094b45882bb8c0d1279797d5a", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 17636, "upload_time": "2014-09-03T17:13:32", "url": "https://files.pythonhosted.org/packages/d4/74/8ad474c3dfb429285c39ffc5d4c2abbc267356387e95d25877aefc7d75e8/pyramid_scheduler-0.3.1.tar.gz" } ], "0.3.2": [ { "comment_text": "", "digests": { "md5": "6576688550757b54fc7f96a860b9272f", "sha256": "0cad422e481b376ece35032784a4f9d88fcedc9090a4ed670187577031b84c21" }, "downloads": -1, "filename": "pyramid_scheduler-0.3.2.tar.gz", "has_sig": false, "md5_digest": "6576688550757b54fc7f96a860b9272f", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 20281, "upload_time": "2015-10-07T19:22:13", "url": "https://files.pythonhosted.org/packages/e9/2b/fc96a0510bcdf63cae10968bfefe513ff89536c62f8d6b163847bd0d3bda/pyramid_scheduler-0.3.2.tar.gz" } ], "0.3.3": [ { "comment_text": "", "digests": { "md5": "f5d9b5c56dc0456f06ee833097fb19ee", "sha256": "1dba90928d11d3bfdb594866c0803fdf0198d84cc5cfccc94c075f87969c41f2" }, "downloads": -1, "filename": "pyramid_scheduler-0.3.3.tar.gz", "has_sig": false, "md5_digest": "f5d9b5c56dc0456f06ee833097fb19ee", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 22045, "upload_time": "2016-12-13T18:19:30", "url": "https://files.pythonhosted.org/packages/98/7f/6907eb509c418e9407a226c72ef00877c81f8488755d87aa24fba20ecefe/pyramid_scheduler-0.3.3.tar.gz" } ], "0.3.4": [ { "comment_text": "", "digests": { "md5": "60dbd9548c9afdf68e7079d516320adc", "sha256": "fd99f4f938c6747c011b257e6c401472cd781293161b2d24b10e51457415de6d" }, "downloads": -1, "filename": "pyramid_scheduler-0.3.4.tar.gz", "has_sig": false, "md5_digest": "60dbd9548c9afdf68e7079d516320adc", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 22091, "upload_time": "2019-03-05T18:41:16", "url": "https://files.pythonhosted.org/packages/cd/a1/5281aa6a44d5a627aca80e51371718d0b604ceee07b590f476a4429c5644/pyramid_scheduler-0.3.4.tar.gz" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "60dbd9548c9afdf68e7079d516320adc", "sha256": "fd99f4f938c6747c011b257e6c401472cd781293161b2d24b10e51457415de6d" }, "downloads": -1, "filename": "pyramid_scheduler-0.3.4.tar.gz", "has_sig": false, "md5_digest": "60dbd9548c9afdf68e7079d516320adc", "packagetype": "sdist", "python_version": "source", "requires_python": null, "size": 22091, "upload_time": "2019-03-05T18:41:16", "url": "https://files.pythonhosted.org/packages/cd/a1/5281aa6a44d5a627aca80e51371718d0b604ceee07b590f476a4429c5644/pyramid_scheduler-0.3.4.tar.gz" } ] }