{ "info": { "author": "Timur Faradzhov", "author_email": "timurfaradzhov@gmail.com", "bugtrack_url": null, "classifiers": [ "License :: OSI Approved :: MIT License", "Operating System :: OS Independent", "Programming Language :: Python :: 3" ], "description": "# runner\nThe aim of this project is to create simple and effective instrument for automatic job scheduling and management in Python.\n\nMain *runner* features:\n* Minimal and user friendly command line interface.\n* Cross platform - application can be used in any OS with Python 3.\n* Opportunity to handle all jobs in one place instead of using many (Oracle Enterprise Scheduler, Linux Crontab and etc.).\n* Generate built-in scheduler.\n* Generate built-in job.\n* Edit generated built-in jobs.\n* Delete generated built-in jobs.\n* Configuration with ini files.\n* Automatic job scheduling and execution at certain time, during certain period or each time cycle.\n* Manual job execution for now or for certain date and time.\n* Scheduling with simple tsv file.\n* Run jobs using virtual Python environment.\n* Run jobs implemented in non Python technologies like C++, Java, Bash, Perl, etc.\n* Built-in logging.\n* Built-in notifications and alarms.\n\nMain *runner* targets:\n* Add some CLI commands.\n* Attach database to *runner* as option for schedule and new data streams storage.\n* Add simple web GUI.\n* Improve notifications' mechanism and how they looks.\n* Optimizations.\n\n## Getting Started\n### Requirements\nOperation systems: Windows, Linux, Mac OS.\n\nPython version: 3.7.1.\n\nPython modules dependencies (use latest releases): [logbook](https://github.com/t3eHawk/logbook), [tables](https://github.com/t3eHawk/tables), [notifier](https://github.com/t3eHawk/notifier)\n\n### Installation\nTo install just download the latest *[runner](https://github.com/t3eHawk/runner/releases)* release and copy /src/runner folder to your/python/folder/Lib/site-packages/.\n\n## How to Use\nTo start working with *runner* you need just one simple file containing *runner.Manager* instance.\n\nLet's go to the root of your disk *C:\\* (or any other location, it does not matter) and create the folder *runner*.\nMove to that folder and create new file named *manager.py* with next few string of code.\n```\n> C:\\runner\\manager.py\n\nimport runner\n\nmanager = runner.Manager()\n```\nNow *manager.py* is your main *runner* interface and the way to deploy the application in *C:\\runner* folder.\n\nExecute the *manager.py*. You will receive the main help note that can also be called with the *manager.py help*.\n\nWhen you execute *manager.py* first time it generates main configuration file *config.ini* in the same folder.\n```\n$ python manager.py\nINFO: File C:\\runner\\config.ini created.\n\nList of available commands:\n\ncreate scheduler Generate a scheduler with all elements.\ncreate job Generate a job with all elements.\nlist jobs Show all jobs registered in the schedule.\nedit schedule Open the schedule file in the editor.\n\nedit job Open the job script in the editor.\nrun job Execute the job by id with or without run time.\nrun jobs Execute the jobs listed in the file.\ndelete job Delete the job by id.\n\nedit config Open one of the configuration files in the editor.\n\nhelp Show this message.\n\nFor more details type [command] help. \n```\nYou could get more specific help for each command:\n```\n$ python manager.py run job help\n\nExecute the job by id with or without run time.\nParameters:\nid integer Id of the job you want to run.\ntrigger yyyy-mm-dd/hh24:mi:ss Date with or without time in ISO format\n yyyy-mm-dd for what you want to run the job.\n```\nWe recommend to repeat that action for each command for more understanding.\n\n### Create Scheduler\nType command *create scheduler* to generate the scheduler.\nYou will be asked about necessary attributes and then scheduler will be generated in the folder.\n```\n$ python manager.py create scheduler\n\nPlease follow the steps to create the scheduler.\nEnter the name or leave empty to use default:\n\nEnter the description or leave empty to use default:\n\n********************************************************************************\nINFO: Creating scheduler scheduler...\nINFO: File C:\\runner\\scheduler.py created.\nINFO: File C:\\runner\\config.ini updated.\nINFO: File C:\\runner\\schedule.tsv created.\nINFO: Folder C:\\runner\\jobs created.\n```\nAbout files:\n\n*C:\\runner\\scheudler.py* - scheduler instance and starter.\n\n*C:\\runner\\config.ini* - main configuration file.\n\n*C:\\runner\\schedule.tsv* - job schedule.\n\n*C:\\runner\\jobs* - folder for future jobs.\n\n### Start Scheduler\nTo run the scheduler just execute file *scheduler.py*.\n```\n$ python scheduler.py\n```\nNow the scheduler is working.\nLogs can be found in the folder *C:\\runner\\logs*.\n\nTo stop the scheduler process just type the CTRL + C.\n\n### Create Job\nScheduler is ready and jobs can be created.\n\nType command *create job* to generate the job.\nYou will be asked about necessary attributes and then job will be generated.\n```\n$ python manager.py create job\n\nFollow the instructions to create the job.\nInputs with * are mandatory.\nEnter the name or leave empty to use default:\ntest\nEnter the short description:\nTest Job\nEnter the environment or leave empty to use Python:\n\nEnter the month day (1-31):\n\nEnter the week day (1-7):\n\nEnter the hour (0-23):\n8\nEnter the minute (0-59):\n0\nEnter the second (0-59):\n0\nActivate job (Y/N)? *:\nN\n********************************************************************************\nINFO: Creating job...\nINFO: Job ID: 0\nINFO: Folder C:\\runner\\jobs\\0 created.\nINFO: File C:\\runner\\jobs\\0\\job.py created.\nINFO: File C:\\runner\\jobs\\0\\config.ini created.\nINFO: File C:\\runner\\jobs\\0\\script.py created.\nINFO: Job test successfully added to schedule!\n```\nAbout files:\n\n*C:\\runner\\jobs\\0* - this is a job folder which corresponds to job id.\n\n*C:\\runner\\jobs\\0\\job.py* - job instance and starter.\n\n*C:\\runner\\jobs\\0\\config.ini* - job configuration file.\n\n*C:\\runner\\jobs\\0\\script.py* - script that must be executed by job.\n\nIn *schedule.tsv* new record for created job will be provided.\n\nNote that we used status *N*.\nJobs with such status will be not executed until status changed to Y.\nIt is done to prevent accidental job run before it is not finalized.\n\n### Scheduling\nSchedule is one of the most important part of the application.\n\nAll jobs scheduled in *schedule.tsv*.\nThe file is scanned by *runner* and its formatting is quite critical.\n\nMain formatting requirements are:\n* Each line is a table row.\n* Column separator is only a tabulation.\n* Last line must be always empty.\n\nWhen *schedule.tsv* is modified *runner* catches the changes, makes necessary updates and also prints message to the log about that.\n\nTo see all scheduled jobs use command *list jobs*:\n```\n$ python manager.py list jobs\n\n-----------------------------------------------------------------------------------------------------------------------------\n|ID |NAME |DESCRIPTION |ENVIRONMENT |FILE |MONTH_DAY |WEEK_DAY |HOUR |MINUTE |SECOND |PARAMETERS |STATUS |\n-----------------------------------------------------------------------------------------------------------------------------\n|0 |test |Test Job |python |C:\\runner\\jobs\\0\\job.py |* |* |8 |0 |0 | |N |\n-----------------------------------------------------------------------------------------------------------------------------\n```\nTo see only active jobs add *active*.\n```\n$ python manager.py list jobs active\n```\n\nAlso you could either open file directly or use command *manager.py edit schedule* to open file using chosen editor in the main config.\n\n#### Schedule Description\n\n|Field name |Description |\n|------------|-----------------------------------------------------------------------------------------------------------------------|\n|ID |Unique id that job receives automatically during creation. |\n|NAME |Name of job. |\n|DESCRIPTION |Description of job. |\n|ENVIRONMENT |Name of environment in which job must be executed. All used in *runner* environments must be listed in the main config.|\n|FILE |Entry file. If you use built-in jobs then it is a *job.py*. In other case it is up to you how job should be started. |\n|MONTH_DAY |Day of month when job must be executed. |\n|WEEK_DAY |Day of week when job must be executed. |\n|HOUR |Hour of day when job must be executed. |\n|MINUTE |Minute of our when job must be executed. |\n|SECOND |Second of minute when job must be executed. |\n|PARAMETERS |Additional arguments that will be passed to FILE in the time of execution. |\n|STATUS |Status of job. Y is an active, N is an inactive. |\n\nJob frequency in *runner* is rather flexible.\nBe attentive because some combinations can force *runner* to execute job every second!\n\nYou could schedule jobs by manipulating with time fields next few ways:\n1. Execute job at certain date and time. Use simple integers.\n1. Execute job every time cycle. Use integers with symbol */* to point that it is a cycle value.\n1. Execute job only during certain period. Use integers with *,* and *-* to point that it is a period value.\n\nFor more understanding we list the most popular combinations:\n\n|MONTH_DAY |WEEK_DAY |HOUR |MINUTE |SECOND |When to Execute |\n|----------|---------|-----|-------|-------|---------------------------------------|\n|1 |* |8 |0 |0 |Each first day of month at 8 am. |\n|* |1 |8 |0 |0 |Each first day of week at 8 am. |\n|* |* |8 |0 |0 |Each day at 8 am. |\n|* |* |/1 |0 |0 |Each hour. |\n|* |* |* |/5 |0 |Each five minutes. |\n|* |1-5 |8 |0 |0 |Each day during weekdays at 8 am. |\n|1,11,21 |* |8 |0 |0 |Each 1, 11 and 21 day of month at 8 am.|\n\n#### More About Scheduling\nAutomatic scheduling in *runner* is based on moments.\nMoment - is a dynamic scheduler attribute that describes current timestamp as a count of seconds past from era begin.\n\nYou can log each scheduler moment if you need to.\nFor this set the *showtime* option in the *LOG* section of the main config to *True*.\nThen in log each moment will be recorded as an empty message:\n```\n2018-12-31 23:59:59|INFO|\n```\n\nMoment includes an active and a passive phases.\n\nActive phase stands for all necessary actions for scheduling and internal *runner* needs.\nAfter active phase finished passive phase begins during which *runner* sleeps till the next step.\n\nIf active phase longs more than 1 second in case of some trouble then scheduler internal time will differ from real one.\nWhen this happened *runner* will automatically synchronize moment with real time.\nThat will be registered in the log that may help in troubleshooting:\n```\n2018-12-31 23:59:59|WARNING|TIME IS BROKEN!\n2018-12-31 23:59:59|INFO|SYNCHRONIZING THE TIME...\n2018-12-31 23:59:59|INFO|SUCCESS\n```\n\nIf you want to log each active phase time consumption then set the *showdelay* option of the *LOG* section in the main config to *True*:\n\n### Tasking\nA task is a set of actions that a job must perform.\nAll tasks must be described in a special job script file.\nFor our test job this is the file *jobs\\0\\script.py*.\nThat file already contains *job* instance so it can be used in the script too.\n\nOpen script file or use special command *edit job* that will open it using chosen editor in main config.\n```\n$ python manager.py edit job 0\n```\nModify file as in the example below:\n```\n> C:\\runner\\jobs\\0\\job.py\n\nfrom job import job\n\n# Write the code to be executed by job down below.\nprint('Wake up, Neo...')\n```\nNow go to *schedule.tsv* and change *STATUS* for modified job from *N* to *Y*.\n\nJob is ready and at next 08:00:00 it will be executed.\n\n### Run Job\nJob can be executed automatically by scheduler or manually by user.\n\nFor automatic execution in *schedule.tsv* must be determined:\n1. *ENVIRONMENT* that is a name of environment described in the main config.\n1. *MONTH_DAY*, *WEEK_DAY*, *HOUR*, *MINUTE*, *SECOND* - time of execution.\n\nUser execution can be done by two ways.\n\nThe first one is to use special command *run job* with mandatory arguments *id* and optional argument *trigger*.\n\nArgument *id* is an integer unique identification number of job that can be found in *schedule.tsv*.\n\nArgument *trigger* is a date or datetime in ISO timestamp format for which job must be executed.\n\nIn the example below we execute the job with id 0 for 1th January 2019, 8 AM.\n```\n$ python manager.py run job 0 2019-01-01/08:00:00\nINFO: REQUEST TO RUN THE JOB.\nINFO: ID: 0.\nINFO: NAME: test.\nINFO: DESCRIPTION: Test Job.\nINFO: TRIGGER: 2019-01-01/08:00:00.\nARE YOU SURE Y/N?\nY\nWake up, Neo...\nINFO: DONE.\n```\n\nSecond one is to execute *job.py* file of the job you want to run.\nFollowing command will execute the job with id 0 for current moment:\n```\n$ python jobs\\0\\job.py\nWake up, Neo...\n```\nOptionally you could use argument -t/--trigger to pass certain date and time.\nSo execution of job with id 0 for 1th January 2019, 8 AM will looks like:\n```\n$ python jobs\\0\\job.py -r 2019-01-01/08:00:00\nWake up, Neo...\n```\n\nAlso if you have many jobs that must be executed it is possible to use special file.\nLet's create file *C:\\runner\\run_jobs.txt* with listed ids and triggers:\n```\n0 2018-12-31/08:00:00\n0 2019-01-01/08:00:00\n```\nNote that between id and trigger there is a space.\n\nExecute the command:\n```\n$ python manager.py run jobs C:\\runner\\run_jobs.txt\nINFO: REQUEST TO RUN THE JOB.\nINFO: ID: 0.\nINFO: NAME: test.\nINFO: DESCRIPTION: Test Job.\nINFO: TRIGGER: 2018-12-31/08:00:00.\nARE YOU SURE Y/N?\nY\nWake up, Neo...\nINFO: DONE.\n********************************************************************************\nINFO: REQUEST TO RUN THE JOB.\nINFO: ID: 0.\nINFO: NAME: test.\nINFO: DESCRIPTION: Test Job.\nINFO: TRIGGER: 2019-01-01/08:00:00.\nARE YOU SURE Y/N?\nY\nWake up, Neo...\nINFO: DONE.\n```\n\n### Logging\nLogging in *runner* implemented on [logbook](https://github.com/t3eHawk/logbook) and available through the *log* object.\nSo you could even use *log* in your *script.py*.\n```\n> C:\\runner\\jobs\\0\\script.py\n\nfrom job import job\n\n# Write the code to be executed by job down below.\nprint('Wake up, Neo...')\njob.log.info('Hello there.')\n```\nLet's imagine that today is 31th December 2018, 23:59:59.\nRun the job and in log file you will see:\n```\n> C:\\runner\\jobs\\0\\logs\\test_20181231235959.log\n\n********************************************************************************\n* *\n* APP: test *\n* DESC: Test Job *\n* VERSION: None *\n* TIMESTAMP: 2018-12-31 23:59:59 *\n* FILE: C:\\runner\\jobs\\0\\logs\\test_20181231235959.log *\n* MACHINE: AMD64 *\n* PROCESSOR: Intel64 Family 6 Model 78 Stepping 3, GenuineIntel *\n* HOSTNAME: Falcon *\n* USER: Kenobi *\n* SYSTEM_NM: Windows *\n* SYSTEM_RLS: 10 *\n* SYSTEM_VER: 10.0.14393 *\n* PYTHON: 3.7.1 *\n* ID: 0 *\n* TRIGGER: 2018-12-31 23:59:59 *\n* CONFIGS: C:\\runner\\jobs\\0\\config.ini *\n* PERSONS: None *\n* *\n********************************************************************************\n2018-12-31 23:59:59|INFO|JOB STARTED.\n2018-12-31 23:59:59|INFO|Hello there.\n2018-12-31 23:59:59|INFO|JOB FINISHED.\n2018-12-31 23:59:59|INFO|TIME SPENT: 0 seconds.\n```\nVisit [logbook](https://github.com/t3eHawk/logbook) page to see more features of it.\n\n### Notifications and Alarms\nSometimes in work you may need to get feedback from the job.\nIt can be easily done with a help of *email* object which allows to write messages on the email.\n\nTo use notifier you need SMTP server.\nConnection must be defined in config: *ip*, *port*, *user*, *password*, *email address* options of the *[EMAIL]* section in the job config (or other external config).\n\nTo send simple text message on the email:\n```\n> C:\\runner\\jobs\\0\\script.py\n\nsubject = 'Help me'\ntext = 'You are my only hope'\nrecipient = 'ObiWanKenobi@email.com'\njob.email.notify(subject, text, recipient)\n```\n\nAnother case of using notifier is to inform that something went wrong in the job execution.\nWe use alarms for this purposes in *runner*.\nUse *alarm()* to send standard alarm notification on emails listed in the *persons* option of the *[GENERAL]* section in the job config.\n\nLet's change our job with id 0 little bit and execute it.\n```\n> C:\\runner\\jobs\\0\\config.ini\n[JOB]\npersons = user@email.com\n```\n```\n> C:\\runner\\jobs\\0\\script.py\ntry:\n 1/0\nexcept ZeroDivisionError:\n job.log.error()\n job.alarm()\n```\nAs a result you will receive a message on user@email.com:\n```\nError occurred during execution.\nJob id: 0.\nJob name: test.\nJob description: Test Job.\nJob trigger: 2018-12-31 23:59:59.\nPlease check the C:\\runner\\jobs\\0\\logs\\test_20181231235959.log for more details.\n```\nIn *C:\\runner\\jobs\\0\\logs\\test_20181231235959.log* you will find the error details:\n```\n2018-12-31 23:59:59|ERROR|FAIL! ERROR: ZeroDivisionError. FILE: C:\\runner\\jobs\\0\\script.py. LINE: 7. REASON: division by zero.\n```\n\nVisit [notifier](https://github.com/t3eHawk/notifier) page to see more features of it.\n\n### Configuration\nYou could configure both scheduler and job using configuration files or writing parameters directly to instances constructors.\nRemember that direct write to instances has a higher priority than configuration files.\n\nAlso you are able to use additional configuration files by passing them through the *config* parameter of *Scheduler* or *Job* instances:\n\n```\nscheduler = runner.Scheduler(config = 'C:/configs/myconfig.ini')\n# or\njob = runner.Job(config = ['C:/configs/myconfig.ini', 'C:/users/Kenobi/myconfig.ini'])\n```\n\nBut always consider that once you mention these additional configuration files then all parameters from them will be copied to *runner* configuration files.\n\nTo edit main configuration file use command *edit config*:\nTo edit certain job configuration file use command *edit config job 0*.\n\nOptions which most likely will be used are already placed to the configuration files.\nBelow the options descriptions.\n\n|Option |Section |Value Example |Description |\n|-------------|--------------|--------------------------------------------|----------------------------------------------------------------------|\n|editor |MANAGER |notepad, nano |Text editor used to edit files. |\n|name |SCHEDULER, JOB|scheduler, job_000 |Name of scheduler or job. |\n|desc |SCHEDULER, JOB|Scheduler, Job 0 |Description of scheduler or job. |\n|schedule |SCHEDULER |C:\\runner\\schedule.tsv, /runner/schedule.tsv|Path to schedule file. |\n|console |LOG |True, False |Output log to console instead of file. |\n|limit_by_day |LOG |True, False |Do we need to close/open log at the start of new day? |\n|limit_by_size|LOG |True, False |Do we need to close/open log when maximum size is reached? |\n|max_size |LOG |10485760 |Maximum size of log. |\n|showtime |LOG |True, False |Do we need to see each scheduler moment in log? |\n|showdelay |LOG |True, False |Do we need to see scheduler active phases time consumptions in log? |\n|python |ENVIRONMENT |python, python3 |Path or command to Python executable. |\n|cpp |ENVIRONMENT |cpp |Path or command to C++ executable. |\n|java |ENVIRONMENT |java |Path or command to Java executable. |\n|persons |JOB |ObiWanKenobi@email.com |Email address (one or more) who receives job notifications and alarms.|\n|ip |EMAIL |127.0.0.1 |Ip address of host with SMTP server. |\n|port |EMAIL |587 |Port of STMP server. |\n|need_login |EMAIL |True, False |Do we need to login to SMTP server? |\n|user |EMAIL |Kenobi |Username to login. |\n|password |EMAIL |4mayBeWithYou |Password to login. |\n|address |EMAIL |LukeSkywalker@email.com |Email address that sends job notifications and alarms. |\n|need_debug |EMAIL |True, False |Do we need to see details of connection process? |\n|need_tls |EMAIL |True, False |Do we need to cover connection to SMTP with TLS protocol? |\n\n#### More About Environments\nIn *runner* you could add any environments for job execution.\nLet's example on Python virtual environment *venv*:\n```\n$ python -m venv C:\\runner\\venv\n```\nThat will deploy Python virtual environment right in the folder with our runner application.\n\nOpen main config and add to the *ENVIRONMENT* section a new option with the path to the deployed *venv* main executable:\n```\n[ENVIRONMENT]\n<...>\nvenv = C:\\runner\\venv\\scripts\\python.exe\n```\nThen change in the schedule the *ENVIRONMENT* for the test job from *python* picked during the creation to *venv*.\n\nNow your job will be handled by virtual Python copy, so *pip install* any modules you need and feel your self free.\n\n---\nFor information about constructors attributes refer to the technical documentation.\n\nFor details of external module objects go to appropriate page with this module documentation.\n\n## Tests\nYou could find tests for Windows and Linux in /test.\n\n## Issues\nTo report about found bugs and problems refer to [issues](https://github.com/t3eHawk/runner/issues)\n\n## Uninstall\nTo uninstall just delete folder your/python/folder/Lib/site-packages/runner.\n\n\n", "description_content_type": "text/markdown", "docs_url": null, "download_url": "", "downloads": { "last_day": -1, "last_month": -1, "last_week": -1 }, "home_page": "https://github.com/t3eHawk/runner", "keywords": "", "license": "MIT", "maintainer": "", "maintainer_email": "", "name": "pypyrus-runner", "package_url": "https://pypi.org/project/pypyrus-runner/", "platform": "", "project_url": "https://pypi.org/project/pypyrus-runner/", "project_urls": { "Homepage": "https://github.com/t3eHawk/runner" }, "release_url": "https://pypi.org/project/pypyrus-runner/0.0.1/", "requires_dist": [ "pypyrus-tables (>=0.0.2)", "pypyrus-logbook (>=0.0.2)" ], "requires_python": "", "summary": "Python scheduler and job manager.", "version": "0.0.1" }, "last_serial": 4799417, "releases": { "0.0.1": [ { "comment_text": "", "digests": { "md5": "d4a910e0708915d7ac9d4314a15feb80", "sha256": "03cca17e36328215bc3f2180cdd1c8712f6ba5cfebf8307b995adfd6df90cdc1" }, "downloads": -1, "filename": "pypyrus_runner-0.0.1-py3-none-any.whl", "has_sig": false, "md5_digest": "d4a910e0708915d7ac9d4314a15feb80", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": null, "size": 20428, "upload_time": "2019-02-09T13:49:43", "url": "https://files.pythonhosted.org/packages/49/ce/f5d110ee142c81373ad782ff0c5fb3a52becabceb330dbdad6c31e9d266d/pypyrus_runner-0.0.1-py3-none-any.whl" } ] }, "urls": [ { "comment_text": "", "digests": { "md5": "d4a910e0708915d7ac9d4314a15feb80", "sha256": "03cca17e36328215bc3f2180cdd1c8712f6ba5cfebf8307b995adfd6df90cdc1" }, "downloads": -1, "filename": "pypyrus_runner-0.0.1-py3-none-any.whl", "has_sig": false, "md5_digest": "d4a910e0708915d7ac9d4314a15feb80", "packagetype": "bdist_wheel", "python_version": "py3", "requires_python": null, "size": 20428, "upload_time": "2019-02-09T13:49:43", "url": "https://files.pythonhosted.org/packages/49/ce/f5d110ee142c81373ad782ff0c5fb3a52becabceb330dbdad6c31e9d266d/pypyrus_runner-0.0.1-py3-none-any.whl" } ] }