The aim of this project is to create simple and effective instrument for automatic job scheduling and management in Python.
Main runner features:
- Minimal and user friendly command line interface.
- Cross platform - application can be used in any OS with Python 3.
- Opportunity to handle all jobs in one place instead of using many (Oracle Enterprise Scheduler, Linux Crontab and etc.).
- Generate built-in scheduler.
- Generate built-in job.
- Edit generated built-in jobs.
- Delete generated built-in jobs.
- Configuration with ini files.
- Automatic job scheduling and execution at certain time, during certain period or each time cycle.
- Manual job execution for now or for certain date and time.
- Scheduling with simple tsv file.
- Run jobs using virtual Python environment.
- Run jobs implemented in non Python technologies like C++, Java, Bash, Perl, etc.
- Built-in logging.
- Built-in notifications and alarms.
Main runner targets:
- Add some CLI commands.
- Attach database to runner as option for schedule and new data streams storage.
- Add simple web GUI.
- Improve notifications' mechanism and how they looks.
- Optimizations.
Operation systems: Windows, Linux, Mac OS.
Python version: 3.7.1.
Python modules dependencies (use latest releases): logbook, tables, notifier
To install just download the latest runner release and copy /src/runner folder to your/python/folder/Lib/site-packages/.
To start working with runner you need just one simple file containing runner.Manager instance.
Let's go to the root of your disk *C:* (or any other location, it does not matter) and create the folder runner. Move to that folder and create new file named manager.py with next few string of code.
> C:\runner\manager.py
import runner
manager = runner.Manager()
Now manager.py is your main runner interface and the way to deploy the application in C:\runner folder.
Execute the manager.py. You will receive the main help note that can also be called with the manager.py help.
When you execute manager.py first time it generates main configuration file config.ini in the same folder.
$ python manager.py
INFO: File C:\runner\config.ini created.
List of available commands:
create scheduler Generate a scheduler with all elements.
create job Generate a job with all elements.
list jobs Show all jobs registered in the schedule.
edit schedule Open the schedule file in the editor.
edit job Open the job script in the editor.
run job Execute the job by id with or without run time.
run jobs Execute the jobs listed in the file.
delete job Delete the job by id.
edit config Open one of the configuration files in the editor.
help Show this message.
For more details type [command] help.
You could get more specific help for each command:
$ python manager.py run job help
Execute the job by id with or without run time.
Parameters:
id integer Id of the job you want to run.
trigger yyyy-mm-dd/hh24:mi:ss Date with or without time in ISO format
yyyy-mm-dd for what you want to run the job.
We recommend to repeat that action for each command for more understanding.
Type command create scheduler to generate the scheduler. You will be asked about necessary attributes and then scheduler will be generated in the folder.
$ python manager.py create scheduler
Please follow the steps to create the scheduler.
Enter the name or leave empty to use default:
Enter the description or leave empty to use default:
********************************************************************************
INFO: Creating scheduler scheduler...
INFO: File C:\runner\scheduler.py created.
INFO: File C:\runner\config.ini updated.
INFO: File C:\runner\schedule.tsv created.
INFO: Folder C:\runner\jobs created.
About files:
C:\runner\scheudler.py - scheduler instance and starter.
C:\runner\config.ini - main configuration file.
C:\runner\schedule.tsv - job schedule.
C:\runner\jobs - folder for future jobs.
To run the scheduler just execute file scheduler.py.
$ python scheduler.py
Now the scheduler is working. Logs can be found in the folder C:\runner\logs.
To stop the scheduler process just type the CTRL + C.
Scheduler is ready and jobs can be created.
Type command create job to generate the job. You will be asked about necessary attributes and then job will be generated.
$ python manager.py create job
Follow the instructions to create the job.
Inputs with * are mandatory.
Enter the name or leave empty to use default:
test
Enter the short description:
Test Job
Enter the environment or leave empty to use Python:
Enter the month day (1-31):
Enter the week day (1-7):
Enter the hour (0-23):
8
Enter the minute (0-59):
0
Enter the second (0-59):
0
Activate job (Y/N)? *:
N
********************************************************************************
INFO: Creating job...
INFO: Job ID: 0
INFO: Folder C:\runner\jobs\0 created.
INFO: File C:\runner\jobs\0\job.py created.
INFO: File C:\runner\jobs\0\config.ini created.
INFO: File C:\runner\jobs\0\script.py created.
INFO: Job test successfully added to schedule!
About files:
C:\runner\jobs\0 - this is a job folder which corresponds to job id.
C:\runner\jobs\0\job.py - job instance and starter.
C:\runner\jobs\0\config.ini - job configuration file.
C:\runner\jobs\0\script.py - script that must be executed by job.
In schedule.tsv new record for created job will be provided.
Note that we used status N. Jobs with such status will be not executed until status changed to Y. It is done to prevent accidental job run before it is not finalized.
Schedule is one of the most important part of the application.
All jobs scheduled in schedule.tsv. The file is scanned by runner and its formatting is quite critical.
Main formatting requirements are:
- Each line is a table row.
- Column separator is only a tabulation.
- Last line must be always empty.
When schedule.tsv is modified runner catches the changes, makes necessary updates and also prints message to the log about that.
To see all scheduled jobs use command list jobs:
$ python manager.py list jobs
-----------------------------------------------------------------------------------------------------------------------------
|ID |NAME |DESCRIPTION |ENVIRONMENT |FILE |MONTH_DAY |WEEK_DAY |HOUR |MINUTE |SECOND |PARAMETERS |STATUS |
-----------------------------------------------------------------------------------------------------------------------------
|0 |test |Test Job |python |C:\runner\jobs\0\job.py |* |* |8 |0 |0 | |N |
-----------------------------------------------------------------------------------------------------------------------------
To see only active jobs add active.
$ python manager.py list jobs active
Also you could either open file directly or use command manager.py edit schedule to open file using chosen editor in the main config.
| Field name | Description |
|---|---|
| ID | Unique id that job receives automatically during creation. |
| NAME | Name of job. |
| DESCRIPTION | Description of job. |
| ENVIRONMENT | Name of environment in which job must be executed. All used in runner environments must be listed in the main config. |
| FILE | Entry file. If you use built-in jobs then it is a job.py. In other case it is up to you how job should be started. |
| MONTH_DAY | Day of month when job must be executed. |
| WEEK_DAY | Day of week when job must be executed. |
| HOUR | Hour of day when job must be executed. |
| MINUTE | Minute of our when job must be executed. |
| SECOND | Second of minute when job must be executed. |
| PARAMETERS | Additional arguments that will be passed to FILE in the time of execution. |
| STATUS | Status of job. Y is an active, N is an inactive. |
Job frequency in runner is rather flexible. Be attentive because some combinations can force runner to execute job every second!
You could schedule jobs by manipulating with time fields next few ways:
- Execute job at certain date and time. Use simple integers.
- Execute job every time cycle. Use integers with symbol / to point that it is a cycle value.
- Execute job only during certain period. Use integers with , and - to point that it is a period value.
For more understanding we list the most popular combinations:
| MONTH_DAY | WEEK_DAY | HOUR | MINUTE | SECOND | When to Execute |
|---|---|---|---|---|---|
| 1 | * | 8 | 0 | 0 | Each first day of month at 8 am. |
| * | 1 | 8 | 0 | 0 | Each first day of week at 8 am. |
| * | * | 8 | 0 | 0 | Each day at 8 am. |
| * | * | /1 | 0 | 0 | Each hour. |
| * | * | * | /5 | 0 | Each five minutes. |
| * | 1-5 | 8 | 0 | 0 | Each day during weekdays at 8 am. |
| 1,11,21 | * | 8 | 0 | 0 | Each 1, 11 and 21 day of month at 8 am. |
Automatic scheduling in runner is based on moments. Moment - is a dynamic scheduler attribute that describes current timestamp as a count of seconds past from era begin.
You can log each scheduler moment if you need to. For this set the showtime option in the LOG section of the main config to True. Then in log each moment will be recorded as an empty message:
2018-12-31 23:59:59|INFO|
Moment includes an active and a passive phases.
Active phase stands for all necessary actions for scheduling and internal runner needs. After active phase finished passive phase begins during which runner sleeps till the next step.
If active phase longs more than 1 second in case of some trouble then scheduler internal time will differ from real one. When this happened runner will automatically synchronize moment with real time. That will be registered in the log that may help in troubleshooting:
2018-12-31 23:59:59|WARNING|TIME IS BROKEN!
2018-12-31 23:59:59|INFO|SYNCHRONIZING THE TIME...
2018-12-31 23:59:59|INFO|SUCCESS
If you want to log each active phase time consumption then set the showdelay option of the LOG section in the main config to True:
A task is a set of actions that a job must perform. All tasks must be described in a special job script file. For our test job this is the file jobs\0\script.py. That file already contains job instance so it can be used in the script too.
Open script file or use special command edit job that will open it using chosen editor in main config.
$ python manager.py edit job 0
Modify file as in the example below:
> C:\runner\jobs\0\job.py
from job import job
# Write the code to be executed by job down below.
print('Wake up, Neo...')
Now go to schedule.tsv and change STATUS for modified job from N to Y.
Job is ready and at next 08:00:00 it will be executed.
Job can be executed automatically by scheduler or manually by user.
For automatic execution in schedule.tsv must be determined:
- ENVIRONMENT that is a name of environment described in the main config.
- MONTH_DAY, WEEK_DAY, HOUR, MINUTE, SECOND - time of execution.
User execution can be done by two ways.
The first one is to use special command run job with mandatory arguments id and optional argument trigger.
Argument id is an integer unique identification number of job that can be found in schedule.tsv.
Argument trigger is a date or datetime in ISO timestamp format for which job must be executed.
In the example below we execute the job with id 0 for 1th January 2019, 8 AM.
$ python manager.py run job 0 2019-01-01/08:00:00
INFO: REQUEST TO RUN THE JOB.
INFO: ID: 0.
INFO: NAME: test.
INFO: DESCRIPTION: Test Job.
INFO: TRIGGER: 2019-01-01/08:00:00.
ARE YOU SURE Y/N?
Y
Wake up, Neo...
INFO: DONE.
Second one is to execute job.py file of the job you want to run. Following command will execute the job with id 0 for current moment:
$ python jobs\0\job.py
Wake up, Neo...
Optionally you could use argument -t/--trigger to pass certain date and time. So execution of job with id 0 for 1th January 2019, 8 AM will looks like:
$ python jobs\0\job.py -r 2019-01-01/08:00:00
Wake up, Neo...
Also if you have many jobs that must be executed it is possible to use special file. Let's create file C:\runner\run_jobs.txt with listed ids and triggers:
0 2018-12-31/08:00:00
0 2019-01-01/08:00:00
Note that between id and trigger there is a space.
Execute the command:
$ python manager.py run jobs C:\runner\run_jobs.txt
INFO: REQUEST TO RUN THE JOB.
INFO: ID: 0.
INFO: NAME: test.
INFO: DESCRIPTION: Test Job.
INFO: TRIGGER: 2018-12-31/08:00:00.
ARE YOU SURE Y/N?
Y
Wake up, Neo...
INFO: DONE.
********************************************************************************
INFO: REQUEST TO RUN THE JOB.
INFO: ID: 0.
INFO: NAME: test.
INFO: DESCRIPTION: Test Job.
INFO: TRIGGER: 2019-01-01/08:00:00.
ARE YOU SURE Y/N?
Y
Wake up, Neo...
INFO: DONE.
Logging in runner implemented on logbook and available through the log object. So you could even use log in your script.py.
> C:\runner\jobs\0\script.py
from job import job
# Write the code to be executed by job down below.
print('Wake up, Neo...')
job.log.info('Hello there.')
Let's imagine that today is 31th December 2018, 23:59:59. Run the job and in log file you will see:
> C:\runner\jobs\0\logs\test_20181231235959.log
********************************************************************************
* *
* APP: test *
* DESC: Test Job *
* VERSION: None *
* TIMESTAMP: 2018-12-31 23:59:59 *
* FILE: C:\runner\jobs\0\logs\test_20181231235959.log *
* MACHINE: AMD64 *
* PROCESSOR: Intel64 Family 6 Model 78 Stepping 3, GenuineIntel *
* HOSTNAME: Falcon *
* USER: Kenobi *
* SYSTEM_NM: Windows *
* SYSTEM_RLS: 10 *
* SYSTEM_VER: 10.0.14393 *
* PYTHON: 3.7.1 *
* ID: 0 *
* TRIGGER: 2018-12-31 23:59:59 *
* CONFIGS: C:\runner\jobs\0\config.ini *
* PERSONS: None *
* *
********************************************************************************
2018-12-31 23:59:59|INFO|JOB STARTED.
2018-12-31 23:59:59|INFO|Hello there.
2018-12-31 23:59:59|INFO|JOB FINISHED.
2018-12-31 23:59:59|INFO|TIME SPENT: 0 seconds.
Visit logbook page to see more features of it.
Sometimes in work you may need to get feedback from the job. It can be easily done with a help of email object which allows to write messages on the email.
To use notifier you need SMTP server. Connection must be defined in config: ip, port, user, password, email address options of the [EMAIL] section in the job config (or other external config).
To send simple text message on the email:
> C:\runner\jobs\0\script.py
subject = 'Help me'
text = 'You are my only hope'
recipient = 'ObiWanKenobi@email.com'
job.email.notify(subject, text, recipient)
Another case of using notifier is to inform that something went wrong in the job execution. We use alarms for this purposes in runner. Use alarm() to send standard alarm notification on emails listed in the persons option of the [GENERAL] section in the job config.
Let's change our job with id 0 little bit and execute it.
> C:\runner\jobs\0\config.ini
[JOB]
persons = user@email.com
> C:\runner\jobs\0\script.py
try:
1/0
except ZeroDivisionError:
job.log.error()
job.alarm()
As a result you will receive a message on user@email.com:
Error occurred during execution.
Job id: 0.
Job name: test.
Job description: Test Job.
Job trigger: 2018-12-31 23:59:59.
Please check the C:\runner\jobs\0\logs\test_20181231235959.log for more details.
In C:\runner\jobs\0\logs\test_20181231235959.log you will find the error details:
2018-12-31 23:59:59|ERROR|FAIL! ERROR: ZeroDivisionError. FILE: C:\runner\jobs\0\script.py. LINE: 7. REASON: division by zero.
Visit notifier page to see more features of it.
You could configure both scheduler and job using configuration files or writing parameters directly to instances constructors. Remember that direct write to instances has a higher priority than configuration files.
Also you are able to use additional configuration files by passing them through the config parameter of Scheduler or Job instances:
scheduler = runner.Scheduler(config = 'C:/configs/myconfig.ini')
# or
job = runner.Job(config = ['C:/configs/myconfig.ini', 'C:/users/Kenobi/myconfig.ini'])
But always consider that once you mention these additional configuration files then all parameters from them will be copied to runner configuration files.
To edit main configuration file use command edit config: To edit certain job configuration file use command edit config job 0.
Options which most likely will be used are already placed to the configuration files. Below the options descriptions.
| Option | Section | Value Example | Description |
|---|---|---|---|
| editor | MANAGER | notepad, nano | Text editor used to edit files. |
| name | SCHEDULER, JOB | scheduler, job_000 | Name of scheduler or job. |
| desc | SCHEDULER, JOB | Scheduler, Job 0 | Description of scheduler or job. |
| schedule | SCHEDULER | C:\runner\schedule.tsv, /runner/schedule.tsv | Path to schedule file. |
| console | LOG | True, False | Output log to console instead of file. |
| limit_by_day | LOG | True, False | Do we need to close/open log at the start of new day? |
| limit_by_size | LOG | True, False | Do we need to close/open log when maximum size is reached? |
| max_size | LOG | 10485760 | Maximum size of log. |
| showtime | LOG | True, False | Do we need to see each scheduler moment in log? |
| showdelay | LOG | True, False | Do we need to see scheduler active phases time consumptions in log? |
| python | ENVIRONMENT | python, python3 | Path or command to Python executable. |
| cpp | ENVIRONMENT | cpp | Path or command to C++ executable. |
| java | ENVIRONMENT | java | Path or command to Java executable. |
| persons | JOB | ObiWanKenobi@email.com | Email address (one or more) who receives job notifications and alarms. |
| ip | 127.0.0.1 | Ip address of host with SMTP server. | |
| port | 587 | Port of STMP server. | |
| need_login | True, False | Do we need to login to SMTP server? | |
| user | Kenobi | Username to login. | |
| password | 4mayBeWithYou | Password to login. | |
| address | LukeSkywalker@email.com | Email address that sends job notifications and alarms. | |
| need_debug | True, False | Do we need to see details of connection process? | |
| need_tls | True, False | Do we need to cover connection to SMTP with TLS protocol? |
In runner you could add any environments for job execution. Let's example on Python virtual environment venv:
$ python -m venv C:\runner\venv
That will deploy Python virtual environment right in the folder with our runner application.
Open main config and add to the ENVIRONMENT section a new option with the path to the deployed venv main executable:
[ENVIRONMENT]
<...>
venv = C:\runner\venv\scripts\python.exe
Then change in the schedule the ENVIRONMENT for the test job from python picked during the creation to venv.
Now your job will be handled by virtual Python copy, so pip install any modules you need and feel your self free.
For information about constructors attributes refer to the technical documentation.
For details of external module objects go to appropriate page with this module documentation.
You could find tests for Windows and Linux in /test.
To report about found bugs and problems refer to issues
To uninstall just delete folder your/python/folder/Lib/site-packages/runner.