Scheduled Tasks¶
By “scheduled task” we mean, a command which runs automatically at a pre-determined time. See also Commands and Writing Scripts.
For now, Rattail assumes that any scheduled tasks will be manually configured to run via cron. See your operating system’s documentation for details.
(In the future, Rattail may add a dedicated scheduler daemon, but it is not yet a priority.)
Luigi¶
Luigi is a “task runner” of sorts. Rattail uses it to help orchestrate a “batch” of commands such as is often needed for overnight automation (aka. “EOD”).
The general idea is to maintain a Python module containing all of the individual tasks which need to run. Rattail can then invoke the module in such a way that Luigi runs all the tasks, keeping track of which tasks have (not yet) ran, and in which order they should run etc.
Overnight¶
Often there is just one “overnight” module, which contains all tasks. But in some cases you may want a couple of separate modules, to run at different times. A common example is like:
run “all” (typical EOD) tasks, starting at 1am
run “backups” for all servers, starting at 4am
In this example the “all” task may include a dozen or more individual “steps” (which, confusingly, are also called “tasks”). Maybe data is exported from one system to another, reports are ran, etc. Then the “backups” task runs later, and may itself also contain multiple steps, or perhaps just one.
There are two main benefits from wiring up overnight tasks via modules to be ran by Luigi.
The first is that each individual step will (typically) start immediately after the previous one completes. This means all steps are “packed in” as tightly as possible, to help keep overall run time down. Whereas scheduling each step separately often involves “guessing” how long each will take, and for example you might schedule them 5 minutes apart just to be safe, even if a given step only needs 2 minutes. (So another benefit really, is just avoiding the hassle of scheduling all those steps separately to begin with.)
The other benefit is to allow for restarting the overnight tasks. Luigi can keep track of which steps have ran thus far, so if there is a failure, which you then address, you can restart the overnight task and Luigi will pick up where it left off. (Steps which already ran successfully will be skipped over.)