Apart from file dependencies you can extend doit to support other ways to determine if a task is up-to-date through the attribute uptodate.
This can be used in cases where you need to some kind of calculation to determine if the task is up-to-date or not.
uptodate is a list where each element can be True, False, None, a callable or a command(string).
- False indicates that the task is NOT up-to-date
- True indicates that the task is up-to-date
- None values will just be ignored. This is used when the value is dynamically calculated
Note
An uptodate value equal to True does not override others up-to-date checks. It is one more way to check if task is not up-to-date.
i.e. if uptodate==True but a file_dep changes the task is still considered not up-to-date.
If an uptodate item is a string it will be executed on the shell. If the process exits with the code 0, it is considered as up-to-date. All other values would be considered as not up-to-date.
uptodate elements can also be a callable that will be executed on runtime (not when the task is being created). The section custom-uptodate will explain in details how to extend doit writing your own callables for uptodate. This callables will typically compare a value on the present time with a value calculated on the last successful execution.
doit includes several implementations to be used as uptodate. They are all included in module doit.tools and will be discussed in detail later:
- result_dep: check if the result of another task has changed
- run_once: execute a task only once (used for tasks without dependencies)
- timeout: indicate that a task should “expire” after a certain time interval
- config_changed: check for changes in a “configuration” string or dictionary
- check_timestamp_unchanged: check access, status change/create or modify timestamp of a given file/directory
A task is not up-to-date if any of file_dep or uptodate is not up-to-date or there is a missing target. If a task does not define any of these dependencies it will always be executed.
Apart from these dependencies used to determine if a task is up-to-date or not. doit also includes other kind of dependencies to help you combine tasks so they are executed in appropriate order.
It is used to enforce tasks are executed on the desired order. By default tasks are executed on the same order as they were defined in the dodo file. To define a dependency on another task use the task name (whatever comes after task_ on the function name) in the “task_dep” attribute.
Note
A task-dependency only indicates that another task should be “executed” before itself. The task-dependency might not really be executed if it is up-to-date.
Note
task-dependencies are not used to determine if a task is up-to-date or not. If a task defines only task-dependency it will always be executed.
This example we make sure we include a file with the latest revision number of the mercurial repository on the tar file.
def task_tar():
return {'actions': ["tar -cf foo.tar *"],
'task_dep':['version'],
'targets':['foo.tar']}
def task_version():
return {'actions': ["hg tip --template '{rev}' > revision.txt"]}
$ doit
. version
. tar
You can define a group of tasks by adding tasks as dependencies and setting its actions to None.
def task_foo():
return {'actions': ["echo foo"]}
def task_bar():
return {'actions': ["echo bar"]}
def task_mygroup():
return {'actions': None,
'task_dep': ['foo', 'bar']}
Note that tasks are never executed twice in the same “run”.
Calculation of dependencies might be an expensive operation, so not suitable to be done on load time by task-creators. For this situation it is better to delegate the calculation of dependencies to another task. The task calculating dependencies must have a python-action returning a dictionary with file_dep, task_dep, uptodate or another calc_dep.
On the example below mod_deps prints on the screen all direct dependencies from a module. The dependencies itself are calculated on task get_dep (note: get_dep has a fake implementation where the results are taken from a dict).
DOIT_CONFIG = {'verbosity': 2}
MOD_IMPORTS = {'a': ['b','c'],
'b': ['f','g'],
'c': [],
'f': ['a'],
'g': []}
def print_deps(mod, dependencies):
print "%s -> %s" % (mod, dependencies)
def task_mod_deps():
"""task that depends on all direct imports"""
for mod in MOD_IMPORTS.iterkeys():
yield {'name': mod,
'actions': [(print_deps,(mod,))],
'file_dep': [mod],
'calc_dep': ["get_dep:%s" % mod],
}
def get_dep(mod):
# fake implementation
return {'file_dep': MOD_IMPORTS[mod]}
def task_get_dep():
"""get direct dependencies for each module"""
for mod in MOD_IMPORTS.iterkeys():
yield {'name': mod,
'actions':[(get_dep,[mod])],
'file_dep': [mod],
}
Some tasks may require some kind of environment setup. In this case they can define a list of “setup” tasks.
Note
A task-dependency is executed before checking if the task is up-to-date. A setup-task is executed after the checking if the task is up-to-date and it is executed only if the task is not up-to-date and will be executed.
Task may also define ‘teardown’ actions. These actions are executed after all tasks have finished their execution. They are executed in reverse order their tasks were executed.
Example:
### task setup env. good for functional tests!
DOIT_CONFIG = {'verbosity': 2,
'default_tasks': ['withenvX', 'withenvY']}
def start(name):
print "start %s" % name
def stop(name):
print "stop %s" % name
def task_setup_sample():
for name in ('setupX', 'setupY'):
yield {'name': name,
'actions': [(start, (name,))],
'teardown': [(stop, (name,))],
}
def task_withenvX():
for fin in ('a','b','c'):
yield {'name': fin,
'actions':['echo x %s' % fin],
'setup': ['setup_sample:setupX'],
}
def task_withenvY():
return {'actions':['echo y'],
'setup': ['setup_sample:setupY'],
}
$ doit withenvX
. setup_sample:setupX
start setupX
. withenvX:c
x c
. withenvX:b
x b
. withenvX:a
x a
stop setupX
$ doit withenvY
. setup_sample:setupY
start setupY
. withenvY
y
stop setupY
Tasks can save computed values by returning a dictionary on it’s python-actions. The values must be JSON encodable.
A cmd-action can also save it’s output. But for this you will need to explicitly import CmdAction and set its save_out parameter with the name used to save the output in values
from doit.action import CmdAction
def task_save_output():
return {
'actions': [CmdAction("echo x1", save_out='out')],
}
# The task values will contain: {'out': u'x1'}
These values can be used on uptodate and getargs. Check those sections for examples.
getargs provides a way to use values computed from one task in another task. The values are taken from “saved computed values” (returned dict from a python-action).
For cmd-action use dictionary-based string formatting.
For python-action the action callable parameter names must match with keys from getargs.
getargs is a dictionary where the key is the argument name used on actions, and the value is a tuple with 2 strings: task name, “value name”.
DOIT_CONFIG = {'default_tasks': ['use_cmd', 'use_python']}
def task_compute():
def comp():
return {'x':5,'y':10, 'z': 20}
return {'actions': [(comp,)]}
def task_use_cmd():
return {'actions': ['echo x=%(x)s, z=%(z)s'],
'getargs': {'x': ('compute', 'x'),
'z': ('compute', 'z')},
'verbosity': 2,
}
def task_use_python():
return {'actions': [show_getargs],
'getargs': {'x': ('compute', 'x'),
'y': ('compute', 'z')},
'verbosity': 2,
}
def show_getargs(x, y):
print "this is x:%s" % x
print "this is y:%s" % y
The values are being passed on to a python-action you can pass the whole dict by specifying the value name as None.
def task_compute():
def comp():
return {'x':5,'y':10, 'z': 20}
return {'actions': [(comp,)]}
def show_getargs(values):
print values
def task_args_dict():
return {'actions': [show_getargs],
'getargs': {'values': ('compute', None)},
'verbosity': 2,
}
If a group-task is used, the values from all its sub-tasks are passed as a dict.
def task_compute():
def comp(x):
return {'x':x}
yield {'name': '5',
'actions': [ (comp, [5]) ]
}
yield {'name': '7',
'actions': [ (comp, [7]) ]
}
def show_getargs(values):
print values
assert sum(v['x'] for v in values.itervalues()) == 12
def task_args_dict():
return {'actions': [show_getargs],
'getargs': {'values': ('compute', None)},
'verbosity': 2,
}
Note
getargs creates an implicit setup-task.
It is common situation to use task information such as targets, dependencies, or changed in its own actions. Note: Dependencies here refers only to file-dependencies.
For cmd-action you can use the python notation for keyword substitution on strings. The string will contain all values separated by a space (” ”).
For python-action create a parameter in the function, doit will take care of passing the value when the function is called. The values are passed as list of strings.
def task_hello():
"""hello"""
def python_hello(targets):
with open(targets[0], "a") as output:
output.write("Python says Hello World!!!\n")
return {
'actions': [python_hello],
'targets': ["hello.txt"],
}
You can also pass the keyword task to have a reference to all task metadata.
def who(task):
print 'my name is', task.name
print task.targets
def task_x():
return {
'actions': [who],
'targets': ['asdf'],
'verbosity': 2,
}