queenbee.recipe.dag module

Queenbee DAG.

A DAG defines a single step in a Recipe. Each DAG is a collection of tasks/steps. Each step indicates what function template should be used and maps inputs and outputs for the specific task.

class queenbee.recipe.dag.DAG(*, type: ConstrainedStrValue = 'DAG', annotations: Dict[str, Any] = None, inputs: List[Union[DAGGenericInput, DAGStringInput, DAGIntegerInput, DAGNumberInput, DAGBooleanInput, DAGFolderInput, DAGFileInput, DAGPathInput, DAGArrayInput, DAGJSONObjectInput]] = None, outputs: List[Union[DAGGenericOutput, DAGStringOutput, DAGIntegerOutput, DAGNumberOutput, DAGBooleanOutput, DAGFolderOutput, DAGFileOutput, DAGPathOutput, DAGArrayOutput, DAGJSONObjectOutput]] = None, name: str, fail_fast: bool = True, tasks: List[DAGTask])

Bases: IOBase

A Directed Acyclic Graph containing a list of tasks.

classmethod check_dag_outputs(values)

Check DAG outputs refer to existing Task outputs.

It can’t be a normal validator because of the order in which the inputs get assigned in Pydantic when a class is a subclass from another.

classmethod check_dependencies(v)

Check that all task dependencies exist.

classmethod check_references(v, values)

Check that input and output references exist.

classmethod check_template_name(v, values)

Check that a task name does not refer to itself in a template.

classmethod check_unique_names(v)

Check that all tasks have unique names.

fail_fast: bool

static find_task_return(tasks: List[DAGTask], reference: Union[TaskReference, TaskFileReference, TaskFolderReference, TaskPathReference]) → Union[TaskReturn, TaskPathReturn]

Find a task output within the DAG from a reference

Parameters:

• Tasks (tasks {List[DAGTask]} -- A list of DAG) –

• {Union[TaskReference (reference) – TaskPathReference]} – A reference to a DAG Task output

• TaskFileReference – TaskPathReference]} – A reference to a DAG Task output

• TaskFolderReference – TaskPathReference]} – A reference to a DAG Task output

:param : TaskPathReference]} – A reference to a DAG Task output

Raises:

• ValueError – The task name cannot be found.

• ValueError – The task return reference cannot be found.

Returns:

Union[TaskReturn, TaskPathReturn] – A task output parameter or artifact

get_task(name)

Get task by name.

Parameters:

task (name {str} -- The name of a) –

Raises:

ValueError – The task name does not exist

Returns:

DAGTask – A DAG Task

inputs: List[Union[DAGGenericInput, DAGStringInput, DAGIntegerInput, DAGNumberInput, DAGBooleanInput, DAGFolderInput, DAGFileInput, DAGPathInput, DAGArrayInput, DAGJSONObjectInput]]

name: str

outputs: List[Union[DAGGenericOutput, DAGStringOutput, DAGIntegerOutput, DAGNumberOutput, DAGBooleanOutput, DAGFolderOutput, DAGFileOutput, DAGPathOutput, DAGArrayOutput, DAGJSONObjectOutput]]

classmethod sort_list(v)

Sort the list of tasks by name

tasks: List[DAGTask]

property templates: Set[str]

A list of unique templates referred to in the DAG.

Returns:

List[str] – A list of task name

type: ConstrainedStrValue