queenbee.recipe.dag module

Queenbee DAG.

A DAG defines a single step in a Recipe. Each DAG is a collection of tasks/steps. Each step indicates what function template should be used and maps inputs and outputs for the specific task.

class queenbee.recipe.dag.DAG(*, type: ConstrainedStrValue = 'DAG', annotations: Dict[str, Any] = None, inputs: List[DAGGenericInput | DAGStringInput | DAGIntegerInput | DAGNumberInput | DAGBooleanInput | DAGFolderInput | DAGFileInput | DAGPathInput | DAGArrayInput | DAGJSONObjectInput] = None, outputs: List[DAGGenericOutput | DAGStringOutput | DAGIntegerOutput | DAGNumberOutput | DAGBooleanOutput | DAGFolderOutput | DAGFileOutput | DAGPathOutput | DAGArrayOutput | DAGJSONObjectOutput] = None, name: str, fail_fast: bool = True, tasks: List[DAGTask])[source]

Bases: IOBase

A Directed Acyclic Graph containing a list of tasks.

classmethod check_dag_outputs(values)[source]

Check DAG outputs refer to existing Task outputs.

It can’t be a normal validator because of the order in which the inputs get assigned in Pydantic when a class is a subclass from another.

classmethod check_dependencies(v)[source]

Check that all task dependencies exist.

classmethod check_references(v, values)[source]

Check that input and output references exist.

classmethod check_template_name(v, values)[source]

Check that a task name does not refer to itself in a template.

classmethod check_unique_names(v)[source]

Check that all tasks have unique names.

fail_fast: bool
static find_task_return(tasks: List[DAGTask], reference: TaskReference | TaskFileReference | TaskFolderReference | TaskPathReference) → TaskReturn | TaskPathReturn[source]

Find a task output within the DAG from a reference

Parameters:

  • Tasks (tasks {List[DAGTask]} -- A list of DAG)

  • {Union[TaskReference (reference) – TaskPathReference]} – A reference to a DAG Task output

  • TaskFileReference – TaskPathReference]} – A reference to a DAG Task output

  • TaskFolderReference – TaskPathReference]} – A reference to a DAG Task output

:param : TaskPathReference]} – A reference to a DAG Task output

Raises:

  • ValueError – The task name cannot be found.

  • ValueError – The task return reference cannot be found.

Returns:

Union[TaskReturn, TaskPathReturn] – A task output parameter or artifact

get_task(name)[source]

Get task by name.

Parameters:

task (name {str} -- The name of a)

Raises:

ValueError – The task name does not exist

Returns:

DAGTask – A DAG Task

inputs: List[DAGGenericInput | DAGStringInput | DAGIntegerInput | DAGNumberInput | DAGBooleanInput | DAGFolderInput | DAGFileInput | DAGPathInput | DAGArrayInput | DAGJSONObjectInput]
name: str
outputs: List[DAGGenericOutput | DAGStringOutput | DAGIntegerOutput | DAGNumberOutput | DAGBooleanOutput | DAGFolderOutput | DAGFileOutput | DAGPathOutput | DAGArrayOutput | DAGJSONObjectOutput]
classmethod sort_list(v)[source]

Sort the list of tasks by name

tasks: List[DAGTask]
property templates: Set[str]

A list of unique templates referred to in the DAG.

Returns:

List[str] – A list of task name

type: ConstrainedStrValue