queenbee.recipe.recipe module

Queenbee recipe class.

Recipe is a collection of plugins and inter-related tasks that describes an end to end steps for the recipe.

class queenbee.recipe.recipe.BakedRecipe(*, type: ConstrainedStrValue = 'BakedRecipe', annotations: Dict[str, Any] = None, api_version: ConstrainedStrValue = 'v1beta1', metadata: MetaData = None, dependencies: List[Dependency] = None, flow: List[DAG], digest: str, templates: List[Union[TemplateFunction, DAG]])

Bases: Recipe

Baked Recipe.

A Baked Recipe contains all the templates referred to in the DAG within a templates list.

classmethod check_inputs(values)

Check DAG Task inputs match template inputs

classmethod check_template_dependency_names(v, values)

Overwrite test function from inherited class

digest: str

classmethod from_folder(folder_path: str, refresh_deps: bool = True, config: ~queenbee.config.Config = type: Config annotations: {} auth: [] repositories: [])

Generate a baked recipe from a recipe folder

Note

Here is an example of a Recipe folder:

.
├── .dependencies
│   ├── plugins
│   │   └── <sha-256>.yaml
│   └── recipes
│       └── <sha-256>.yaml
├── flow
|   ├── sub-dag.yaml
│   └── main.yaml
├── dependencies.yaml
└── package.yaml

Parameters:

read (folder_path {str} -- The path to the folder to) –

Keyword Arguments:

• of (refresh_deps {bool} -- Fetch the dependencies from their source instead) – the .dependencies folder (default: {True})

• (default (config {Config} -- A queenbee config object) – {Config()})

Returns:

BakedRecipe – A baked recipe

classmethod from_recipe(recipe: ~queenbee.recipe.recipe.Recipe, config: ~queenbee.config.Config = type: Config annotations: {} auth: [] repositories: [])

Bake a recipe

Parameters:

recipe (recipe {Recipe} -- A Queenbee) –

Keyword Arguments:

(default (config {Config} -- A queenbee config object) – {Config()})

Raises:

ValueError – The dependencies or templates do not match the flow

Returns:

BakedRecipe – A baked recipe

classmethod remove_duplicates(v)

Remove duplicated templates by name

classmethod replace_template_refs(dependencies: List[Dependency], dags: List[DAG], digest_dict: Dict[str, str]) → List[DAG]

Replace DAG Task template names with unique dependency digest based names

Parameters:

• dependencies (dependencies {List[Dependency]} -- A list of locked) –

• DAGs (dags {List[DAG]} -- A list of) –

• Recipe (digest {str} -- The digest of the Recipe used to create the Baked) –

Raises:

ValueError – Unresolvable dependency in a task

Returns:

List[DAG] – A list of DAGs with updated template names

property root_dag: DAG

static template_by_name(templates: List[Union[TemplateFunction, DAG]], name: str) → Union[DAG, TemplateFunction]

Retrieve a template from a list by name

Parameters:

• {List[Union[DAG (templates) –

• templates (TemplateFunction]]} -- A list of) –

• byt (name {str} -- The name to retrieve a template) –

Raises:

ValueError – Template not found

Returns:

Union[DAG, TemplateFunction] – A template

templates: List[Union[TemplateFunction, DAG]]

type: ConstrainedStrValue

class queenbee.recipe.recipe.Recipe(*, type: ConstrainedStrValue = 'Recipe', annotations: Dict[str, Any] = None, api_version: ConstrainedStrValue = 'v1beta1', metadata: MetaData = None, dependencies: List[Dependency] = None, flow: List[DAG])

Bases: BaseModel

A Queenbee Recipe

api_version: ConstrainedStrValue

classmethod check_dag_names(v, values)

Check DAG names do not overlap with dependency names

classmethod check_entrypoint(v)

Check the main DAG exists in the flow

classmethod check_template_dependency_names(v, values)

Check all DAG template names exist in dependencies or other DAGs

static dag_by_name(flow: List[DAG], name: str) → DAG

Retrieve a DAG from a list by its name

Parameters:

• DAGs (flow {List[DAG]} -- A list of) –

• by (name {str} -- The name to search) –

Raises:

ValueError – No DAG found with the input name

Returns:

DAG – A DAG

dependencies: List[Dependency]

static dependency_by_name(dependencies: List[Dependency], name: str) → Dependency

Retrieve a dependency by its reference name (name or alias)

Parameters:

• dependencies (dependencies {List[Dependency]} -- A list of) –

• by (name {str} -- The name or alias to search dependencies) –

Raises:

ValueError – No dependency found with the input name

Returns:

Dependency – A dependency

flow: List[DAG]

classmethod from_folder(folder_path: str)

Generate a recipe from a folder

Note

Here is an example of a Recipe folder:

.
├── .dependencies
│   ├── plugins
│   │   └── <sha-256>.yaml
│   └── recipes
│       └── <sha-256>.yaml
├── flow
|   ├── sub-dag.yaml
│   └── main.yaml
├── dependencies.yaml
└── package.yaml

Parameters:

folder (folder_path {str} -- Path to the) –

Returns:

Recipe – A Recipe

property inputs: List[Union[DAGGenericInput, DAGStringInput, DAGIntegerInput, DAGNumberInput, DAGBooleanInput, DAGFolderInput, DAGFileInput, DAGPathInput, DAGArrayInput, DAGJSONObjectInput]]

Get the Recipe’s inputs

Returns:

List[DAGInputs] – The inputs of the entrypoint DAG

property is_locked: bool

Indicates whether the Recipe dependencies are all locked

Returns:

bool – True if all dependencies are locked

lock_dependencies(config: ~queenbee.config.Config = type: Config annotations: {} auth: [] repositories: [])

Lock the dependencies by fetching them and storing their digest

metadata: MetaData

property outputs: List[Union[DAGGenericOutput, DAGStringOutput, DAGIntegerOutput, DAGNumberOutput, DAGBooleanOutput, DAGFolderOutput, DAGFileOutput, DAGPathOutput, DAGArrayOutput, DAGJSONObjectOutput]]

Get the Recipe’s outputs

Returns:

List[DAGOutputs] – The outputs of the entrypoint DAG

property root_dag: DAG

classmethod sort_list(v)

Sort the list of DAGs by name

to_folder(folder_path: str, readme_string: Optional[str] = None)

Write a Recipe to a folder

Note

Here is an example of a Recipe folder:

.
├── .dependencies
│   ├── plugin
│   │   └── plugin-dep-name
│   │       ├── functions
│   │       │   ├── func-1.yaml
│   │       │   ├── ...
│   │       │   └── func-n.yaml
│   │       ├── config.yaml
│   │       └── package.yaml
│   └── recipe
│       └── recipe-dep-name
│           ├── .dependencies
│           │   ├── plugin
│           │   └── recipe
│           ├── flow
│           │   └── main.yaml
│           ├── dependencies.yaml
│           └── package.yaml
├── flow
│   └── main.yaml
├── dependencies.yaml
└── package.yaml

Parameters:

folder (folder_path {str} -- The path to the Recipe) –

Keyword Arguments:

(default (readme_string {str} -- The README file string) – {None})

type: ConstrainedStrValue

write_dependencies(folder_path: str, config: ~queenbee.config.Config = type: Config annotations: {} auth: [] repositories: [])

Fetch dependencies manifests and write them to the .dependencies folder

Parameters:

folder (folder_path {str} -- Path to the recipe) –

Keyword Arguments:

(default (config {Config} -- A queenbee config object) – {Config()})

write_dependency_file(folder_path: str)

Write the locked dependencies to a Recipe folder

Parameters:

folder (folder_path {str} -- The path to the recipe) –

class queenbee.recipe.recipe.RecipeInterface(*, type: ConstrainedStrValue = 'RecipeInterface', annotations: Dict[str, Any] = None, api_version: ConstrainedStrValue = 'v1beta1', metadata: MetaData, source: str = None, inputs: List[Union[DAGGenericInput, DAGStringInput, DAGIntegerInput, DAGNumberInput, DAGBooleanInput, DAGFolderInput, DAGFileInput, DAGPathInput, DAGArrayInput, DAGJSONObjectInput]] = None, outputs: List[Union[DAGGenericOutput, DAGStringOutput, DAGIntegerOutput, DAGNumberOutput, DAGBooleanOutput, DAGFolderOutput, DAGFileOutput, DAGPathOutput, DAGArrayOutput, DAGJSONObjectOutput]] = None)

Bases: BaseModel

An interface object for creating a Recipe.

Recipe information only includes metadata, source, inputs and outputs of a Recipe. This object is useful for creating user interface for Recipes.

api_version: ConstrainedStrValue

classmethod create_empty_list(v)

classmethod from_recipe(recipe: Union[Recipe, BakedRecipe], source: Optional[str] = None)

Create a Recipe info from a recipe.

inputs: List[Union[DAGGenericInput, DAGStringInput, DAGIntegerInput, DAGNumberInput, DAGBooleanInput, DAGFolderInput, DAGFileInput, DAGPathInput, DAGArrayInput, DAGJSONObjectInput]]

metadata: MetaData

outputs: List[Union[DAGGenericOutput, DAGStringOutput, DAGIntegerOutput, DAGNumberOutput, DAGBooleanOutput, DAGFolderOutput, DAGFileOutput, DAGPathOutput, DAGArrayOutput, DAGJSONObjectOutput]]

source: str

type: ConstrainedStrValue

class queenbee.recipe.recipe.TemplateFunction(*, type: ConstrainedStrValue = 'TemplateFunction', annotations: Dict[str, Any] = None, inputs: List[Union[FunctionStringInput, FunctionIntegerInput, FunctionNumberInput, FunctionBooleanInput, FunctionFolderInput, FunctionFileInput, FunctionPathInput, FunctionArrayInput, FunctionJSONObjectInput]] = None, outputs: List[Union[FunctionStringOutput, FunctionIntegerOutput, FunctionNumberOutput, FunctionBooleanOutput, FunctionFolderOutput, FunctionFileOutput, FunctionPathOutput, FunctionArrayOutput, FunctionJSONObjectOutput]] = None, name: str, description: str = None, command: Optional[str] = None, language: ScriptingLanguages = ScriptingLanguages.Python, source: Optional[str] = None, config: PluginConfig)

Bases: Function

Function template.

config: PluginConfig

classmethod from_plugin(plugin: Plugin) → list

Generate a list of template functions from a plugin

Parameters:

plugin (plugin {Plugin} -- A) –

Returns:

list – A list of template functions

type: ConstrainedStrValue