queenbee.recipe.recipe module¶
Queenbee recipe class.
Recipe is a collection of plugins and inter-related tasks that describes an end to end steps for the recipe.
- class queenbee.recipe.recipe.BakedRecipe(*, type: ConstrainedStrValue = 'BakedRecipe', annotations: Dict[str, Any] = None, api_version: ConstrainedStrValue = 'v1beta1', metadata: MetaData = None, dependencies: List[Dependency] = None, flow: List[DAG], digest: str, templates: List[TemplateFunction | DAG])[source]¶
Bases:
Recipe
Baked Recipe.
A Baked Recipe contains all the templates referred to in the DAG within a templates list.
- classmethod check_template_dependency_names(v, values)[source]¶
Overwrite test function from inherited class
- digest: str¶
- classmethod from_folder(folder_path: str, refresh_deps: bool = True, config: ~queenbee.config.Config = type: Config annotations: {} auth: [] repositories: [])[source]¶
Generate a baked recipe from a recipe folder
Note
Here is an example of a Recipe folder:
. ├── .dependencies │ ├── plugins │ │ └── <sha-256>.yaml │ └── recipes │ └── <sha-256>.yaml ├── flow | ├── sub-dag.yaml │ └── main.yaml ├── dependencies.yaml └── package.yaml
- Parameters:
read (folder_path {str} -- The path to the folder to)
- Keyword Arguments:
of (refresh_deps {bool} -- Fetch the dependencies from their source instead) – the
.dependencies
folder (default: {True})(default (config {Config} -- A queenbee config object) – {Config()})
- Returns:
BakedRecipe – A baked recipe
- classmethod from_recipe(recipe: ~queenbee.recipe.recipe.Recipe, config: ~queenbee.config.Config = type: Config annotations: {} auth: [] repositories: [])[source]¶
Bake a recipe
- Parameters:
recipe (recipe {Recipe} -- A Queenbee)
- Keyword Arguments:
(default (config {Config} -- A queenbee config object) – {Config()})
- Raises:
ValueError – The dependencies or templates do not match the flow
- Returns:
BakedRecipe – A baked recipe
- classmethod replace_template_refs(dependencies: List[Dependency], dags: List[DAG], digest_dict: Dict[str, str]) List[DAG] [source]¶
Replace DAG Task template names with unique dependency digest based names
- Parameters:
dependencies (dependencies {List[Dependency]} -- A list of locked)
DAGs (dags {List[DAG]} -- A list of)
Recipe (digest {str} -- The digest of the Recipe used to create the Baked)
- Raises:
ValueError – Unresolvable dependency in a task
- Returns:
List[DAG] – A list of DAGs with updated template names
- static template_by_name(templates: List[TemplateFunction | DAG], name: str) DAG | TemplateFunction [source]¶
Retrieve a template from a list by name
- Parameters:
{List[Union[DAG (templates)
templates (TemplateFunction]]} -- A list of)
byt (name {str} -- The name to retrieve a template)
- Raises:
ValueError – Template not found
- Returns:
Union[DAG, TemplateFunction] – A template
- templates: List[TemplateFunction | DAG]¶
- type: ConstrainedStrValue¶
- class queenbee.recipe.recipe.Recipe(*, type: ConstrainedStrValue = 'Recipe', annotations: Dict[str, Any] = None, api_version: ConstrainedStrValue = 'v1beta1', metadata: MetaData = None, dependencies: List[Dependency] = None, flow: List[DAG])[source]¶
Bases:
BaseModel
A Queenbee Recipe
- api_version: ConstrainedStrValue¶
- classmethod check_dag_names(v, values)[source]¶
Check DAG names do not overlap with dependency names
- classmethod check_template_dependency_names(v, values)[source]¶
Check all DAG template names exist in dependencies or other DAGs
- static dag_by_name(flow: List[DAG], name: str) DAG [source]¶
Retrieve a DAG from a list by its name
- Parameters:
DAGs (flow {List[DAG]} -- A list of)
by (name {str} -- The name to search)
- Raises:
ValueError – No DAG found with the input name
- Returns:
DAG – A DAG
- dependencies: List[Dependency]¶
- static dependency_by_name(dependencies: List[Dependency], name: str) Dependency [source]¶
Retrieve a dependency by its reference name (name or alias)
- Parameters:
dependencies (dependencies {List[Dependency]} -- A list of)
by (name {str} -- The name or alias to search dependencies)
- Raises:
ValueError – No dependency found with the input name
- Returns:
Dependency – A dependency
- classmethod from_folder(folder_path: str)[source]¶
Generate a recipe from a folder
Note
Here is an example of a Recipe folder:
. ├── .dependencies │ ├── plugins │ │ └── <sha-256>.yaml │ └── recipes │ └── <sha-256>.yaml ├── flow | ├── sub-dag.yaml │ └── main.yaml ├── dependencies.yaml └── package.yaml
- Parameters:
folder (folder_path {str} -- Path to the)
- Returns:
Recipe – A Recipe
- property inputs: List[DAGGenericInput | DAGStringInput | DAGIntegerInput | DAGNumberInput | DAGBooleanInput | DAGFolderInput | DAGFileInput | DAGPathInput | DAGArrayInput | DAGJSONObjectInput]¶
Get the Recipe’s inputs
- Returns:
List[DAGInputs] – The inputs of the entrypoint DAG
- property is_locked: bool¶
Indicates whether the Recipe dependencies are all locked
- Returns:
bool – True if all dependencies are locked
- lock_dependencies(config: ~queenbee.config.Config = type: Config annotations: {} auth: [] repositories: [])[source]¶
Lock the dependencies by fetching them and storing their digest
- property outputs: List[DAGGenericOutput | DAGStringOutput | DAGIntegerOutput | DAGNumberOutput | DAGBooleanOutput | DAGFolderOutput | DAGFileOutput | DAGPathOutput | DAGArrayOutput | DAGJSONObjectOutput]¶
Get the Recipe’s outputs
- Returns:
List[DAGOutputs] – The outputs of the entrypoint DAG
- to_folder(folder_path: str, readme_string: str = None)[source]¶
Write a Recipe to a folder
Note
Here is an example of a Recipe folder:
. ├── .dependencies │ ├── plugin │ │ └── plugin-dep-name │ │ ├── functions │ │ │ ├── func-1.yaml │ │ │ ├── ... │ │ │ └── func-n.yaml │ │ ├── config.yaml │ │ └── package.yaml │ └── recipe │ └── recipe-dep-name │ ├── .dependencies │ │ ├── plugin │ │ └── recipe │ ├── flow │ │ └── main.yaml │ ├── dependencies.yaml │ └── package.yaml ├── flow │ └── main.yaml ├── dependencies.yaml └── package.yaml
- Parameters:
folder (folder_path {str} -- The path to the Recipe)
- Keyword Arguments:
(default (readme_string {str} -- The README file string) – {None})
- type: ConstrainedStrValue¶
- write_dependencies(folder_path: str, config: ~queenbee.config.Config = type: Config annotations: {} auth: [] repositories: [])[source]¶
Fetch dependencies manifests and write them to the .dependencies folder
- Parameters:
folder (folder_path {str} -- Path to the recipe)
- Keyword Arguments:
(default (config {Config} -- A queenbee config object) – {Config()})
- class queenbee.recipe.recipe.RecipeInterface(*, type: ConstrainedStrValue = 'RecipeInterface', annotations: Dict[str, Any] = None, api_version: ConstrainedStrValue = 'v1beta1', metadata: MetaData, source: str = None, inputs: List[DAGGenericInput | DAGStringInput | DAGIntegerInput | DAGNumberInput | DAGBooleanInput | DAGFolderInput | DAGFileInput | DAGPathInput | DAGArrayInput | DAGJSONObjectInput] = None, outputs: List[DAGGenericOutput | DAGStringOutput | DAGIntegerOutput | DAGNumberOutput | DAGBooleanOutput | DAGFolderOutput | DAGFileOutput | DAGPathOutput | DAGArrayOutput | DAGJSONObjectOutput] = None)[source]¶
Bases:
BaseModel
An interface object for creating a Recipe.
Recipe information only includes metadata, source, inputs and outputs of a Recipe. This object is useful for creating user interface for Recipes.
- api_version: ConstrainedStrValue¶
- classmethod from_recipe(recipe: Recipe | BakedRecipe, source: str = None)[source]¶
Create a Recipe info from a recipe.
- inputs: List[DAGGenericInput | DAGStringInput | DAGIntegerInput | DAGNumberInput | DAGBooleanInput | DAGFolderInput | DAGFileInput | DAGPathInput | DAGArrayInput | DAGJSONObjectInput]¶
- outputs: List[DAGGenericOutput | DAGStringOutput | DAGIntegerOutput | DAGNumberOutput | DAGBooleanOutput | DAGFolderOutput | DAGFileOutput | DAGPathOutput | DAGArrayOutput | DAGJSONObjectOutput]¶
- source: str¶
- type: ConstrainedStrValue¶
- class queenbee.recipe.recipe.TemplateFunction(*, type: ConstrainedStrValue = 'TemplateFunction', annotations: Dict[str, Any] = None, inputs: List[FunctionStringInput | FunctionIntegerInput | FunctionNumberInput | FunctionBooleanInput | FunctionFolderInput | FunctionFileInput | FunctionPathInput | FunctionArrayInput | FunctionJSONObjectInput] = None, outputs: List[FunctionStringOutput | FunctionIntegerOutput | FunctionNumberOutput | FunctionBooleanOutput | FunctionFolderOutput | FunctionFileOutput | FunctionPathOutput | FunctionArrayOutput | FunctionJSONObjectOutput] = None, name: str, description: str = None, command: str | None = None, language: ScriptingLanguages = ScriptingLanguages.Python, source: str | None = None, config: PluginConfig)[source]¶
Bases:
Function
Function template.
- config: PluginConfig¶
- classmethod from_plugin(plugin: Plugin) list [source]¶
Generate a list of template functions from a plugin
- Parameters:
plugin (plugin {Plugin} -- A)
- Returns:
list – A list of template functions
- type: ConstrainedStrValue¶