ploomber.executors.Serial¶

class ploomber.executors.Serial(build_in_subprocess=True, catch_exceptions=True, catch_warnings=True)¶

Executor than runs one task at a time

Parameters

build_in_subprocess (bool, optional) – Determines whether tasks should be executed in a subprocess or in the current process. For pipelines with a lot of PythonCallables loading large objects such as pandas.DataFrame, this option is recommended as it guarantees that memory will be cleared up upon task execution. Defaults to True.
catch_exceptions (bool, optional) – Whether to catch exceptions raised when building tasks and running hooks. If True, exceptions are collected and displayed at the end, downstream tasks of failed ones are aborted (not executed at all), If any task raises a DAGBuildEarlyStop exception, the final exception raised will be of such type. If False, no catching is done, on_failure won’t be executed and task status will not be updated and tracebacks from build and hooks won’t be logged. Setting of to False is only useful for debugging purposes.
catch_warnings (bool, optional) – If True, the executor catches all warnings raised by tasks and displays them at the end of execution. If catch_exceptions is True and there is an error building the DAG, capture warnings are still shown before raising the collected exceptions.

Examples

Spec API:

# add at the top of your pipeline.yaml
executor: serial

tasks:
  - source: script.py
    nb_product_key: [nb_ipynb, nb_html]
    product:
        nb_ipynb: nb.ipynb
        nb_html: report.html

Python API:

>>> from ploomber import DAG
>>> from ploomber.executors import Serial
>>> dag = DAG(executor='parallel') # use with default values
>>> dag = DAG(executor=Serial(build_in_subprocess=False)) # customize

DAG can exit gracefully on function tasks (PythonCallable):

>>> from ploomber.products import File
>>> from ploomber.tasks import PythonCallable
>>> from ploomber.exceptions import DAGBuildEarlyStop
>>> # A PythonCallable function that raises DAGBuildEarlyStop
>>> def early_stop_root(product):
...     raise DAGBuildEarlyStop('Ending gracefully')

>>> # Since DAGBuildEarlyStop is raised, DAG will exit gracefully.
>>> dag = DAG(executor='parallel')
>>> t = PythonCallable(early_stop_root, File('file.txt'), dag=dag)
>>> dag.build()  

DAG can also exit gracefully on notebook tasks:

>>> from pathlib import Path
>>> from ploomber.tasks import NotebookRunner
>>> from ploomber.products import File
>>> def early_stop():
...     raise DAGBuildEarlyStop('Ending gracefully')

>>> # Use task-level hook "on_finish" to exit DAG gracefully.
>>> dag = DAG(executor='parallel')
>>> t = NotebookRunner(Path('nb.ipynb'), File('report.html'), dag=dag)
>>> t.on_finish = early_stop
>>> dag.build()  

Contents

ploomber.executors.Serial¶