ploomber.executors.Serial

class ploomber.executors.Serial(build_in_subprocess=True, catch_exceptions=True, catch_warnings=True)

Executor than runs one task at a time

Parameters
  • build_in_subprocess (bool, optional) – Determines whether tasks should be executed in a subprocess or in the current process. For pipelines with a lot of PythonCallables loading large objects such as pandas.DataFrame, this option is recommended as it guarantees that memory will be cleared up upon task execution. Defaults to True.

  • catch_exceptions (bool, optional) – Whether to catch exceptions raised when building tasks and running hooks. If True, exceptions are collected and displayed at the end, downstream tasks of failed ones are aborted (not executed at all), If any task raises a DAGBuildEarlyStop exception, the final exception raised will be of such type. If False, no catching is done, on_failure won’t be executed and task status will not be updated and tracebacks from build and hooks won’t be logged. Setting of to False is only useful for debugging purposes.

  • catch_warnings (bool, optional) – If True, the executor catches all warnings raised by tasks and displays them at the end of execution. If catch_exceptions is True and there is an error building the DAG, capture warnings are still shown before raising the collected exceptions.

Examples

Spec API:

# add at the top of your pipeline.yaml
executor: serial

Python API:

>>> from ploomber import DAG
>>> from ploomber.executors import Serial
>>> dag = DAG(executor='parallel') # use with default values
>>> dag = DAG(executor=Serial(build_in_subprocess=False)) # customize

See also

ploomber.executors.Parallel

Parallel executor

Methods