ploomber.executors.Parallel¶

class ploomber.executors.Parallel(processes=None, print_progress=False)¶

Runs a DAG in parallel using multiprocessing

Parameters

processes (int, default=None) – The number of processes to use. If None, uses os.cpu_count
print_progress (bool, default=False) – Whether to print progress to stdout, otherwise just log it

Examples

Spec API:

# add at the top of your pipeline.yaml
executor: parallel

Python API:

>>> from ploomber import DAG
>>> from ploomber.executors import Parallel
>>> dag = DAG(executor='parallel') # use with default values
>>> dag = DAG(executor=Parallel(processes=2)) # customize

Notes

If any task crashes, downstream tasks execution is aborted, building continues until no more tasks can be executed