To run this example locally, install Ploomber and execute: ploomber examples -n guides/first-pipeline

To start a free, hosted JupyterLab: binder-logo

Found an issue? Let us know.

Have questions? Ask us anything on Slack.

Your first Python pipeline

Introductory tutorial to learn the basics of Ploomber.

Introduction

Ploomber helps you build modular pipelines. A pipeline (or DAG) is a group of tasks with a particular execution order, where subsequent (or downstream tasks) use previous (or upstream) tasks as inputs.

Pipeline declaration

This example pipeline contains five tasks, 1-get.py, 2-profile-raw.py, 3-clean.py, 4-profile-clean.py and 5-plot.py; we declare them in a pipeline.yaml file:

# Content of pipeline.yaml
tasks:
   # source is the code you want to execute (.ipynb also supported)
  - source: 1-get.py
    # products are task's outputs
    product:
      # scripts generate executed notebooks as outputs
      nb: output/1-get.html
      # you can define as many outputs as you want
      data: output/raw_data.csv

  - source: 2-profile-raw.py
    product: output/2-profile-raw.html

  - source: 3-clean.py
    product:
      nb: output/3-clean.html
      data: output/clean_data.parquet

  - source: 4-profile-clean.py
    product: output/4-profile-clean.html

  - source: 5-plot.py
    product: output/5-plot.html

Note: YAML is a human-readable text format similar to JSON.

Note: Ploomber supports Python scripts, Python functions, Jupyter notebooks, R scripts, and SQL scripts.

Opening .py files as notebooks

Ploomber integrates with Jupyter. Among other things, it allows you to open ``.py`` files as notebooks (via jupytext).

lab-open-with-nb

What sets the execution order?

Ploomber infers the pipeline structure from your code. For example, to clean the data, we must get it first; hence, we declare the following in 3-clean.py:

# 3-clean.py

# this tells Ploomber to execute the '1-get' task before '3-clean'
upstream = ['1-get']

Plotting the pipeline

[1]:
%%bash
ploomber plot
Loading pipeline...
Plot saved at: pipeline.png
100%|██████████| 5/5 [00:00<00:00, 15477.14it/s]
[2]:
from IPython.display import Image
Image(filename='pipeline.png')
[2]:
../_images/get-started_first-pipeline_5_0.png

You can see that our pipeline has a defined execution order.

Note: This is a sample predefined five-task pipeline, Ploomber can manage arbitrarily complex pipelines and dependencies among tasks.

Running the pipeline

[3]:
%%bash
# takes a few seconds to finish
ploomber build
Loading pipeline...
name             Ran?      Elapsed (s)    Percentage
---------------  ------  -------------  ------------
3-clean          True          5.43922       28.676
4-profile-clean  True          4.76502       25.1215
5-plot           True          8.76362       46.2025
1-get            False         0              0
2-profile-raw    False         0              0
Building task '3-clean':   0%|          | 0/3 [00:00<?, ?it/s]
Executing:   0%|          | 0/9 [00:00<?, ?cell/s]
Executing:  11%|█         | 1/9 [00:01<00:11,  1.46s/cell]
Executing: 100%|██████████| 9/9 [00:05<00:00,  1.79cell/s]
Building task '4-profile-clean':  33%|███▎      | 1/3 [00:05<00:10,  5.44s/it]
Executing:   0%|          | 0/7 [00:00<?, ?cell/s]
Executing:  14%|█▍        | 1/7 [00:01<00:06,  1.04s/cell]
Executing:  43%|████▎     | 3/7 [00:02<00:03,  1.25cell/s]
Executing:  71%|███████▏  | 5/7 [00:03<00:01,  1.58cell/s]
Executing:  86%|████████▌ | 6/7 [00:03<00:00,  1.87cell/s]
Executing: 100%|██████████| 7/7 [00:04<00:00,  1.59cell/s]
Building task '5-plot':  67%|██████▋   | 2/3 [00:10<00:05,  5.04s/it]
Executing:   0%|          | 0/8 [00:00<?, ?cell/s]
Executing:  12%|█▎        | 1/8 [00:02<00:17,  2.46s/cell]
Executing:  62%|██████▎   | 5/8 [00:02<00:01,  2.18cell/s]
Executing:  75%|███████▌  | 6/8 [00:05<00:01,  1.04cell/s]
Executing: 100%|██████████| 8/8 [00:08<00:00,  1.05s/cell]
Building task '5-plot': 100%|██████████| 3/3 [00:18<00:00,  6.32s/it]

This pipeline saves all the output in the output/ directory; we have the output notebooks and data files:

[4]:
%%bash
ls output
1-get.html
2-profile-raw.html
3-clean.html
4-profile-clean.html
5-plot.html
clean_data.parquet
raw_data.csv

Updating the pipeline

Ploomber automatically caches your pipeline’s previous results and only runs tasks that changed since your last execution.

Execute the following to modify the 3-clean.py script

[5]:
from pathlib import Path

path = Path('3-clean.py')
clean = path.read_text()

# add a print statement at the end of 3-clean.py
path.write_text(clean + """
print("hello")
""")
[5]:
397

Execute the pipeline again:

[6]:
%%bash
# takes a few seconds to finish
ploomber build
Loading pipeline...
name             Ran?      Elapsed (s)    Percentage
---------------  ------  -------------  ------------
3-clean          True          2.22814       14.2248
4-profile-clean  True          4.7084        30.0591
5-plot           True          8.72726       55.7161
1-get            False         0              0
2-profile-raw    False         0              0
Building task '3-clean':   0%|          | 0/3 [00:00<?, ?it/s]
Executing:   0%|          | 0/9 [00:00<?, ?cell/s]
Executing:  11%|█         | 1/9 [00:01<00:10,  1.29s/cell]
Executing: 100%|██████████| 9/9 [00:01<00:00,  5.04cell/s]
Building task '4-profile-clean':  33%|███▎      | 1/3 [00:02<00:04,  2.23s/it]
Executing:   0%|          | 0/7 [00:00<?, ?cell/s]
Executing:  14%|█▍        | 1/7 [00:00<00:05,  1.10cell/s]
Executing:  43%|████▎     | 3/7 [00:02<00:03,  1.23cell/s]
Executing:  71%|███████▏  | 5/7 [00:03<00:01,  1.58cell/s]
Executing:  86%|████████▌ | 6/7 [00:03<00:00,  1.91cell/s]
Executing: 100%|██████████| 7/7 [00:04<00:00,  1.60cell/s]
Building task '5-plot':  67%|██████▋   | 2/3 [00:06<00:03,  3.69s/it]
Executing:   0%|          | 0/8 [00:00<?, ?cell/s]
Executing:  12%|█▎        | 1/8 [00:02<00:16,  2.39s/cell]
Executing:  62%|██████▎   | 5/8 [00:02<00:01,  2.24cell/s]
Executing:  75%|███████▌  | 6/8 [00:05<00:01,  1.03cell/s]
Executing: 100%|██████████| 8/8 [00:08<00:00,  1.05s/cell]
Building task '5-plot': 100%|██████████| 3/3 [00:15<00:00,  5.22s/it]
[7]:
# restore contents
path.write_text(clean)
[7]:
381

You’ll see that 1-get.py & 2-profile-raw.py didn’t run because it was not affected by the change!

Where to go from here

Bring your own code! Check out the tutorial to migrate your code to Ploomber.

Have questions? Ask us anything on Slack.

Want to dig deeper into Ploomber’s core concepts? Check out the basic concepts tutorial.

Want to start a new project quickly? Check out how to get examples.