To run this locally, install Ploomber and execute: ploomber examples -n guides/first-pipeline
Found an issue? Let us know.
Questions? Ask us on Slack.
Your first Python pipeline¶
Introductory tutorial to learn the basics of Ploomber.
Introduction¶
Ploomber helps you build modular pipelines. A pipeline (or DAG) is a group of tasks with a particular execution order, where subsequent (or downstream tasks) use previous (or upstream) tasks as inputs.
Pipeline declaration¶
This example pipeline contains five tasks, 1-get.py, 2-profile-raw.py, 3-clean.py, 4-profile-clean.py and 5-plot.py; we declare them in a pipeline.yaml file:
# Content of pipeline.yaml
tasks:
# source is the code you want to execute (.ipynb also supported)
- source: 1-get.py
# products are task's outputs
product:
# scripts generate executed notebooks as outputs
nb: output/1-get.html
# you can define as many outputs as you want
data: output/raw_data.csv
- source: 2-profile-raw.py
product: output/2-profile-raw.html
- source: 3-clean.py
product:
nb: output/3-clean.html
data: output/clean_data.parquet
- source: 4-profile-clean.py
product: output/4-profile-clean.html
- source: 5-plot.py
product: output/5-plot.html
Note: YAML is a human-readable text format similar to JSON.
Note: Ploomber supports Python scripts, Python functions, Jupyter notebooks, R scripts, and SQL scripts.
Opening .py files as notebooks¶
Ploomber integrates with Jupyter. Among other things, it allows you to open ``.py`` files as notebooks (via jupytext).
What sets the execution order?¶
Ploomber infers the pipeline structure from your code. For example, to clean the data, we must get it first; hence, we declare the following in 3-clean.py:
# 3-clean.py
# this tells Ploomber to execute the '1-get' task before '3-clean'
upstream = ['1-get']
Plotting the pipeline¶
[1]:
%%bash
ploomber plot
Loading pipeline...
Plot saved at: pipeline.png
100%|██████████| 5/5 [00:00<00:00, 15477.14it/s]
[2]:
from IPython.display import Image
Image(filename='pipeline.png')
[2]:
You can see that our pipeline has a defined execution order.
Note: This is a sample predefined five-task pipeline, Ploomber can manage arbitrarily complex pipelines and dependencies among tasks.
Running the pipeline¶
[3]:
%%bash
# takes a few seconds to finish
ploomber build
Loading pipeline...
name Ran? Elapsed (s) Percentage
--------------- ------ ------------- ------------
3-clean True 5.43922 28.676
4-profile-clean True 4.76502 25.1215
5-plot True 8.76362 46.2025
1-get False 0 0
2-profile-raw False 0 0
Building task '3-clean': 0%| | 0/3 [00:00<?, ?it/s]
Executing: 0%| | 0/9 [00:00<?, ?cell/s]
Executing: 11%|█ | 1/9 [00:01<00:11, 1.46s/cell]
Executing: 100%|██████████| 9/9 [00:05<00:00, 1.79cell/s]
Building task '4-profile-clean': 33%|███▎ | 1/3 [00:05<00:10, 5.44s/it]
Executing: 0%| | 0/7 [00:00<?, ?cell/s]
Executing: 14%|█▍ | 1/7 [00:01<00:06, 1.04s/cell]
Executing: 43%|████▎ | 3/7 [00:02<00:03, 1.25cell/s]
Executing: 71%|███████▏ | 5/7 [00:03<00:01, 1.58cell/s]
Executing: 86%|████████▌ | 6/7 [00:03<00:00, 1.87cell/s]
Executing: 100%|██████████| 7/7 [00:04<00:00, 1.59cell/s]
Building task '5-plot': 67%|██████▋ | 2/3 [00:10<00:05, 5.04s/it]
Executing: 0%| | 0/8 [00:00<?, ?cell/s]
Executing: 12%|█▎ | 1/8 [00:02<00:17, 2.46s/cell]
Executing: 62%|██████▎ | 5/8 [00:02<00:01, 2.18cell/s]
Executing: 75%|███████▌ | 6/8 [00:05<00:01, 1.04cell/s]
Executing: 100%|██████████| 8/8 [00:08<00:00, 1.05s/cell]
Building task '5-plot': 100%|██████████| 3/3 [00:18<00:00, 6.32s/it]
This pipeline saves all the output in the output/ directory; we have the output notebooks and data files:
[4]:
%%bash
ls output
1-get.html
2-profile-raw.html
3-clean.html
4-profile-clean.html
5-plot.html
clean_data.parquet
raw_data.csv
Updating the pipeline¶
Ploomber automatically caches your pipeline’s previous results and only runs tasks that changed since your last execution.
Execute the following to modify the 3-clean.py script
[5]:
from pathlib import Path
path = Path('3-clean.py')
clean = path.read_text()
# add a print statement at the end of 3-clean.py
path.write_text(clean + """
print("hello")
""")
[5]:
397
Execute the pipeline again:
[6]:
%%bash
# takes a few seconds to finish
ploomber build
Loading pipeline...
name Ran? Elapsed (s) Percentage
--------------- ------ ------------- ------------
3-clean True 2.22814 14.2248
4-profile-clean True 4.7084 30.0591
5-plot True 8.72726 55.7161
1-get False 0 0
2-profile-raw False 0 0
Building task '3-clean': 0%| | 0/3 [00:00<?, ?it/s]
Executing: 0%| | 0/9 [00:00<?, ?cell/s]
Executing: 11%|█ | 1/9 [00:01<00:10, 1.29s/cell]
Executing: 100%|██████████| 9/9 [00:01<00:00, 5.04cell/s]
Building task '4-profile-clean': 33%|███▎ | 1/3 [00:02<00:04, 2.23s/it]
Executing: 0%| | 0/7 [00:00<?, ?cell/s]
Executing: 14%|█▍ | 1/7 [00:00<00:05, 1.10cell/s]
Executing: 43%|████▎ | 3/7 [00:02<00:03, 1.23cell/s]
Executing: 71%|███████▏ | 5/7 [00:03<00:01, 1.58cell/s]
Executing: 86%|████████▌ | 6/7 [00:03<00:00, 1.91cell/s]
Executing: 100%|██████████| 7/7 [00:04<00:00, 1.60cell/s]
Building task '5-plot': 67%|██████▋ | 2/3 [00:06<00:03, 3.69s/it]
Executing: 0%| | 0/8 [00:00<?, ?cell/s]
Executing: 12%|█▎ | 1/8 [00:02<00:16, 2.39s/cell]
Executing: 62%|██████▎ | 5/8 [00:02<00:01, 2.24cell/s]
Executing: 75%|███████▌ | 6/8 [00:05<00:01, 1.03cell/s]
Executing: 100%|██████████| 8/8 [00:08<00:00, 1.05s/cell]
Building task '5-plot': 100%|██████████| 3/3 [00:15<00:00, 5.22s/it]
[7]:
# restore contents
path.write_text(clean)
[7]:
381
You’ll see that 1-get.py & 2-profile-raw.py didn’t run because it was not affected by the change!
Where to go from here¶
Bring your own code! Check out the tutorial to migrate your code to Ploomber.
Have questions? Ask us anything on Slack.
Want to dig deeper into Ploomber’s core concepts? Check out the basic concepts tutorial.
Want to start a new project quickly? Check out how to get examples.