Shell tasks¶
Note
Click here to see a complete example
You can run shell/bash scripts as Ploomber tasks, giving you complete flexibility to execute code in a different programming language.
Let’s say you want to use a shell script to copy a file; a sample script would look like this:
cp origin.txt target.txt
To turn that into a Ploomber shell script, you need to add placeholders
for upstream dependencies and the product. Let’s say that origin.txt
was
generated by a task called download
and that target.txt
is declared
as the product of the given task in the pipeline.yaml
file, then:
cp {{upstream['download']}} {{product}}
Then, in your pipeline.yaml
, add the task:
# ... more tasks
- source: copy.sh
product: target.txt
Executing source code in other programming languages¶
By default, Ploomber will keep track of changes to your .sh
file, so it only
executes it when it changes. In our previous
example, all the logic exists in the copy.sh
file; however, if you’re
using shell scripts to execute source in a different programming language,
say Julia, your .sh
script may look like this:
julia my-script.jl input.csv output.csv
First, we add the placeholders:
julia my-script.jl {{upstream["some-task"]}} {{product}}
The script above will work, however, Ploomber won’t track changes to my-script.jl
. To fix this,
you can use the resources_
option, which tells Ploomber to keep track
of extra files:
# ... more tasks
- source: scripts/run-julia.sh
product: output.csv
params:
resources_:
julia_script: my-script.jl
Note
To learn more about the resources_
section click here..
If you wish, you can update your script, to remove the duplicated my-script.jl
value:
julia {{params["resources_"]["julia_script"]}} {{upstream["task"]}} {{product}}