Shell tasks¶

Note

Click here to see a complete example

You can run shell/bash scripts as Ploomber tasks, giving you complete flexibility to execute code in a different programming language.

Let’s say you want to use a shell script to copy a file; a sample script would look like this:

cp origin.txt target.txt

To turn that into a Ploomber shell script, you need to add placeholders for upstream dependencies and the product. Let’s say that origin.txt was generated by a task called download and that target.txt is declared as the product of the given task in the pipeline.yaml file, then:

cp {{upstream['download']}} {{product}}

Then, in your pipeline.yaml, add the task:

# ... more tasks

- source: copy.sh
  product: target.txt

Executing source code in other programming languages¶

By default, Ploomber will keep track of changes to your .sh file, so it only executes it when it changes. In our previous example, all the logic exists in the copy.sh file; however, if you’re using shell scripts to execute source in a different programming language, say Julia, your .sh script may look like this:

julia my-script.jl input.csv output.csv

First, we add the placeholders:

julia my-script.jl {{upstream["some-task"]}} {{product}}

The script above will work, however, Ploomber won’t track changes to my-script.jl. To fix this, you can use the resources_ option, which tells Ploomber to keep track of extra files:

# ... more tasks

- source: scripts/run-julia.sh
  product: output.csv
  params:
    resources_:
        julia_script: my-script.jl

Note

To learn more about the resources_ section click here..

If you wish, you can update your script, to remove the duplicated my-script.jl value:

julia {{params["resources_"]["julia_script"]}} {{upstream["task"]}} {{product}}

Contents

Shell tasks¶

Executing source code in other programming languages¶