Roadmap

These are some of the features we have in the pipeline, sorted by priority. Please help us prioritize this list; go to the related GitHub issue and comment on it.

  1. Export Ploomber pipelines to Slurm. (Note: this will be implemented in Soopervisor). (This is now in beta)

  2. Improve code difference detection (i.e., detect when an imported function changes). (#111)

  3. Improve feedback in Jupyter when failing to load the pipeline (we only display feedback to the Jupyter console). (#415)

  4. Give more visibility to the Python API: better documentation, examples, etc. (#417)

  5. Building online APIs from pipelines with scripts/notebooks (we only support exporting online APIs from pipelines with Python functions). (#418)

Done

  1. Better support for non-Jupyter editors like VSCode or PyCharm (available in 0.14). User guide.

To send general feedback, open an issue or send us a message on Slack.

Ideas

These are some ideas we have that we haven’t prioritized yet.

  • A frontend Jupyter plug-in (e.g., to visualize execution status in real-time).

  • Autocompletion and linting in Jupyter when editing pipeline.yaml.

  • Automated pipeline testing.

  • Integration with data versioning tools such as LakeFS. (#414)

  • Expand integration with Google Cloud (we only support uploading to Cloud Storage).

  • Expand integration with AWS (we only support S3 and AWS Batch).

  • Integration with Azure Machine Learning services.

  • Support for Julia.

To send general feedback, open an issue or send us a message on Slack.