Is there a way to pass variables between steps in the pipeline?

[lucasg1]

Hi everyone!

My company is starting to create a MLOps process and I found Elyra to be a nice tool for us to build easy pipelines. I know that it is possible to pass information between steps in the pipeline usings files (txt, csv, etc...) but I really want to use variables directly instead (strings, vectors, etc...).

Is there a way to do that?

I really tried to search this topic elsewhere, didn't find anything...

[ptitzler]

Hi @lucasg1, with custom components you can share data between nodes. We have a tutorial for Kubeflow Pipelines and Apache Airflow that outlines the basics. Generic components (those that execute notebooks or scripts) currently don't support that.

[lucasg1]

Oh, okay! I was trying to find a way to share data between components that execute notebooks, but since this feature is not available I'll dig into creating a custom component. Thanks!

[ptitzler]

Sorry! One of the reasons why we don't have this kind of support at the moment is that it would require the introduction of an interface layer between the notebook/script and Elyra that allows for the declaration of inputs and outputs (similar to what is already available for KFP components and Airflow operators) and the exchange of data.