Exportation of metadata instances in Elyra

[akchinSTC]

Scoping!

[kevin-bates]

Some initial thoughts:
I think exporting instances of a given schema (vs schemaspace) is consistent with the install options. If there is a single schema in the given schemaspace, then the --schema_name parameter is not required.

For starters, I think exporting the instances to a given directory is appropriate.
Here's a prototype command:

elyra-metadata export runtimes --schema_name=kfp --directory=/tmp/foo

or, for single-schema schemaspaces...

elyra-metadata export code-snippets --directory=/tmp/foo

I like the idea of exports being idempotent, so, although the directory should not have to exist (create on the fly), we should tolerate existing files that are overwritten. (Yes, import will need to check the instance to ensure it matches the desired schema name.)

Down the road, we will likely want the ability to archive instances with a --archive=/tmp/foo.tar or something like that instead of --directory (so the two options would be mutually exclusive).

[kevin-bates]

Some more thoughts that might help with long-term planning. (I know this is outside the scope of this discussion, but having an idea of what the end-game is might be helpful from a planning perspective.)

• By having the output directory be able to contain mixed-schema and schemaspace instances, one could easily build a command that exports all metadata instances to the directory. (e.g., elyra-metadata export --all --directory=/tmp/foo).

• We could also build a manifest file (MANIFEST.ini?) that contains checksums for each file (e.g., elyra-metadata export --checksum ...). Imports would honor the well-known file and apply checksum checks for those files referenced in the manifest.

• Import will likely only support single-schema targets, primarily so one could adjust the inbound instance-data by overriding specific properties on the command line, leveraging changes in #2378 (e.g., elyra-metadata import runtimes --schema_name=kfp --directory=/tmp/foo --cos_authentication=/tmp/my_local_cos_auth.json)

• We could add support for --from, --to (or --src, --dest) options (to all elyra-metadata commands) to, instead, interact with live systems. This would enable the ability to administrate from a single location ...

elyra-metadata replicate runtimes --schema_name=kfp --from=http://foo.com:8888 --to=http://bar.com --cos_authentication=/tmp/bar_auth.json

elyra-metadata list runtimes --from=http://foo.com:8888 

elyra-metadata install runtimes --schema_name=kfp --replace --to=http://bar.com --cos_authentication=/tmp/bar_auth.json

[ptitzler]

We could add support for --from, --to (or --src, --dest) options (to all elyra-metadata commands) to, instead, interact with live systems. This would enable the ability to administrate from a single location ...

I do like the idea but am concerned about the security implications if anonymous access to the API is allowed. For example, an unauthorized user would be able to obtain sensitive information such as passwords and API keys, if the JupyterLab port is open.

[kevin-bates]

Yes, that would need to be addressed in some manner - I agree.

[ptitzler]

Update for export command after today's discussion in the dev meeting:

elyra-metadata export <schemaspace> [--schema_name=<schema>] [--valid-only] --directory=<path>

• <schemaspace> (required): one of the supported schemaspaces, like runtimes, code-snippets, or runtime-images
• schema_name (optional): if supplied and applicable, only instances of the provided <schema> are exported
• valid-only (optional): if specified, only valid instances are exported. The default is to export all instances, which is identical to the behavior of the elyra-metadata list command
• directory (required):
  • local file system path where the exported metadata will be stored
  • the specified directory is created if it does not exist
  • the exported <schemaspace> metadata is stored in a subdirectory of <path>, named after the <schemaspace>, e.g. <path>/runtimes or <path>/code-snippets.
  • The export format is the [JSON] format that is used internally to store metadata.

Examples

• Export all (valid and invalid) runtime configurations to /tmp/foo

  elyra-metadata export runtimes --directory=/tmp/foo
  

  This command creates the /tmp/foo/runtimes directory and stores all valid and invalid runtime configuration instances in that directory.

• Export all Kubeflow Pipelines runtime configurations to /tmp/foo

  elyra-metadata export runtimes --schema_name=kfp --directory=/tmp/foo
  

  This command creates the /tmp/foo/runtimes directory and stores all valid and invalid Kubeflow Pipelines runtime configuration instances in that directory.

• Export all valid Kubeflow Pipelines runtime configurations to /tmp/foo

  elyra-metadata export runtimes --schema_name=kfp --valid-only --directory=/tmp/foo
  

  This command creates the /tmp/foo/runtimes directory and stores all valid Kubeflow Pipelines runtime configuration instances in that directory.

[ptitzler]

Regarding "export all instances for all schemaspaces", how about making <schemaspace> optional, e.g.

elyra-metadata export  [--valid-only] --directory=<path>

Doing so would imply that the schema_name parameter is only valid if <schemaspace> was specified.

Examples

• Export all valid and invalid instances

  elyra-metadata export --directory=/tmp/foo
  

  This command creates the /tmp/foo directory and a subdirectory for each schemaspace, like /tmp/foo/runtimes, /tmp/foo/code-snippets. Each schemaspace subdirectory contains a JSON file for each exported instance.

• Export all valid instances

  elyra-metadata export --valid-only --directory=/tmp/foo
  

  This command creates the /tmp/foo directory and a subdirectory for each schemaspace, like /tmp/foo/runtimes, /tmp/foo/code-snippets. Each schemaspace subdirectory contains a JSON file for each exported valid instance.

[kevin-bates]

Thanks @ptitzler - I like this. One concern I have about not providing a schemaspace is that folks may accidentally export their entire system of metadata and I wonder if an explicit indicator would be better (like --all-schemaspaces).

The other issue is more from a design/implementation perspective and independent of how "all schemaspaces" is conveyed. I believe the way the class hierarchy works (one class per action) is that the schemaspace may be a parameter to the constructor. This implies it may be difficult to decouple that, but I'm sure there are ways to address it - just not sure about the degree of difficulty in doing so. (Just looked at the code, each schemaspace is essentially a subcommand so supporting "all schemaspaces" will not be straightforward.)

[ptitzler]

Metadata export using the CLI is now fully exported. Refer to the documentation for details.