Registry for (versioned) WDL files/packages

[DavyCats]

This is something which has been brought up before (#226), but there seems to have been very little discussion on this since then. Although the recent discourse about an extended library might also be related (#488).

I think it might be useful to have some system like PyPI or dockerhub in which versioned (collections of) WDL files can be stored and from which these specific versions can be imported. Something of this nature could currently be considered supported through the use of http URIs. However, having a standardized form for indicating versions could make it easier to write and read WDL files (as all the versioned imports would follow the same conventions you wouldn't have to decode the urls of various different sources) and allow for a standard protocol of localizing these WDL files for offline usage or archival purposes.

Packages

The packages themselves could essentially just be zip files (like those produced by http://github.com/biowdl/wdl-packager). They would contain one or more WDL files, potentially including sub directories and supplementary files (eg. a license or readme).
If a WDL inside the package imports a WDL file using the "file://" protocol then the path will point to some location within the same zip file.

Syntax

A syntax like used for docker images might be a versatile and straight forwards approach to take for handing import statements. Amongst the import statements one might be able to say something like the following to import a wdl file from some package:

from "registry.org/organization/package" import "imported.wdl" # retrieves the "latest" tag
from "registry.org/organization/package:tag"  import "imported.wdl" # retrieves some tagged version
from "registry.org/organization/package@sha256:12345etc." import "imported.wdl"  # retrieves a version by digest
from "registry.org/organization/package" import "subdir/imported.wdl"  # import wdl file from some subdirectory in the package

EDIT 2022/9/1
Given that the package specification proposed in #499 would make the versions immutable, I would change the terminology here from
"tag" to "version". I think digest based imports might still be useful, since SNAPSHOTS would still be mutable and a specific one could be
needed.

Registry

The registry would have to provide at least the following two endpoints:

• GET /v1/r/<organization>/<package>/tags/<tag>: returns the digest the tag points to
• GET /v1/r/<organization>/<package>/digests/<digest>: returns the zip file containing the package

An implementation could add further endpoints to retrieve additional details, such as lists of available tags and digests. But these two endpoints must always be available in order for the execution engines to be able to download packages from any registry.

EDIT 2022/9/1
Since #499 proposes that versions should be immutable. The "tag" endpoint might be replaced with a "version" endpoint instead. I'm
inclined to say that the digest endpoint might still be of use, since it also states that SNAPSHOT version can be changed.
I'm also wondering now if it might be useful to also incorporate the WDL version, allowing you to have, say, a v1 of the package for both
WDL 1.0 and WDL 1.1.
The endpoints would then end up looking like this:

• GET /v1/r/<organization>/<package>/versions/<version>/<WDL_version>: returns the digest the version points to
• GET /v1/r/<organization>/<package>/digests/<digest>: returns the zip file containing the package

Localization or "installing" packages

Execution engines could localize the packages to a directory defined by some environment variable (or some default location like ~/.wdl_packages). When a workflow imports something from a package, the execution engine could then download that package's zip, unpack it in an appropriate subdirectory and then import it from there.
eg.

.wdl_packages/
└─ registry.org/
   └─ organization/
      └─ package/
         ├─ digests/
         │  └─ digest/
         │     └─ imported.wdl
         └─ tags/
            └─ tag (link to digest?)

EDIT 2022/9/1

Building stand-alone "executable" packages

Tooling could be made to make executable packages that include all the packages that get imported. This would allow them to get
passed to an execution engine without the need for the engine to retrieve all the dependencies. Similarly to how the zips produced by
miniwdl zip can be used. This might simply be done by including the .wdl_packages/ folder inside of the package.
You wouldn't want such a package (which included all the imported packages) to get uploaded to the registry, since that would lead to a
lot of duplicate files getting stored there. But the registry could remove the .wdl_packages/ directory from the tarballs if it does get
uploaded.

[rhpvorderman]

A package registry would be a great thing for WDL. We could host the principle version at LUMC. If the packages are packaged as .tar.xz archives this should take very little space and bandwith.

Ping @mlin, since he just added a packager to miniwdl.

[patmagee]

@rhpvorderman @DavyCats I think this is a really interesting Idea. I wonder if there is an opportunity to build on existing tools like https://dockstore.org, or embed support directly for TRS (https://ga4gh.github.io/tool-registry-service-schemas/) uris in WDL files

[DavyCats]

I'm not sure how this interacts with dockstore. I don't think dockstore really provides any way to import workflows/tasks stored on it into other workflows. I might be wrong though, I'm not to familiar with how it works. I tried to add some of our biowdl workflow to dockstore at one point, but dockstore doesn't seem to support git submodules. So I kinda got stuck there, since basically all our workflow rely on at least one submodule.

This TRS does look like it could be interesting as well.

ps. my apologies for the rather slow response

[microbioticajon]

Some form of versioned wdl package management would be very welcome - the suggestion above looks nice.

If possible I would like to see a system that could prebuild these imports into the imports.zip and still be satisfied on the end server. This would allow us to prebuild the library structure ahead of time without exposing the cluster to external networks/systems.

As an aside, a pattern we have been using is for each high-level workflow to provide a lightweight entrypoint.wdl and a zip of the entire workflow and it's dependencies. These bundles are exposed to the user/system as predefined workflows. The zip can contain multiple packages managed and versioned in different git repositories. Version management and zip building is managed via npm (using a private npm registry and a package.json) and a few small helper scripts. This way all the tasks and workflows are fully reusable and can be built into different, versioned workflows upfront.

Advantages to this approach:

• lets npm manage the fetching of dependencies and version resolution/conflicts
• prebuilt zips work anywhere (provided the user/system has access to the appropriate container registries)
• relative paths within packages still work as we are bundling entire packages
• separate out actual workflows from the "entry-point" wdl file supplied to the implementing server so they can be included in reusable packages.
• our ci system can deploy versioned wdl packages to our npm registry and build and deploy workflow zips where needed.

Drawbacks to this approach:

• all our packages look like npm modules e.g. /@my_package/.../my.wdl
• for any given workflow you can only include a package with exactly one version.
• package dependencies are managed outside the wdl files in a npm package.json file which is part of the package git repository
• import paths to other packages are relative to the current wdl file i.e. import ../../../../@my_other_package/thing.wdl and are not very intuitive. Namespaces would have helped with this: One idea to make imports more intuitive #214 wdl development doesn't import files broadinstitute/cromwell#6441

The key thing for us was to prebuild the imports.zip and not rely on the clusters ability to access remote resources.

It is very likely that there was a better way to manage this.... @DavyCats suggestion above is more elegant and may alleviate many of our issues - especially if we can prebuild.

[DavyCats]

I added some comments in the initial proposal based on @rhpvorderman proposal of a WDL package specification (#499) and @microbioticajon comment above.