This tutorial is a continuation of Getting Started with Dockstore Tools. Please complete the tutorial prior to doing this one.
- Learn the differences between tools and workflows across Descriptor Languages
- Register a workflow on Dockstore
- Publish your workflow
This tutorial walks through the process of registering and sharing more complex workflows which are comprised of multiple tools (whether they are registered on Dockstore or not). Workflows as defined via the Dockstore are a composition of multiple tools, strung together in some sort of order (often a directed acyclic graph (DAG)). Workflows also are different from tools since they are not required to define their own environment, instead a workflow engine like Arvados or Cromwell will provide the ability to execute a CWL or WDL workflow respectively.
Comparison of Tools and Workflows Across Descriptor Languages¶
When Dockstore was created, CWL was the first descriptor language we supported. It had a very clear distinction between a Tool and a Workflow. Descriptor languages like WDL and Nextflow are less clear about this distinction so we briefly describe our working definitions below:
Prepare Your Workflow for Dockstore¶
This tutorial does not go through the creation of a workflow and its registration to GitHub, Bitbucket or GitLab. It assumes that you already have a repository which contains a workflow and are now trying to register it in Dockstore. There are some ways to make the registration process more seamless.
- For your primary workflow descriptor, use the filename
nextflow.configdepending on the descriptor language at the root of your repository
- For your test parameter files, use the filename
test.jsonat the root of your repository
- There should be one workflow per repository
By default, Dockstore will search the root of your repository for workflow related files. Following the above tips will help streamline the registration process, though you can still register workflows with non-standard format.
Register Your Workflow in Dockstore¶
There are a variety of ways to get your workflows into Dockstore. There are two types of quick registration and one type of manual registration. These quick methods create workflow stubs, which are simply workflows that have not yet been confirmed to be real workflows. You can individually refresh a workflow stub that you identify as a real workflow in order to start pulling in workflow files and other data.
To register content on Dockstore, you must have an account on Dockstore and link the necessary third-party accounts. Once this is done you can register workflows from the My Workflows page.
Quick Registration via the Web UI¶
Quick registration is best used for workflows that follow the simple format that Dockstore suggests. It can still be used if your workflows are non-standard format, however there can be some drawbacks.
Some users have multiple workflows within one Git repository, however each workflow entry on Dockstore only contains a single workflow. This is a problem as the Git path is used to uniquely identify a Dockstore workflow. The solution to this is to allow users to specify a workflow name that is appended to the Dockstore path. This would allow them to have multiple Dockstore workflows with the same Git repository. Quick registration does not allow you to create workflows with workflow names. To do that you must do manual registration, which is described later.
Refresh all will look at all of your third-party accounts and do the following:
- Create stub workflows for all Git repositories which do not exist on Dockstore
- Refresh all workflows that have been converted from stub to full workflows
- Add user to any workflows that exist on Dockstore that they should have access to
To refresh all, select the refresh all icon button on the left side of the My Workflows page.
For existing organizations, you can select Refresh Organization to perform a refresh all on a specific organization.
One drawback of refresh all is that it creates a stub for every single repository that you have access to, whether or not it contains a workflow. The following approach is a bit less automated but allows for more control.
Quick register provides a flow that lets you browse the repositories you have access to and quickly create standard stub workflows. The benefit of this approach is that you get some automation without having lots of stubs created.
You can access quick register by clicking the plus button on the My Workflows page. The flow of this process is shown in the screenshots below.
Once you’ve selected a Git registry and organization, you can see a list of all available repositories that you can add to Dockstore. There are three states the sliders can be in.
- Off - There is no matching workflow on Dockstore.
- On - This repository already exists on Dockstore and can be deleted.
- Disabled - This repository exists on Dockstore and cannot be deleted.
If sliders are in the off state then you can turn them on to quickly register a stub workflow for the repository.
Manual Registration of Workflows¶
In certain cases, you may wish to register workflows in a different source code structure, especially when working with complex project structures. For example, if you want to register two workflows from the same repository.
You can access manual register by clicking the plus button on the My Workflows page. The flow of this process is shown in the screenshots below.
Source Code Provider allows you to choose between GitHub, BitBucket, and
GitLab (your respective accounts for these third party repositories need
to be linked to your Dockstore account). The Source Code Repository
field must be filled out and is in the format
two paths may differ). The Workflow (descriptor) path and test parameter
path are relative to the root of the Source Code Repository (and must
begin with ‘/’). These will be the default locations to find their
corresponding files, unless specified otherwise in the tags. The
Workflow Name is an optional ‘suffix’ appended to the Dockstore path. It
allows for two workflows to share the same Git paths; the Workflow Name
uniquely distinguishes workflow repositories in Dockstore.
Upon successful submission and publishing of the workflow, a resynchronization call will be made to fetch all available data from the given sources.
The user may then browse to the ‘Versions’ tab of the new container, where tags (corresponding to GitHub/Bitbucket/GitLab tag names) may be edited.
The fields in the form should correspond to the actual values on
GitHub/Bitbucket/GitLab in order for the information to be useful to
other users. Selecting
Hidden will prevent the tag from appearing in
the public listing of tags for the workflow.
dockstore command line has several options. When working with
dockstore workflow to get a full list of options. We
recommend you first use
dockstore workflow refresh to ensure the
latest GitHub, Bitbucket, and GitLab information is indexed properly.
You can then use
dockstore workflow publish to see the list of
available workflows you can register with Dockstore and then register
them. This is for you to publish workflows with the simplest structure.
For now, use manual registration if your workflow has a different
structure. The key is that workflows you wish to (simply) publish have
the following qualities:
- at least one valid tag. In order to be valid, a tag has to:
- have the reference be linked a corresponding
Dockstore.wdlhosted at the root of the repository
- have the reference be linked a corresponding
dockstore workflow manual_publish command can be used to
manually register a workflow on GitHub, Bitbucket or GitLab. Its usage
is outlined in the manual_publish help menu.
Find Other Workflows¶
You can find tools on the Dockstore website or also through the
dockstore workflow search command line option.