Verification
Important
Verification will eventually be replaced by Workflow Metrics. Workflow Metrics achieve the same goal as verification by tracking the platforms that the workflow was successfully run on. Workflow Metrics are preferred because it is a more scalable way of tracking that a workflow was successfully run on a platform.
What is a verified tool/workflow?
A verified tool/workflow indicates that it was successfully run and verified by either:
our team
an outside party
Historically, the majority of tool validation has been done by the docktesters team currently headed by Miguel Vazquez and formerly headed by Francis Ouellette.
We also strive to use this to highlight tools that share a common set of recommended characteristics:
tools should include a description and an author
tools should include a README.md or similar in their source repo describing any other relevant information about the tool
tools should include at least one test parameter file indicating how to run the tool on some sample data
the Dockerfile should be helpful in reconstructing how a tool was built from source
tools and/or their reference data should be publically available
Why would I want to verify?
There are several reasons why you would want a tool/workflow to be verified. If you’re a platform owner, verifying would indicate to others that your platform is compatible with many tools/workflows on Dockstore so others will be more likely to use your platform. If you’re tool/workflow developer, verifying would assure others that your tool/workflow is of high quality and is very likely to work for others.
How do I verify?
If you are an admin/curator, follow the Verification Process section in this document. If you are not an admin/curator, please send us a heads-up via our GitHub issues or discourse!
How do I tell if a tool/workflow is verified?
There are 3 new indicators on Dockstore.org that indicates whether or not the tool/workflow is verified.
First, go to the page of the tool/workflow such as https://dockstore.org/containers/quay.io/briandoconnor/dockstore-tool-md5sum:1.0.4?tab=info. Since this tool/workflow is verified, 3 indicators can be seen:
At the top left, the checkmark indicates that at least one of the tool/workflow’s version has been verified. As a whole, this tool/workflow is considered verified.
At the top right, the recent versions of the tool/workflow are listed. There is a checkmark if a specific version of the tool/workflow is verified.
The bottom right shows whether the currently selected/viewed version is verified. The selected version is indicated in the URL as well as the title (e.g. quay.io/briandoconnor/dockstore-tool-md5sum:1.0.4). In this example, the version 1.0.4 is selected/viewed. This bottom right verification box contains more details such as the platform and verifier. This example shows that “Dockstore CLI” is the platform and “Phase 1 GA4GH Tool Execution Challenge” is the verifier.
Additional information for all verified versions can be viewed at a glance in the versions tab:
Once again, the checkmarks indicate the version is verified. Platforms which the version was verified on are displayed to the right of it. In this case, it’s “Dockstore CLI”.
Verification Process
Note
Verifying is only available for admins and curators. Please contact one if you want your tool/workflow to be verified.
Go to https://dockstore.org/api/static/swagger-ui/index.html#/extendedGA4GH/verifyTestParameterFilePost
Click “Try it out”
Provide a “type”. See the description for allowable values.
Provide the TRS ID for the tool/workflow being verified. For example, the dockstore-tool-md5sum tool has the TRS ID: “quay.io/briandoconnor/dockstore-tool-md5sum” as shown in the “Info” tab with the label: “TRS CWL” or “TRS WDL”
Provide the version_id of the tool/workflow to verify. It can be any version listed in the Version tab of the tool/workflow. dockstore-tool-md5sum has the following versions currently: 1.0.4, master, develop, 1.0.3, latest, 1.0.2, 1.0.1, and 1.0.0. It is recommended to only verify versions that are unlikely to change (tags).
Provide the “relative_path” of the test parameter file being verified. The path of the test parameter file is relative to the primary descriptor. This path can be found using the files endpoint or by viewing the files tab of a tool/workflow such as: https://dockstore.org/workflows/github.com/DataBiosphere/topmed-workflows/UM_variant_caller_wdl:1.32.0?tab=files and then further selecting the Test Parameter Files tab and view the right-most “File” dropdown. This relative path must be a test parameter file, providing a descriptor will not work.
Provide the “platform”. Some examples are: HCA, Cromwell, Arvados, etc.
Select the “verified” status either as “true” or “false”. Use “true” to verify, “false” to “unverify”.
Provide “metadata”, this is typically the verifier’s identity which can be something like “GA4GH/DREAM Challenge”
Lastly, provide your Dockstore token using the lock icon at the top right of the endpoint
Below is a screenshot of someone verifying the “test.json” test parameter file of the “master” version of the “dockstore-tool-md5sum” tool.
The curl command results in something like:
curl -X POST "https://dockstore.org/api/api/ga4gh/v2/extended/quay.io%2Fbriandoconnor%2Fdockstore-tool-md5sum/versions/master/CWL/tests/test.json?platform=Dockstore%20CLI&verified=true&metadata=Phase%201%20GA4GH%20Tool%20Execution%20Challenge" -H "accept: application/json" -H "Authorization: Bearer iamafakebearertoken"
A successful response will result in something like:
{
"Dockstore CLI": {
"metadata": "Phase 1 GA4GH Tool Execution Challenge",
"verified": true
}
}
Additional Verification Information
To see more verification information about a specific version, first select the version.
Then click “More Info” in the “Verification and Logs” panel in the bottom right.
A popup will appear:
It lists the platform it was verified on, the platform version, test parameter file that was used, and metadata (verifier). Below it, there may be an additional Logs section which contains information from Dockstore ToolTester.
Dockstore ToolTester
Dockstore ToolTester is a semi-automated process where Dockstore will attempt to launch certain verified tools/workflows through the latest Dockstore CLI. Typically this launching process occurs before a Dockstore CLI release and is done so in order to ensure compatibility. The logs contain much useful information:
Dockstore CLI version used
pip packages installed
version of the tool/workflow that was launched
time when launched
runner that was used (cromwell, cwltool, etc)
files used (which descriptor file, which test parameter file)