Workflow Metrics

Note

Dockstore Workflow Metrics is currently in preview mode. Also note that the screenshots below were taken on our staging site.

What are workflow metrics?

Workflow metrics are metrics of workflow executions on a platform. Metrics include things like the resources used during the execution, and how often workflow executions succeeded or failed.

Users are able to execute workflows on various platforms using Dockstore’s Launch With feature, shown on the right side in the screenshot below.

Platforms are able to submit metrics of workflows executed on their platform to Dockstore and we aggregate the metrics and display them in the UI to the users in the Metrics tab.

Why would I want to submit workflow metrics?

As a platform owner, workflow metrics indicate to others that your platform is compatible with many workflows on Dockstore. Workflow metrics provide valuable information to users, including information about the resources and time needed to run the workflow. It helps the user determine if the workflow is high quality and is very likely to work for others.

How do I view workflow metrics?

To view workflow metrics for a workflow on Dockstore, you will need to use the metrics feature flag.

To use the metrics feature flag, append metrics to the Dockstore URL as a query parameter. You only need to do this once, unless you refresh/close your browser.

For example, if you’re on the https://dockstore.org page, append metrics such that it looks like this: https://dockstore.org?metrics.

If you’re on a page that already contains query parameters, indicated by the presence of a question mark, append metrics to the URL using an ampersand. For example, if you’re on the https://dockstore.org/workflows/github.com/gatk-workflows/seq-format-conversion/BAM-to-Unmapped-BAM:3.0.0?tab=info page, append metrics such that it looks like https://dockstore.org/workflows/github.com/gatk-workflows/seq-format-conversion/BAM-to-Unmapped-BAM:3.0.0?tab=info&metrics.

After applying the metrics feature flag, a Metrics tab will appear when viewing a workflow on Dockstore. Click on the Metrics tab and you will see workflow metrics if they are available.

How do I submit workflow metrics?

Note

Submitting workflow metrics is only available for admins, curators, and platform partners. If you’re a platform owner and you don’t have a platform partner user, please contact us via our GitHub issues or open a helpdesk ticket on Discourse and we will help you get set up.

Go to https://dockstore.org/api/static/swagger-ui/index.html#/extendedGA4GH/executionMetricsPost. This is the endpoint used to submit metrics to Dockstore. Click the “Try it out” button.

First, fill out the query parameters. This is where you’ll input information about which workflow was executed and what platform it was executed on.

For the id parameter, provide the TRS ID of the workflow that you want to submit metrics for. For example, the dockstore-tool-bamstats workflow has the TRS ID #workflow/github.com/dockstore/dockstore-tool-bamstats/wdl as shown in the “Info” tab with the label “TRS”.

For the version_id parameter, provide the name of the workflow version that was executed. It can be any version listed in the “Versions” tab of the workflow. For example, dockstore-tool-bamstats has the following versions currently: 1.25-9 and develop. It is recommended to submit execution metrics for versions that are unlikely to change, like tags.

Select the platform that the workflow was executed on from the available values.

For the description, you may provide an optional description about the metrics you’re submitting.

In the request body, provide the workflow execution metrics that you want to submit. Click on Schema to view the schema of the request body.

You can provide a individual executions through runExecutions and validationExecutions.

A run execution is an execution that runs the workflow. The only required metric for run executions is executionStatus. If you want to provide additional metrics that are not defined in the schema, use the additionalProperties key to provide your metric.

A validation execution is an execution of a validator tool, like miniwdl, on the workflow. The following are required for a validation execution:

• validatorTool

• validatorToolVersion

• isValid

• dateExecuted

If you want to provide additional metrics that are not defined in the schema, use the additionalProperties key to provide your metric.

Instead of providing individual executions, you may also provide aggregated metrics that summarize data for multiple executions through aggregatedExecutions. There are no required fields, but you must provide at least one metric. If you want to provide aggregated metrics that are not defined in the schema, use the additionalAggregatedMetrics key to provide your metric.

Lastly, provide your Dockstore token using the lock icon at the top right of the endpoint.

Below is an example of what a request for submitting individual execution metrics looks like. The request is for a workflow that was executed on Terra. The request body submits one run execution that was successful and took 30 seconds to execute, and one validation execution of miniwdl version 1.9.1 which validated the workflow successfully.

The curl command results in something like:

curl -X 'POST' \
   'https://dockstore.org/api/api/ga4gh/v2/extended/%23workflow%2Fgithub.com%2Fdockstore%2Fdockstore-tool-bamstats%2Fwdl/versions/1.25-9/executions?platform=TERRA' \
   -H 'accept: */*' \
   -H 'Authorization: Bearer iamafakebearertoken' \
   -H 'Content-Type: application/json' \
   -d '{
   "runExecutions": [
      {
         "executionStatus": "SUCCESSFUL",
         "executionTime": "PT30S"
      }
   ],
   "validationExecutions": [
      {
         "validatorTool": "miniwdl",
         "validatorToolVersion": "1.9.1",
         "isValid": true,
         "dateExecuted": "2023-03-31T15:06:49.888745366Z"
      }
   ]
}'

If it was submitted successfully, you should receive a 204 response code.

Below is an example of what a request for submitting aggregated execution metrics looks like. The request body submits an aggregated execution status metric, indicating that the workflow was successfully executed 5 times, and it failed twice, once because it was run time invalid and once because it was semantically invalid.

The curl command results in something like:

curl -X 'POST' \
   'https://dockstore.org/api/api/ga4gh/v2/extended/%23workflow%2Fgithub.com%2Fdockstore%2Fdockstore-tool-bamstats%2Fwdl/versions/1.25-9/executions?platform=TERRA' \
   -H 'accept: */*' \
   -H 'Authorization: Bearer iamafakebearertoken' \
   -H 'Content-Type: application/json' \
   -d '{
   "aggregatedExecutions": [
      {
         "executionStatusCount": {
         "count": {
            "SUCCESSFUL": 5,
            "FAILED_RUNTIME_INVALID": 1,
            "FAILED_SEMANTIC_INVALID": 1
         },
         "numberOfSuccessfulExecutions": 5,
         "numberOfFailedExecutions": 2
         }
      }
   ]
}'

If it was submitted successfully, you should receive a 204 response code.