Automatic Syncing with GitHub Apps and .dockstore.yml – Overview

Note

Summary of GitHub App usage:

  1. Install Dockstore GitHub App

  2. Put a valid .dockstore.yml file into the top level of your repository

  3. Push a new commit with a .dockstore.yml file present

  4. Wait up to 5 minutes for Dockstore to process the new version

  5. See workflow on Dockstore

Our docs sometimes use “GitHub App registration” and “registration with a .dockstore.yml file” interchangeably, as they are equivalent terms. The Dockstore GitHub App and a .dockstore.yml file are two components of the same registration method.

This document gives a high level overview of how Dockstore uses GitHub apps. For extra details on configuring and using the Dockstore GitHub App, see our other docs on the here.

With the Dockstore GitHub App installed, authors do not need to manually refresh their workflows/services/tools on Dockstore to get the latest changes from GitHub. Dockstore will automatically update whenever the corresponding repository is updated on GitHub.

What are GitHub Apps?

GitHub apps are a GitHub feature used to improve the interaction between external applications and GitHub. Users can grant a GitHub app specific permissions on the repos and/or organizations of their choosing. When events occur on the GitHub repos, e.g., creating a new release, the GitHub app issues notifications.

Why have a Dockstore GitHub App?

Without a GitHub App, Dockstore does not know when you have modified a GitHub repository.

For example, take the case when you first add a workflow to Dockstore from GitHub. Dockstore indexes the repository – it reads the the relevant repository content, branches, and releases from GitHub. When you subsequently make changes to your GitHub repo, such as push new commits, create new branches and/or publish new releases, Dockstore is unaware of those changes. You are responsible for going to Dockstore, finding the tool/workflow that corresponds to the GitHub repo, and manually refreshing the tool/workflow by either clicking the Refresh button or making an API call to the Dockstore API.

Due to the manual nature of this process, it is easy for Dockstore to get out of sync with the linked GitHub repository if the Dockstore GitHub App is not being used.

How the Dockstore GitHub App works

With the Dockstore GitHub App installed, the synchronization is done automatically. When you add a new branch or release of a workflow on GitHub, Dockstore is notified, and Dockstore updates its copy of the workflow. For example, After publishing a new release of a workflow on GitHub, a new version of the workflow will be present in Dockstore shortly afterwards. For this to work, a .dockstore.yml file is required in the root directory (or inside the .github directory) of each GitHub repository you want to associate with a workflow/tool/service on Dockstore. It should not be inside any subfolder (except .github).

Simple templates for tools, workflows, and services are shown below, as well as links to more advanced explanations of every field. For every branch/release on GitHub that has one of these files, a corresponding entry will be made on Dockstore.

Error Handling

An invalid .dockstore.yml will cause errors. If we cannot read the file, then we do not know which workflow or service to associate the error with. For now, please ensure that your file is a valid YML file and compare it with our examples/documentation to confirm that you filled it in correctly. If the file is at least present, an error will generally appear in the GitHub App logs (see our FAQ document).

Another possible issue is that we received the message from GitHub, but the user who triggered the message event is not registered on Dockstore with the corresponding GitHub account. This is only an issue if the workflow or service does not already exist on Dockstore. When creating new workflows and services, we need to be able to associate them with a user. If the workflow or service already exists on Dockstore, then this error will not occur and the version will be properly added/updated/deleted on Dockstore.

See our FAQ document for assistance in troubleshooting, including how to interpret GitHub App logs.

As always, you can reach out to our team on our discussion forum to discuss any issues you are facing.

Example YML Files

Workflow YML File

For a workflow, the .dockstore.yml has the following general structure:

version: 1.2
workflows:
  - subclass: <CWL | WDL | NFL | GALAXY>
    primaryDescriptorPath: <String>
    testParameterFiles: <String Array>
    authors:
      - name: <String>
        email: <String>

As an example, here is a filled-out .dockstore.yml for a single WDL workflow which happens to have more than one test parameter file. Note that the name is not present since the name field is optional when only a single workflow is involved.

version: 1.2
workflows:
  - subclass: WDL
    primaryDescriptorPath: /assoc-aggregate/assoc-aggregate.wdl
    testParameterFiles:
      - /assoc-aggregate/assoc-aggregate-terra-allele.json
      - /assoc-aggregate/assoc-aggregate-terra-position.json
      - /assoc-aggregate/assoc-aggregate-terra-weights.json
    authors:
      - orcid: 0000-0003-4896-1858

Important

Though the name field is optional when a .dockstore.yml has one workflow in it, it must be used when a .dockstore.yml has multiple workflows in it. Each entry within a .dockstore.yml file corresponds to a unique entry on Dockstore.

For more examples, including multi-workflow examples and a complete breakdown of all possible fields, please see our .dockstore.yml templates and examples for workflows.

Tool YML File

The .dockstore.yml file for a tool is very similiar in structure to that of a workflow.

version: 1.2
tools:
  - subclass: CWL
    primaryDescriptorPath: <String>
    testParameterFiles: <String Array>
    authors:
      - name: <String>
        email: <String>

For more examples, including multi-workflow examples and a complete breakdown of all possible fields, please see our .dockstore.yml templates and examples for tools.

Service YML File

The .dockstore.yml for a service will depend heavily on the nature of the service it describes. A template .dockstore.yml file for registering services, with explanations in the comments, can be found in our Service 1.2 Template. For more info on services and registering them, check out our Getting Started with Services.

See Also