Dockstore Notebooks is currently in preview mode. Also note that the screenshots below were taken on our staging site.
Get familiarized with the concept of notebooks on Dockstore
Learn the basics of creating a notebook using a .dockstore.yml file
Learn how to register a notebook
Publish your notebook to Dockstore
Notebooks are documents that can include code, text, images, and other media resources. They can be executed in notebook environments like Google Colab and JupyterLab. This document will outline what is needed to register, update, and publish a notebook onto Dockstore. Please keep in mind that notebook functionality is currently in preview mode.
In this tutorial, you will register a notebook using the Dockstore GitHub App, which automatically syncs your notebook from GitHub to Dockstore. More information about the Dockstore GitHub App can be found here.
Dockstore Notebooks takes advantage of Google Colab’s integration with GitHub. The integration allows you to develop your notebook in Google Colab and push your changes to GitHub from Google Colab. With the Dockstore GitHub App installed on your repository, any edits saved to your notebook from Google Colab will automatically be synced to Dockstore.
To get started, you will need a Google Colab account and a GitHub account.
In your Google Colab account, link your Google Colab account to your GitHub account. Click on the Settings gear icon in the upper right corner, then select GitHub in the dialog box. Click on Authorise with GitHub and authorize Google Colab to access your GitHub account.
Next, we will set up a notebook on Google Colab so that it’s also stored in one of your GitHub repositories. If you already have a notebook in Google Colab that’s saved to a GitHub repository, you may proceed to create the .dockstore.yml file for your repository.
We will cover two scenarios:
You don’t have a notebook in GitHub. You may or may not have a notebook in Google Colab.
If you don’t have a notebook in GitHub nor Google Colab, navigate to Google Colab and create a new notebook by clicking on File > New notebook. Edit your new notebook if you like.
If you already have a notebook in Google Colab, proceed to the next step.
Click on File > Save a copy in GitHub to save your notebook to a GitHub repository. You may want to create a new repository for your notebook if you don’t already have a repository that you want to save the notebook to.
Configure where you want to save the notebook to on GitHub then click OK. Google Colab will push a commit to your repository with the notebook.
Click on File > Open notebook then select the GitHub tab in the dialog box. Select the notebook that you want to import from GitHub. This will open the notebook in Google Colab.
The first step is to create a file named
.dockstore.yml which acts as a configuration file used to describe your notebook. This file should be created in the GitHub repository containing your notebook. We will cover an example .dockstore.yml written for a simple hello world Jupyter notebook written in Python. You can also view template .dockstore.yml files here that describe the possible fields and values you can use to configure your notebook.
version: 1.2 notebooks: - name: simple path: /notebook.ipynb publish: true
Line 1 specifies the
.dockstore.yml version. You should use the latest version, which is version 1.2.
Next is a required key named
notebooks where your notebooks will be configured.
Within this, you can to specify the
language of the notebook. These are optional keys and if left out, like in the example above, default to
Python respectively. Currently, we only support
Jupyter notebooks. For a list of all the languages supported, view the notebook template files.
Next, you need to specify the absolute path to the notebook file in the Git repository using the
Finally, you must set
publish: true to publish your notebook in order for you to be able to view your notebook in preview mode. In the future, we will implement a My Notebooks page where you can view your unpublished notebooks.
You do not need an account to search for workflows on Dockstore or to launch them with our compute partners. However, to register content on Dockstore, you must have an account on Dockstore and link the necessary third-party accounts. Once this is done you can register workflows from the My Workflows page, tools from the My Tools page, services from the My Services page, or notebooks from the My Workflows page (currently in preview mode).
To register your Notebook on Dockstore, you need to install our Dockstore GitHub application on the repository containing your notebook and .dockstore.yml file. By doing so, Dockstore will automatically register the notebooks and update your notebooks whenever your repository is updated on GitHub.
Our docs sometimes use “GitHub App registration” and “registration with a .dockstore.yml file” interchangeably, as they are equivalent terms. The Dockstore GitHub App and a .dockstore.yml file are two components of the same registration method.
Installing the GitHub App is simple. Navigate to
/my-services by navigating to My Dashboard then selecting the desired option in the left sidebar. In these screenshots, we will go via
/my-tools, but the process is essentially the same for any of the other options.
Since Dockstore Notebooks is still in preview mode, it does not have a dedicated
/my-notebooks page yet. To register a notebook, navigate to
/my-workflows in the step above.
Register a Tool button on the left hand sidebar.
A window will appear asking how you would like to register your tool, workflow, or service. Select
Register using GitHub Apps.
+ Manage Dockstore Installations on GitHub.
You’ll then be redirected to GitHub where you can install the app in an organization or your personal account. For example, the username for my personal GitHub account is aofarrel. If I want to install the GitHub App so it could access aofarrel/mycoolrepo, I would choose the first option here.
After selection of an organization or a personal account, you can select whether to give access to all current and future repositories or only select ones. If the organization or personal account you choose is intended to be just for Dockstore tools/workflows/services/notebooks, you may want to allow access to all repositories. Otherwise, it may be more intuitive to select only certain repositories. Click save and you will be taken back to the page you started on in Dockstore – either
/my-services, depending on where you started.
The GitHub user who first adds a workflow onto Dockstore must correspond to a user on Dockstore.
On Dockstore, under the GITHUB section, you should see the names of GitHub accounts that you have access to, such as organizations that you belong to and your personal account. If your repositories that you chose to keep track of contained a .dockstore.yml at the time of installing the GitHub App, then you will see the repositories under the GitHub personal/organization account name that it belongs to. Here’s an example involving
You will not see unpublished notebooks because there is not a dedicated
/my-notebooks page yet. To view your notebook, which should’ve been configured to automatically publish on Dockstore via the .dockstore.yml, navigate to
and locate your notebook in the list of published notebooks.
Only organization admins and repository admins can install the Dockstore GitHub App.
Organization admins will have the easiest time installing the Dockstore GitHub App because they can install it to any repository in the organization on the installation page. Users who are not organization admins can only install the Dockstore GitHub App on repositories that they are an admin of.
For more information on troubleshooting GitHub App permissions, please see this FAQ entry.
Automatic Syncing with GitHub Apps and .dockstore.yml - details on writing a .dockstore.yml file
Migrating Your Existing Workflows - a tutorial on converting already registered workflows
Troubleshooting and FAQ - tips on resolving Dockstore Github App issues.
Upon installing the GitHub App, Dockstore will find branches in your repository that contain a .dockstore.yml and attempt to register your workflows, tools, services, and notebooks. There may be some cases where Dockstore is unable to find all branches containing a .dockstore.yml, for example, if the GitHub repository has many branches.
If your workflow, tool, service, or notebook is not showing up on Dockstore after 5 minutes, push one additional commit to the branch in your repository that contains the .dockstore.yml that’s not being synchronized. Dockstore will synchronize the branch that was updated, which helps make sure that your workflows, tools, services, and notebooks show up in Dockstore.
If your workflow, tool, service, or notebook still doesn’t show up, check the GitHub App logs to see if Dockstore encountered an error while processing your .dockstore.yml.
In this release, you are only able to publish your notebook using the
publish key in the .dockstore.yml file, and your notebook must be published in order for you to view it on Dockstore.
Publishing will create a public page that is very similar to the ones we have for tools and workflows.
At this point, your Google Colab and GitHub integration should be set up, you have installed the Dockstore GitHub App to your GitHub repository containing your notebook, and the published notebook appears on Dockstore.
You may wish to continue developing your notebook on Google Colab by modifying and re-running it. In order for the changes to your notebook to appear on Dockstore, you must click on File > Save a copy in GitHub to save your notebook to your GitHub repository.
Configure the GitHub repository and branch that you want to save the notebook to. This should be the repository and branch that contains your .dockstore.yml describing the notebook you have on Dockstore. Once configured, click OK. Google Colab will push a commit to your repository with the notebook.
Dockstore will detect the new commit and automatically sync your notebook from GitHub to Dockstore.
When viewing a public notebook on Dockstore, you can launch the notebook into Google Colab by clicking on the Google Colab link on the Launch With panel located on the right side of the page. This will open the notebook in Google Colab.
There is a
notebooks feature flag that can be used to explore Dockstore Notebooks features that are in development.
The features hidden behind this flag may not be fully functional, but using the flag offers you a preview of the features to come.
To use the
notebooks feature flag, append
notebooks to the Dockstore URL as a query parameter. You only need to do this once, unless you refresh/close your browser.
If you’re on a page that already contains query parameters, indicated by the presence of a question mark, append
notebooks to the URL using an ampersand. For example, if you’re on the https://dockstore.org/starred?tab=workflows page, append
notebooks such that it looks like https://dockstore.org/starred?tab=workflows¬ebooks.
After applying the notebooks feature flag, Dockstore Notebooks features will appear throughout the site, marked with a warning banner.
Features that are activated with the feature flag:
Notebooks component is displayed on the My Dashboard page for a logged-in user
Notebooks tab is displayed on the My Starred page for a logged in user
Notebooks tab is displayed on the Search page
Notebooks link is displayed on the Sitemap