Notebooks
Note
Dockstore Notebooks is currently in preview mode. Also note that the screenshots below were taken on our staging site.
Tutorial Goals
Get familiarized with the concept of notebooks on Dockstore
Learn the basics of creating a notebook using a .dockstore.yml file
Learn how to register a notebook
Publish your notebook to Dockstore
Notebooks are documents that can include code, text, images, and other media resources. They can be executed in notebook environments like Google Colab and JupyterLab. This document will outline what is needed to register, update, and publish a notebook onto Dockstore. Please keep in mind that notebook functionality is currently in preview mode.
Overview
In this tutorial, you will register a notebook using the Dockstore GitHub App, which automatically syncs your notebook from GitHub to Dockstore. More information about the Dockstore GitHub App can be found here.
Dockstore Notebooks takes advantage of Google Colab’s integration with GitHub. The integration allows you to develop your notebook in Google Colab and push your changes to GitHub from Google Colab. With the Dockstore GitHub App installed on your repository, any edits saved to your notebook from Google Colab will automatically be synced to Dockstore.
Setting up the Google Colab and GitHub integration
To get started, you will need a Google Colab account and a GitHub account.
In your Google Colab account, link your Google Colab account to your GitHub account. Click on the Settings gear icon in the upper right corner, then select GitHub in the dialog box. Click on Authorise with GitHub and authorize Google Colab to access your GitHub account.
Next, we will set up a notebook on Google Colab so that it’s also stored in one of your GitHub repositories. If you already have a notebook in Google Colab that’s saved to a GitHub repository, you may proceed to create the .dockstore.yml file for your repository.
We will cover two scenarios:
You don’t have a notebook in GitHub. You may or may not have a notebook in Google Colab.
You don’t have a notebook in GitHub
If you don’t have a notebook in GitHub nor Google Colab, navigate to Google Colab and create a new notebook by clicking on File > New notebook. Edit your new notebook if you like.
If you already have a notebook in Google Colab, proceed to the next step.
Click on File > Save a copy in GitHub to save your notebook to a GitHub repository. You may want to create a new repository for your notebook if you don’t already have a repository that you want to save the notebook to.
Configure where you want to save the notebook to on GitHub then click OK. Google Colab will push a commit to your repository with the notebook.
You have a notebook in GitHub, but it’s not in Google Colab
Click on File > Open notebook then select the GitHub tab in the dialog box. Select the notebook that you want to import from GitHub. This will open the notebook in Google Colab.
Create Your .dockstore.yml File
The first step is to create a file named .dockstore.yml which acts as a configuration file used to describe your notebook. This file should be created in the GitHub repository containing your notebook. We will cover an example .dockstore.yml written for a simple hello world Jupyter notebook written in Python. You can also view template .dockstore.yml files here that describe the possible fields and values you can use to configure your notebook.
version: 1.2
notebooks:
- name: simple
path: /notebook.ipynb
publish: true
Line 1 specifies the .dockstore.yml version. You should use the latest version, which is version 1.2.
Next is a required key named notebooks where your notebooks will be configured.
Within this, you can to specify the format and language of the notebook. These are optional keys and if left out, like in the example above, default to Jupyter and Python respectively. Currently, we only support Jupyter notebooks. For a list of all the languages supported, view the notebook template files.
Next, you need to specify the absolute path to the notebook file in the Git repository using the path key.
Finally, you must set publish: true to publish your notebook in order for you to be able to view your notebook in preview mode. In the future, we will implement a My Notebooks page where you can view your unpublished notebooks.
Registering Your Notebook
Note
You do not need an account to search for workflows on Dockstore or to launch them with our compute partners. However, to register content on Dockstore, you must have an account on Dockstore and link the necessary third-party accounts. Once this is done you can register workflows from the My Workflows page, tools from the My Tools page, services from the My Services page, or notebooks from the My Workflows page (currently in preview mode).
To register your Notebook on Dockstore, you need to install our Dockstore GitHub application on the repository containing your notebook and .dockstore.yml file. By doing so, Dockstore will automatically register the notebooks and update your notebooks whenever your repository is updated on GitHub.
Note
Our docs sometimes use “GitHub App registration” and “registration with a .dockstore.yml file” interchangeably, as they are equivalent terms. The Dockstore GitHub App and a .dockstore.yml file are two components of the same registration method.
Install the Dockstore GitHub App
Installing the GitHub App is simple. Navigate to /my-tools, /my-workflows, or /my-services by navigating to My Dashboard then selecting the desired option in the left sidebar. In these screenshots, we will go via /my-tools, but the process is essentially the same for any of the other options.
Note
Since Dockstore Notebooks is still in preview mode, it does not have a dedicated /my-notebooks page yet. To register a notebook, navigate to /my-workflows in the step above.
Click the Register a Tool button on the left hand sidebar.
A window will appear asking how you would like to register your tool, workflow, or service. Select Register using GitHub Apps.
Click + Manage Dockstore Installations on GitHub.
You’ll then be redirected to GitHub where you can install the app in an organization or your personal account. For example, the username for my personal GitHub account is aofarrel. If I want to install the GitHub App so it could access aofarrel/mycoolrepo, I would choose the first option here.
Install our GitHub App in an organization or your personal account
After selection of an organization or a personal account, you can select whether to give access to all current and future repositories or only select ones. If the organization or personal account you choose is intended to be just for Dockstore tools/workflows/services/notebooks, you may want to allow access to all repositories. Otherwise, it may be more intuitive to select only certain repositories. Click save and you will be taken back to the page you started on in Dockstore – either /my-tools, /my-workflows, or /my-services, depending on where you started.
Important
The GitHub user who first adds a workflow onto Dockstore must correspond to a user on Dockstore.
On Dockstore, under the GITHUB section, you should see the names of GitHub accounts that you have access to, such as organizations that you belong to and your personal account. If your repositories that you chose to keep track of contained a .dockstore.yml at the time of installing the GitHub App, then you will see the repositories under the GitHub personal/organization account name that it belongs to. Here’s an example involving /my-services:
Note
You will not see unpublished notebooks because there is not a dedicated /my-notebooks page yet. To view your notebook, which should’ve been configured to automatically publish on Dockstore via the .dockstore.yml, navigate to /notebooks
and locate your notebook in the list of published notebooks.
A note on permissions when installing the Dockstore GitHub App to a GitHub organization
Only organization admins and repository admins can install the Dockstore GitHub App.
Organization admins will have the easiest time installing the Dockstore GitHub App because they can install it to any repository in the organization on the installation page. Users who are not organization admins can only install the Dockstore GitHub App on repositories that they are an admin of.
For more information on troubleshooting GitHub App permissions, please see this FAQ entry.
See also
Automatic Syncing with GitHub Apps and .dockstore.yml - details on writing a .dockstore.yml file
Migrating Your Existing Workflows - a tutorial on converting already registered workflows
Troubleshooting and FAQ - tips on resolving Dockstore Github App issues.
Ensuring sychronization
Upon installing the GitHub App, Dockstore will find branches in your repository that contain a .dockstore.yml and attempt to register your workflows, tools, services, and notebooks. There may be some cases where Dockstore is unable to find all branches containing a .dockstore.yml, for example, if the GitHub repository has many branches.
If your workflow, tool, service, or notebook is not showing up on Dockstore after 5 minutes, push one additional commit to the branch in your repository that contains the .dockstore.yml that’s not being synchronized. Dockstore will synchronize the branch that was updated, which helps make sure that your workflows, tools, services, and notebooks show up in Dockstore.
If your workflow, tool, service, or notebook still doesn’t show up, check the GitHub App logs to see if Dockstore encountered an error while processing your .dockstore.yml.
Publishing Your Notebook
In this release, you are only able to publish your notebook using the publish key in the .dockstore.yml file, and your notebook must be published in order for you to view it on Dockstore.
Publishing will create a public page that is very similar to the ones we have for tools and workflows.
Now other users can view and star your notebook. You will also have the option to add published notebooks to a collection.
Saving Changes to Your Notebook on Google Colab to GitHub
At this point, your Google Colab and GitHub integration should be set up, you have installed the Dockstore GitHub App to your GitHub repository containing your notebook, and the published notebook appears on Dockstore.
You may wish to continue developing your notebook on Google Colab by modifying and re-running it. In order for the changes to your notebook to appear on Dockstore, you must click on File > Save a copy in GitHub to save your notebook to your GitHub repository.
Configure the GitHub repository and branch that you want to save the notebook to. This should be the repository and branch that contains your .dockstore.yml describing the notebook you have on Dockstore. Once configured, click OK. Google Colab will push a commit to your repository with the notebook.
Dockstore will detect the new commit and automatically sync your notebook from GitHub to Dockstore.
Launch With Google Colab
When viewing a public notebook on Dockstore, you can launch the notebook into Google Colab by clicking on the Google Colab link on the Launch With panel located on the right side of the page. This will open the notebook in Google Colab.
Using the Notebooks Feature Flag
There is a notebooks feature flag that can be used to explore Dockstore Notebooks features that are in development.
Warning
The features hidden behind this flag may not be fully functional, but using the flag offers you a preview of the features to come.
To use the notebooks feature flag, append notebooks to the Dockstore URL as a query parameter. You only need to do this once, unless you refresh/close your browser.
For example, if you’re on the https://dockstore.org page, append notebooks such that it looks like this: https://dockstore.org?notebooks.
If you’re on a page that already contains query parameters, indicated by the presence of a question mark, append notebooks to the URL using an ampersand. For example, if you’re on the https://dockstore.org/starred?tab=workflows page, append notebooks such that it looks like https://dockstore.org/starred?tab=workflows¬ebooks.
After applying the notebooks feature flag, Dockstore Notebooks features will appear throughout the site, marked with a warning banner.
Features that are activated with the feature flag:
Notebooks component is displayed on the My Dashboard page for a logged-in user
Notebooks tab is displayed on the My Starred page for a logged in user
Notebooks tab is displayed on the Search page
Notebooks link is displayed on the Sitemap