Set Up File Provisioning Plugins

Note

Currently, the Dockstore CLI’s GS plugin does not support files that are located in requester-pays buckets.

File provisioning plugins can be used to extend the Dockstore CLI to allow for input and output files to be downloaded or uploaded to new and/or proprietary services. By default, the Dockstore CLI tries to read input/output files from the local filesystem and copy/hard-link to them. If that does not work, it will look at whether any of the installed plugins can handle the files. Finally, it will try using commons-vfs2 to handle the files.

Generally, the Dockstore CLI downloads a default set of file provisioning plugins, Amazon S3, Synapse, DOS, GS, and ICGC Storage Client, when it first installs. However, if you chose to download and configure the CLI yourself via step 3 of Part 1 during the quickstart tutorial and did not have a Dockstore account set up, then no file provisioning plugins will have downloaded. To check if you have the plugins, run the command dockstore plugin list. You should see something like

file provisioning plugins

If not, run dockstore plugin download.

The specification file for the plugins will be located at ~/.dockstore/plugins.json by default and is created the first time you run the command dockstore plugins download. The location can be overridden by using the key plugins-json-location in your Dockstore config file.

Configuration for plugins can be added to your ~/.dockstore/configand is indexed by id. For example:

token = <your generated by the dockstore site>
server-url = https://www.dockstore.org/api

# options below this are optional

use-cache = false                           #set this to true to cache input files for rapid development
cache-dir = /home/<user>/.dockstore/cache   #set this to determine where input files are cached (should be the same filesystem as your tool working directories)

[dockstore-file-synapse-plugin]

[dockstore-file-s3-plugin]
endpoint = #set this to point at a non AWS S3 endpoint

[dockstore-file-icgc-storage-client-plugin]
client = /media/large_volume/icgc-storage-client-1.0.23/bin/icgc-storage-client

To try an example and learn more about file provisioning, check out this tutorial.

If you would like to create your own plugins, follow along here.