Set Up File Provisioning Plugins
Currently, the Dockstore CLI’s GS plugin does not support files that are located in requester-pays buckets.
File provisioning plugins can be used to extend the Dockstore CLI to allow for input and output files to be downloaded or uploaded to new and/or proprietary services. By default, the Dockstore CLI tries to read input/output files from the local filesystem and copy/hard-link to them. If that does not work, it will look at whether any of the installed plugins can handle the files. Finally, it will try using commons-vfs2 to handle the files.
Generally, the Dockstore CLI downloads a default set of file
provisioning plugins, Amazon S3, Synapse, DOS, GS, and ICGC Storage
Client, when it first installs. However, if you chose to download and
configure the CLI yourself via step 3 of Part 1 during the
quickstart tutorial and did not
have a Dockstore account set up, then no file provisioning plugins will
have downloaded. To check if you have the plugins, run the command
dockstore plugin list. You should see something like
If not, run
dockstore plugin download.
The specification file for the plugins will be located at
~/.dockstore/plugins.json by default and is created the first time
you run the command
dockstore plugins download. The location can be
overridden by using the key
plugins-json-location in your Dockstore
Configuration for plugins can be added to your
~/.dockstore/configand is indexed by id. For example:
token = <your generated by the dockstore site> server-url = https://www.dockstore.org/api # options below this are optional use-cache = false #set this to true to cache input files for rapid development cache-dir = /home/<user>/.dockstore/cache #set this to determine where input files are cached (should be the same filesystem as your tool working directories) [dockstore-file-synapse-plugin] [dockstore-file-s3-plugin] endpoint = #set this to point at a non AWS S3 endpoint [dockstore-file-icgc-storage-client-plugin] client = /media/large_volume/icgc-storage-client-1.0.23/bin/icgc-storage-client
To try an example and learn more about file provisioning, check out this tutorial.
If you would like to create your own plugins, follow along here.