IRIS Knowledge Base :: Downloading data from the IRIS DMC using FTP or HTTP robustly

Knowledgebase Home

Search the Knowledgebase

Browse by Category

Downloading data from the IRIS DMC using FTP or HTTP robustly

It is common to download data using either FTP or HTTP from the IRIS DMC. Most of these downloads are completed quickly and without interruption. On occasion, a user's connection to the DMC is broken before the download is complete. This can be due to system problems at the user's end, network problems in-between or system problems at the DMC.

When downloading very large files or volumes that take a long time (many hours or days) the chances of the connection being interrupted are increased and it is useful to know how to restart or resume the transfer without downloading data that has already been transferred. The DMC's HTTP and FTP servers both support resuming downloads after a connection break. This capability must also be supported by the user's client software.

Note: When downloading files greater than 2 Gigabytes in size make sure your local filesystem is capable of storing large files, some older operating systems might not be capabie of storing files greater than 2 Gigabytes.

Restarting downloads using FTP

The FTP client you are using must support the reget command. After starting a transfer and the connection is broken, the download may be resumed simply by using the reget command in place of the get command:

ftp> reget mydata.seed
local: mydata.seed remote: mydata.seed
200 PORT command successful.
350 Restarting at 35732296. Send STORE or RETRIEVE to initiate transfer.
150 Opening BINARY mode data connection for testgsn.data (3545657856 bytes).

Restarting downloads using curl and wget

curl and wget are common command line download clients, both of which support both HTTP and FTP and are capable of resuming broken downloads.

curl supports resuming by using the -C option, the '-' argument must be used with the -C option to resume the download for an existing file. An example:
$ curl -C - -O http://www.iris.edu/pub/userdata/mydata.seed

wget supports resuming by using the -c or --continue option. An example:
$ wget -c http://www.iris.edu/pub/userdata/mydata.seed
or equivalently
$ wget --continue http://www.iris.edu/pub/userdata/mydata.seed

For more information on wget please visit: http://www.gnu.org/software/wget/
For more information on curl please visit: http://curl.haxx.se/

Restarting downloads using web browsers and other clients

Web browers and other HTTP clients behave in a variety of ways, it would be more common than not for a browser to start downloading a new copy of a file instead of resuming a previously started download.