NetDC manual [ back ] [ forward ]

Requests will arrive at a data center through the email account netdc. When the email arrives to that address, a program reads the message and determines if it is indeed a NetDC request. If this is the case, then the message is processed and a copy of the request is kept in a text file.

The request, if coming directly from the user, is assigned a "hub ID", which is a unique timestamp label just for that request. The hub ID is the identifier that will be used to track the progress of the request as it works its way through the NetDC system. This tag will take the form of:

<DC_NAME>:<MMM_DD,HH:MM:SS>:<PID>

where the DC_NAME represents the name of the hub data center and <MMM_DD,HH:MM:SS> represents the date and time in GMT (Greenwich Mean Time) of the request arrival. <PID> is the UNIX process ID of the code that initially runs when the request is received, and is for all intents and purposes a random number. The hub ID tag will never be modified once set. An example hub ID would read:

.HUB_ID NCEDC:Jul_04,02:00:06:4435

which would indicate that Northern California Earthquake Center at Berkeley received the original NetDC request at 2:00 a.m. on July 4^th.

The request is examined and critical identifying fields are noted so that a unique directory can be made for processing the request. In this directory, certain processing files will be generated:

• A copy of the NetDC request. The filename will be of the form netdc_req.<HUB_ID>.
• Outgoing NetDC request files, which are to be sent to delegate data centers for processing.
• A file named check.list, used for logging where requests have been delegated and "checking off" shipments as they arrive from the delegate data centers.
• Text files containing information used in request processing.
• Any data files that are ready to be shipped to the user.
• Other files that may be used to flag the status of the request.

If the request lacks needed information, the data center technicians or the requesting user will be notified so that the problem can be remedied. Otherwise, the data request lines are split into two groups: Those to be delegated to other data centers and those to be processed locally.

As each data request line is read, NetDC will look at the network code requested to determine if the line is to be delegated to another data center. A routing table maintained on the local system contains network code references for all networked data centers and acts as a "directory listing" to determine where a given request line is to be sent. Using this information, the request line will either be processed locally or will be written to an outgoing file destined for a primary or secondary provider of that network’s data.

Fig. 3.1 - Each data request line is routed to a request file destined for the appropriate data center. Destination is determined by matching the DATA_CENTER and NETWORK fields to entries in the routing table.

The typical routing table looks like the example below. The fields are separated by the vertical bar (‘|’) character. All network codes are listed with its data center name and contact information. In the example below, ‘<cr>‘ signifies where a carriage return would be in the text file:

BK|NCEDC|PRIMARY|netdc@quake.geo.berkeley.edu|Northern California Earthquake Data Center|Seismographic Station, Earth Sci. Bldg., Univ. of California, Berkeley, CA 94720|Douglas Neuhauser|(510) 642-0931 |doug@seismo.berkeley.edu|4096|ver3<cr>
G|GEOSCOPE|PRIMARY|netdc@geosp6.ipgp.jussieu.fr|Institut de Physique du globe de Paris|Department de Sismologie, Programme GEOSCOPE, 4, Place Jussieu, 75252 Paris cedex 05, France|Genvieve Roult|33 1 44274888|groult@ipgp.jussieu.fr|8192|9802<cr>
GE|GEOFON|PRIMARY|netdc@gfz-potsdam.de|GFZ, Telegrafenberg A6, D-14473, Potsdam, Germany|Winfried Hanka|49-331-288-1213|hanka@gfz-potsdam.de|10240|2A<cr>
GE|IRIS_DMC|SECONDARY|netdc@dmc.iris.washington.edu|IRIS Data Management Center|1408 NE 45th St., Ste. 201, Seattle, WA 98105|Tim Ahern|(206) 547-0393 |tim@iris.washington.edu|4096|SEP02<cr>
IU|IRIS_DMC|PRIMARY|netdc@dmc.iris.washington.edu|IRIS Data Management Center|1408 NE 45th St., Ste. 201, Seattle, WA 98105|Tim Ahern|(206) 547-0393|tim@iris.washington.edu|4096|1.3<cr>
II|IRIS_DMC|PRIMARY|netdc@dmc.iris.washington.edu|IRIS Data Management Center|1408 NE 45th St., Ste. 201, Seattle, WA 98105|Tim Ahern|(206) 547-0393|tim@iris.washington.edu|4096|1.3<cr>

The format of these entries can be described as follows:

<NETWORK>|<DC_NAME>|<PRIORITY>|<EMAIL>|<DATA_CENTER_NAME>|<ADDRESS>
|<CONTACT>|<PHONE>|<CONTACT_EMAIL>|<PEAK_MERGE_KB>|<VERSION>

When deciding where a given data request line should be routed, NetDC looks at the network code in the request line and compares it to the network codes in the route table (Current routing table can be downloaded here). If there is a match, then this line is selected, and added to a request destined for the site that will process that request line. If a specific data center was listed in the DC_NAME field then that center will be used as the destination for the request line.

However, it gets more specific than that. You'll notice that there are sites listed as PRIMARY and there is one example of a line that is listed as SECONDARY. These entries indicate the priority that data center holds for providing that network's data. In the example above, GEOFON is the primary provider of GE data and will take precedence in processing requests for GE data over secondary sites, such as IRIS_DMC as shown above.

There is always just one PRIMARY site. Should several sites be listed as providing data for a given network, the PRIMARY site is always selected, should it be listed. If the PRIMARY site for that network is not listed, then NetDC scans for the first SECONDARY site in the routing table and uses that site instead. There can potentially be several SECONDARY sites but there should never be more than one PRIMARY site.

The separate request files that are delegated to other sites are created in the user's request directory and take the form of:

netdc_out.<DC_NAME>

such as:

netdc_out.GEOSCOPE

which would be a request file destined for the GEOSCOPE site.

The exception is with request lines that are delegated locally. Their files take on generic names such as "inventory.request" and "data.request" and contain just the request lines without a header. These files are read directly by the local NetDC process after routing to other sites has completed.

As these requests are sent out to the delegate sites, a checklist file is made, indicating which data centers have been delegated tasks along with the current status of request processing. The file will include email addresses from the routing table and look something like this

GEOFON	netdc@gfz-potsdam.de	DATA PENDING
GEOFON	netdc@gfz-potsdam.de	RESP COMPLETE
GEOSCOPE	netdc@ipgp.jussieu.fr	DATA COMPLETE
GEOSCOPE	netdc@ipgp.jussieu.fr	INV COMPLETE
IRIS_DMC	netdc@iris.washington.edu	DATA PENDING

Those items marked as PENDING represent responses still being waited on. Items marked as COMPLETE have returned from processing and are ready to be merged with other data before shipment (assuming the requester asked to merge the data). If the user requested that the data not be merged, or certain conditions arise where merging is not possible, this field would be NOMERGE. As the course of request processing and data merging go forth, these values will be updated automatically by NetDC.

introduction •• overall concept •• request format •• request reception and delegation
datagrams •• local request processing •• inventory requests •• response requests
waveform requests •• product shipment •• installation and setup •• writing interface code
troubleshooting •• future implementations •• conclusion •• appendix A - summary of NetDC datagrams

About IRIS | Members | Programs | USArray | Seismic Monitor | Earthquakes | SeismoArchives
Mailing Lists | Stations | Data | Software | Publications | News | Contact | Site Map | Search