NetDC manual [ back ] [ forward ]

Perhaps the most challenging aspect of NetDC installation can be found in the necessary step of writing the routines to allow NetDC to "talk" to a data center's information base, which can consist of a database, flat file tables, disk archives, and/or mass storage systems. Because of the wide variation between data centers in terms of how data is stored and retrieved, the writing of this code must be left up to the individual who is installing NetDC.

Effort has been made to reduce the amount of code that the installer must modify or write. Essentially, the bulk of the code in NetDC can remain as-is, and generally should be left alone in the interest of technical support and ease of future upgrades of NetDC. Essentially, the local processing system is broken into data type branches, as described earlier in the manual. Currently, there are three branches: one for inventory processing, one for response data, and one for waveform data. Each of these branches has a stub that must be implemented by the installer. Each stub takes the form of an independent executable that NetDC calls. It is typical to name the executable with the prefix "process_", followed by a shorthand variation of the data type. For DATA requests, it's "process_data", for INV, it's "process_inv".

Example "C" code of these implementations is provided with the distribution, and represents an actual working executable, most likely a version that currently runs at IRIS Data Management Center. It is recommended that the administrator read the example code and consider how to modify the code to work at their site, as opposed to writing the interface from scratch. The "process" code can be written in one of any number of compiled or scripting languages, but the key is that the interface code must be able to run from the UNIX shell with no command line arguments. The "process" code gets its parameters and request lines through "standard input." The output data always ends up in a file named by the FILENAME parameter that is passed to it.

To see an example of the flow of a NetDC interface, let's look at the code found in "process_inv.c." First, the code is called from the NetDC function "local_inventory()" with a "popen()" call that contains the command "process_inv". The command is opened in "write" mode, so that "process_inventory()" can read parameter information as well as the INV request lines themselves. When process_inv starts, it goes through a process of initialization, setting up a few preliminary values and some storage memory. Then comes the line reading loop, where each line from standard input is read. The line is "dressed up" for parsing, and then the component fields are copied and examined separately (see Fig. 11.1).

Fig 11.1 - Process inventory flow of operation

For each line that is read, the code assesses whether the input line has ".INV" as the first field. If it is not an inventory request line, then it runs through a series of checks to see if the first field indicates a particular parameter that will provide information regarding the request. "process_inv", as well as the other interface code, will need to be able to accept and store the following parameters:

.NAME			(the name of the requesting user)
.EMAIL			(the email address of the requesting user)
.HUB_ID			(the assigned ID tag of the netdc_request)
.LABEL			(the label of the netdc_request)
.DISPOSITION_USER	(how to deliver output to the user)
.DISPOSITION_HUB	(how  to deliver output to the hub data center)
.FILENAME			(the filename where the output will be placed)

When an inventory request line is read, each field is separately stored. Because of the variable-length nature of inventory requests, extra checks are put in place to determine how many request fields are present, which will set certain triggers as to what kind of information is being requested.

As the request lines are being read, an output file is opened using the filename specified with the FILENAME parameter. With the file opened and ready to have data written to it, "process_inv" goes on to form an SQL query for the local database. With the SQL string constructed from the request fields, a read pipe is opened that will call upon the database's query program, with the SQL query being included on the command line. A loop is then created to read the lines that return through the read pipe. This will be the inventory output, which is filtered and formatted into NetDC inventory format (see Chapter 6.0) before being written to the output file. After all the output is read, the query pipe is closed.

As a special case, an inventory request asking just for information on Networked Data Centers (".INV *") will return information directly from the routing table, as opposed to the database.

After all of the request lines have been processed in this fashion, the output file is closed and the interface program exits. Context then returns to the "local_inventory()" function, where it proceeds to get the output file named by the FILENAME parameter and this data is forwarded onto the NetDC request directory.

The example code provided in the NetDC distribution may stray slightly in its behavior from the above description, but the basic flow of the code remains the same.

In some cases, it may be necessary to set up a different query calling structure than the example described above. One possibility is to have the SQL command passed through standard input and have the output written to a file that can be retrieved afterward. Another case could be where the query routine is called with both an input pipe and output pipe, with SQL being fed into the input pipe and the output being read from the other pipe. The application of these techniques is left up to the individual developer.

Fig 11.2 - Three examples of connecting to the information base

The end result of the interface code (such as "process_inv") should be that it reads request input and places the output in a file with a specified name. This can be tested in stand-alone mode by setting up a test request input and passing it to your interface code. The interface code should be able to access the information base and produce a formatted output file that can be retrieved later by the local function (such as "local_inv()").

A slightly different approach must be taken with data types that take a protracted length of time to produce, such as waveform data. In this case, the "process" interface should not wait until the data is returned. Instead, "process_data" sets up a batch request and forwards that on to the waveform processing system, followed by an exit call. The interface code does not wait for input to come back another instance of "process_data" will not be invoked later to gather the output.

So how is the output retrieved? The answer is with the use of a datagram message sent from the waveform processing system called by "process_data". When NetDC receives the message DATA::LOCALDONE, it immediately triggers the function "local_data_receive()" where the output data file is retrieved and the contents are copied to the NetDC request directory of the intended recipient.

The datagram is formatted like the example below:

%%ACTION DATA::LOCALDONE
.HUB_ID NCEDC:Feb_25,22:07:30:17501	<this is the ID for the request>
.NAME Bill_Mantle				<the name of the user>
.EMAIL bmantle@upthrust.seismolab.edu	 <the user's email address>
.LABEL Sample_01				<the label for the request>
.DISPOSITION_USER				<how the user wants to receive data>
.DISPOSITION_HUB				<how to merge data to the hub site>
.FILENAME /usr/local/processing/netdc/local_data.Feb_25.001	<name of data file>
.END						<always include an END terminator>

In instances where the waveform information system cannot produce any data matching the request specifications, the datagram DATA::NODATA can be sent instead. This datagram contains the same information as LOCALDONE, but results in notification to the user that no data could be produced. As for specifying a FILENAME in the datagram, it is safe to use a dummy reference, since no attempt will be made to extract the information from the listed filename.

A final emphasis should be made to the reader to examine the code provided in the distribution as pertains to the local functions and the interface functions. Become familiar with how they are intended to perform before starting down the road of writing the interface code. Writing your interface from an informed standpoint will help avoid running into development difficulties.

introduction •• overall concept •• request format •• request reception and delegation
datagrams •• local request processing •• inventory requests •• response requests
waveform requests •• product shipment •• installation and setup •• writing interface code
troubleshooting •• future implementations •• conclusion •• appendix A - summary of NetDC datagrams

About IRIS | Members | Programs | USArray | Seismic Monitor | Earthquakes | SeismoArchives
Mailing Lists | Stations | Data | Software | Publications | News | Contact | Site Map | Search