DTL Project 1: OPeNDAP with NetCDF
Overview
The mission of the Data Transport Laboratory (DTL) is to support the
employment of data transport technologies that are compatible with data
management and communications (DMAC) at the local and regional levels;
this is accomplished through the identification, evaluation, and
documentation of relevant data transport technology candidates.
In following that mission, the first DTL project is the consideration
of the OPeNDAP protocols for use with in-situ data in the local and
regional arenas. OPeNDAP, or the Open-source Project for a Network
Data Access Protocol, has been a significant data-sharing technology
in the oceanography community. Several different implementations of
the protocols exist, including the Live Access Server and the THREDDS
server. This project used the latest version of the base OPeNDAP
server-accessed from the main OPeNDAP Web site (www.opendap.org) -
available at the time of the project's start.
This project has concluded. For detailed findings, see the final report.
Project Selection
Suggestions for candidate technologies to be implemented in
the DTL are accepted from the Integrated Ocean Observing System
(IOOS) community. The primary sources for the identification of
candidates are the local observatories, regional associations,
Ocean.US Data Management and Communications (DMAC) team and its expert
teams, and deliberate outreach efforts of the DTL staff, such
as meetings and workshops.
Candidate technologies, nominated through these mechanisms, are filtered
through a set of criteria to assess their relevance and importance to the
stakeholders. For more details, read about the DTL technology selection process.
The selection of the OPeNDAP base server with network Common Data Form
(NetCDF) data was based on the most recent "Concrete Guidance for Data
Providers" released by the Ocean.US DMAC.
DMAC has not yet released any recommendations for "operational"
standards. The strongest recommendation, to date, is for OPeNDAP as a
"pre-operational" standard. Also, there is an existing install base
of the servers among the NOAA Coastal Services Center's local and
regional partners. The use of the NetCDF file format for the data
repository was based on the widespread use of the format by the OPeNDAP
servers in the IOOS community and the format's traditional combination
with the OPeNDAP servers. The following table lists the scoring and
rationale of that scoring for this project.
| Criteria |
Score |
Rationale |
| Significance to Local Observatories |
4 |
A number of local observatories already employ OPeNDAP servers. While the OPeNDAP
servers were developed first for gridded data, they may have significant utility
in serving in-situ data at the local level. |
| Significance to Regional Associations |
5 |
Regional data-collecting facilities will be handling more and heavier data
than the local observatories. They will need to be able to aggregate local
feeds and then publish the aggregated data sets up to the next level,
possibly to the federal backbone. With larger geographic coverages and
more dense data types, the regional facilities will need efficient
services. The DAP2 data model and format may show significant utility
in this context. |
| Relevance to DMAC Guidance |
5 |
The DMAC Data Transport Expert Team catagorizes its recommendations as
research, pilot, pre-opreational, and opertational. To date, there are no
recommended standards for "operational" technologies. The only recommendation
for a pre-operational standard is the DAP2 protocol
combined with the Climate and Forecast(CF) conventions. A common
implementation of the DAP2 protocol is the OPeNDAP server with NetCDF files. |
| Relative Maturity of Candidate |
4 |
The base Common Gateway Interface OPeNDAP servers are several years old
and benefit from years of use and exercise. The required libraries and applications
are available for a variety of common operating systems. Some question remains
about the availability of the NetCDF libraries for the Windows operating systems. |
| Appropriate Level of Effort for DTL |
5 |
The installation and testing of the OPeNDAP servers requires a reasonable and appropirate
level of effort from the DTL. |
| Coincidence of other Center Efforts |
0 |
While there are no active implementations of the OPeNDAP servers at the NOAA Coastal
Services Center, there is some use of the NetCDF file format on several Center
projects. Other projects are exploring their possible use. |
Total Score |
23 |
|
Related Links (top)
Server Details (top)
The OPeNDAP NetCDF server was installed on a Dell PowerEdge 1850 machine with two
3.0 MHz Xeon CPUs, four gigabytes of RAM, 300 gigabytes of 10,000 RPM disks,
and Gigabit network cards. The operating system is Red Hat Linux
Enterprise 4, kernel 2.6.9-42.
OPeNDAP NetCDF Server Installation, Configuration, and Testing (top)
Full documentation for the installation of the OPeNDAP server and NetCDF libraries is available
from those projects via the links above. The following details our process for installing them
on the DTL servers.
- Installation
- Package Downloads
OPeNDAP main download page: www.opendap.org/download/index.html
OPeNDAP CGI Server Base Software: www.opendap.org/download/CGI_server.html
NetCDF: www.unidata.ucar.edu/downloads/netcdf/index.jsp
NetCDF handler: www.opendap.org/download/nc_server.html
Windows builds: www.opendap.org/winPort.html
Builds for Microsoft Windows can also be found on the main download page.
However, there is no Windows build since OPeNDAP version 3.4.8.
- Installation under Linux
The installation of the OPeNDAP NetCDF server components was accomplished
following the bundled installation instructions. There were no OPeNDAP RPM
packages for RHL 4. The DTL staff attempted to install the RPM for RHL 3, which
failed. The source was downloaded and compiled as described below.
The following commands created a new user account for compiling the source.
You must have root priviledges to use the "useradd" command. Compiling as the
root user is considered unsafe.
useradd username
passwd userpassword
The following commands downloaded, compiled, and installed libdap 3.6.2,
which is required by dap-server 3.6.1:
[prompt]$ su username (created in the previous step)
[prompt]$ cd ~/opendap
[prompt]$ tar zxvf libdap-3.6.2.tar.gz
[prompt]$ cd libdap-3.6.2
[prompt]$ ./configure
[prompt]$ make
[prompt]$ su root
[prompt]$ make install
The libraries installed in the default location, /usr/local/lib.
The following commands downloaded, compiled, and installed dap-server 3.6.1:
[prompt]$ su username
[prompt]$ cd ~/opendap
[prompt]$
wget ftp://ftp.unidata.ucar.edu/pub/opendap/source/dap-server-3.6.1.tar.gz
[prompt]$ tar zxvf dap-server-3.6.1.tar.gz
[prompt]$ cd dap-server-3.6.1
[prompt]$ ./configure
[prompt]$ make
[prompt]$ su root
[prompt]$ make install
The Perl modules installed in the default location, /usr/local/share/dap-server.
The CGI program "nph-dods" installed in the default location, /usr/local/share/dap-server-cgi.
The following commands downloaded, compiled, and installed netcdf-3.6.1:
[prompt]$ su username
[prompt]$ cd ~/opendap
[prompt]$ wget http://www.unidata.ucar.edu/downloads/netcdf/ftp/netcdf-3.6.1.tar.gz
[prompt]$ tar zxvf netcdf-3.6.1.tar.gz
[prompt]$ cd netcdf-3.6.1/src
[prompt]$ ./configure --prefix=/usr/local/share/netcdf
[prompt]$ make
[prompt]$ su root
[prompt]$ mkdir /usr/local/share/netcdf
[prompt]$ make install
NetCDF installed in the specified location, /usr/local/share/netcdf.
The following commands downloaded, compiled, and installed netcdf_handler-3.6.0.
(NetCDF is required for configuring this NetCDF handler.)
[prompt]$ su username
[prompt]$ cd ~/opendap
[prompt]$ wget ftp://ftp.unidata.ucar.edu/pub/opendap/source/netcdf_handler-3.6.0.tar.gz
[prompt]$ tar zxvf netcdf_handler-3.6.0.tar.gz
[prompt]$ cd netcdf_handler-3.6.0
[prompt]$ ./configure --prefix=/usr/local/share/netcdf-handler --with-netcdf=/usr/local/share/netcdf
[prompt]$ make
[prompt]$ su root
[prompt]$ mkdir /usr/local/share/netcdf-handler
[prompt]$ make install
The NetCDF handler installed in the specified location, /usr/local/share/netcdf-handler.
-
Installation under Windows
The DTL staff was unable to acquire a recent version of the OPeNDAP NetCDF server
for Windows.
- Configuration
- Configuration under Linux
The DTL staff harvested several years of National Data Buoy Center (NDBC)
buoy data in NetCDF files and stored it in /var/www/html/data/.
To configure the OPeNDAP server to handle the NetCDF data, modify its configuration file.
Open the configuration file for OPeNDAP and change the following items.
(The default settings that haven't been changed are not listed here.)
maintainer you@yourmail.server
curl /usr/bin/curl
#handler .*\.(HDF|hdf|EOS|eos)(.Z|.gz|.bz2)?$ /usr/local/bin/dap_hdf4_handler
handler .*\.(NC|nc|cdf|CDF)(.Z|.gz|.bz2)?$ /usr/local/share/netcdf-handler/bin/dap_nc_handler
#handler .*\.(dat|bin)$ /usr/local/bin/dap_ff_handler
#handler .*\.(pvu)(.Z|.gz)?$ /usr/local/bin/dap_dsp_handler
#handler .*/test$ /usr/local/bin/jg_handler
Copy the OPeNDAP CGI program and configuration file into the cgi-bin directory:
[prompt]$ cp /usr/local/share/dap-server-cgi/* /var/www/cgi-bin/.
Initial attempts to test the server, produced an error, "...error while loading shared libraries: libdap.so.4...".
In order to facilitate the server finding the required libdap files, the DTL staff
created a link to the libdap library, which will be dynamically linked to dap_nc_handler and others.
[prompt]$ cd /usr/lib
[prompt]$ ln /usr/local/lib/libdap.so.4.0.0 libdap.so.4
Note that the recommended method of setting the LD_LIBRARY_PATH environment variable did not work.
- Base Testing
-
To demonstrate the successful installation of the OPeNDAP NetCDF server, ask the server
for its version information by adding the argument '/version' to the URL. This was
done using the following link in a Web browser:
http://csc-s-ial-d/cgi-bin/nph-dods/version
This successfully generated the following output:
OPeNDAP server core software: DAP2/3.6.1
-
To access data stored on your machine or network, in a
Web browser, add the path and name of the file of interest, relative to your Web server
root, to the end of the OPeNDAP server URL. For instance, if you have a data file at
"/var/www/html//data/ndbc/netcdf/ocean/15319o9999.nc"
you can use the follwing links, except that you will need to change the server name and
path to your OPeNDAP server installation.
The data descriptor structure (DDS):
http://csc-s-ial-d/cgi-bin/nph-dods/data/ndbc/netcdf/ocean/15319o9999.nc.dds
The data attribute structure (DAS):
http://csc-s-ial-d/cgi-bin/nph-dods/data/ndbc/netcdf/ocean/15319o9999.nc.das
The content of the file as plain ASCII text:
http://csc-s-ial-d/cgi-bin/nph-dods/data/ndbc/netcdf/ocean/15319o9999.nc.asc
The OPeNDAP Server Dataset Access Form for the file:
http://csc-s-ial-d/cgi-bin/nph-dods/data/ndbc/netcdf/ocean/15319o9999.nc.html
-
To test another data file, simply change the path and/or filename you append
to the end of the OPeNDAP CGI URL. For instance, given a file at "/var/www/html/data/ndbc/netcdf/stdmet/31201h2005.nc",
use the URL,
http://csc-s-ial-d/cgi-bin/nph-dods/data/ndbc/netcdf/stdmet/31201h2005.nc.html
-
Testing and Critique with External Partners (top)
The DTL solicited participation in the testing and critiquing of this technology from external
partners. The external partners will install, configure, and run instances of
the base OPeNDAP NetCDF server. Objective speed and through-put tests will be
performed, and more subjective issues will be discussed and reported. In general, the items
to be considered are as follows:
- Light and heavy load request response times measured with Apache JMeter. The definition of heavy
load will be case specific. In general, the test should load the server to the point that requests
are queued at the server. Repeat 100 iterations and record each. Calc average, max, and min.
The Coastal Services Center will design a basic JMeter test plan that can be refined for each
project and implemented.
- Relative ease or difficulty of installation of server software
- Relative ease or difficulty of installing and configuring any supporting software or libraries
- Relative ease or difficulty of modification in the event of evolving data models and standards
- Degree of intrusiveness of the transport layer into the data storage facility
- Cost of server implementation (infrastructure and human resources)
- Cost of client implementations (infrastructure and human resources)
- Support for server technology in the IOOS community (e.g. continued development, documentation, and user support)
- Support for server technology in the wider IT community(e.g. continued development, documentation, and user support)
- Availability of or support for development of clients in the IOOS community
- Availability of or support for development of clients in the IT community
- Relative ease of use with conventional or popular client package(s)
- Degree to which the technology supports IOOS data exchange needs for
- Through-put
- Ease of use
- Discovery
- Scalability
- Extendibility
- Security
- Metadata access to determine appropriate use contexts for a given data source.
- Mission critical reliability
- Data integrity (transactions, error checking missing/redundant/out of range data)
Once the test phase is complete, the DTL will compile the test results
and commentary in a report that will be available on the DTL Web site and on the Community Information Repository.
![[Skip Navigation]](/images/pixel.gif){kind=tracking}
