Post on 11-Jan-2016
MBARI’s SSDS
Data Management for Ocean Observatories
Brian Schlining ブライアン シュリニング
Overview
• About MBARI
• Overview of MOOS
• Functional Requirements of SSDS
• Metadata use by SSDS
• Dataflow through SSDS
Monterey Bay Aquarium Research Institute
Established in 1987
Santa Cruz
Monterey
Monterey Canyon
MBARI
MOOS History
Monterey Ocean Observing System• 1989: Began mooring-based observations.
• 1999: Adopted the term “MOOS” (Monterey Ocean Observing System)
• 1999: Began development of an integrated ocean observatory.
MOOS – The Present Concept
Benthic Node
Mooring
Autonomous Underwater Vehicle (AUV)
MBARI
MOOS – The Present Concept
Benthic Node
Mooring
AUV
Mooring
‘Plug and Work’ Instruments
Future of Ocean Observing
Variety of platforms
Data Management
Challenges:• Large number of data sources
• Large variety of data sources
• Dynamic systemData sources may appear and disappear
• No standard data formatData can be instrument ‘native’
SSDS
Shore-side Data System• Data management system for MOOS
SSDS – Functional Requirements
1. Capture and store data from MOOS data sources Includes files and ‘streams’
2. Capture information (i.e. metadata) about the stored data Location, instrument, platform, data format, etc.
3. Capture and store data products Derived products, quality controlled data, plots, etc.
SSDS – Functional Requirements
4. Provide access to the original ‘raw’ data
5. Convert data to common formats for user application tools (Excel , Matlab, ARCView, …)
6. Present simple plots of any well-described data.
7. Provide access to data through an application program interface (API) and a web interface.
Metadata – What we have learned
• Metadata must accompany the data– A system’s power relies on good knowledge of its data– Without metadata, the data quickly becomes unusable
• Metadata must accompany the instrument– Every connector between the two increases error rates– Once data and metadata detached, reattaching is painful
• Metadata must be flexible and structured– Flexible: you’ll need to define new kinds of data sources– Structured: consistency => automation => value
SSDS – Metadata (Object View)
‘Deployment’ information.
SSDS tracks:• Where the data was collected.• When it was collected.• What other data was used.• Relation to other deployments
SSDS – Metadata (Object View)
The data source.
SSDS tracks:• Software or hardware source• Unique identifier• Manufacturer information• References to documentation
SSDS – Metadata (Object View)
References to the data.
SSDS tracks:• The data storage location.• The deployment that produced this data.
SSDS – Metadata (Object View)
Format and contents of a DataContainer.
SSDS tracks:• The contents of a data-set.• The data format (To allow parsing by software).
SSDS – Metadata (Object View)
Why not?• Multiple overlapping standards. Which to
choose?– None have achieved broad community support– Some are not yet complete (Marine XML)– Excessive documentation requirements (FGDC)
• May use with SSDS later (Dublin Core)
SSDS does not currently use metadata standards such as ESML, ESRI’s Marine Data Model, FGDC, Dublin Core or Marine XML.
SSDS – Metadata Standards
Mooring Portal SSDS
Example SIAM to SSDS Data Flow
Mooring Portal SSDSDevice
Example SIAM to SSDS Data Flow
A device is connected to a platform, such as a Mooring.
Mooring Portal SSDSDevice
<?xml version="1.0" encoding="UTF-8"?><Metadata> <Deployment name="2003.30.01" role="platform“> <Device id="9873"/> <Deployment role="instrument" > <Device id="101"/> <output> <DataStream> <RecordDescription bufferStyle="binary" bufferLengthType="fixed" endian="little"> <RecordVariable name="time" columnIndex="1" format="double"
longName="Time(GMT)" units="milliseconds since Jan 01, 1970"/> <RecordVariable name="temperature" columnIndex="3" format="double"
longName="Seawater temperature" units="degrees C"/> </RecordDescription> </DataStream> </output> </Deployment> </Deployment></Metadata>
Example SIAM to SSDS Data Flow
The mooring retrieves the metadata from the device.
Portal SSDSDevice
Metadata Packet
<?xml …> <Metadata> … </Metadata>
Mooring
Example SIAM to SSDS Data Flow
The metadata is packaged and sent to a portal on shore before any data is sent to shore.
Portal SSDSDevice Mooring
Example SIAM to SSDS Data Flow
Metadata Packet
<?xml …> <Metadata> … </Metadata>
The portal forwards the metadata to SSDS.
PortalDevice Mooring
DB
SSDS
<?xml …> <Metadata> … </Metadata>
Example SIAM to SSDS Data Flow
SSDS stores the metadata in a database.
This allows applications to query for and use data.
PortalDevice Mooring
DB
SSDS
Example SIAM to SSDS Data Flow
PortalDevice Mooring
DB
SSDS
34,56.234,0.0023,...
Example SIAM to SSDS Data Flow
The device produces a data record.
PortalDevice Mooring
DB
SSDS
Data Packet
34,56.234,0.0023,...
Example SIAM to SSDS Data Flow
The data is packaged and sent to SSDS.
PortalDevice Mooring
DB
SSDS
VersionID,DeviceID,MetadataID,RecordType,PlatformID,SystemTime,SequenceNumber,DataBuffer(34,56.234,0.0023,…)
Serialized
Example SIAM to SSDS Data Flow
SSDS uses information in the packet to sort and store the data in a ‘raw’ format.
PortalDevice Mooring
DB
SSDS
netCDF
Example SIAM to SSDS Data Flow
Serialized
VersionID,DeviceID,MetadataID,RecordType,PlatformID,SystemTime,SequenceNumber,DataBuffer(34,56.234,0.0023,…)
The ‘raw’ data is parsed and stored as netCDF for easier access.
MBARI Software
SSDS
Software applications allow users to discover and obtain data in formats useful to the typical MBARI user. (netCDF, text, etc.)
PortalDevice Mooring
DB
netCDF
Example SIAM to SSDS Data Flow
Serialized
netcdf parosci {dimensions: time = UNLIMITED ; // (17761 currently)variables: double time(time) ; time:long_name = "Time (GMT)" ; time:units = "seconds since 1970-01-01 00:00:00" ; double depth(time) ; depth:long_name = "depth" ; depth:units = "UNKNOWN" ;
// global attributes: :title = "AUV data" ; :created = "2003-06-12T23:34:58Z" ; :history0 = ": Deployment information for parosci.log" ; :deploymentName = "2003.099.10" ; :instrumentId = "3699" ;}
PortalDevice Mooring
DB
SSDS
netCDF
Example SIAM to SSDS Data Flow
Serialized
MBARI Software
Software applications also provide simple visual representations of data
PortalDevice Mooring
DB
SSDS
netCDF
Example SIAM to SSDS Data Flow
Serialized
Web Pages
MBARI Software
Provide internet access
PortalDevice Mooring
DB
SSDS
netCDF
Existing netCDF Software
Example SIAM to SSDS Data Flow
Serialized
MBARI Software
Web Pages
Save development time by using existing software applications
Requirements Development
2002 2003 2004
MTM
AUV CTD
MTM II
Relational DB Schema
Infrastructure & Capabilities
Access & VisualizationExtreme Week
Tools Training
MTM Deployed
AUV Science Ops Start
SSDS Schedule
MetadataMetadata
MOOS ‘Lite’
Development Team
• John Graybeal
• Kevin Gomes
• Michael McCann
• Brian Schlining ( 武頼庵 洲美守 )
• Rich Schramm
• Daniel Wilkin
• John Ryan• Francisco Chavez• Ken Johnson• Drew Gashler• Mark Chaffee• Tom O’Reilly• Mark Chaffee
Science and Technical Advisors
Funding provided by the David and Lucile Packard Foundation