February 08-10, 2010 Falda del Carmen, Córdoba, ARGENTINA
description
Transcript of February 08-10, 2010 Falda del Carmen, Córdoba, ARGENTINA
Mission Operations Review
February 08-10, 2010Falda del Carmen, Córdoba, ARGENTINA
DAY 2SECTION 08
Aquarius Data Processing System (ADPS)
Gene Carl FeldmanFred Patt
2 of 71February 8-10, 2010
Section 08 – Aquarius Ground System Overview and Data ProcessingMission Operations Review
MOR Agenda – Page 2 of 3
3 of 71February 8-10, 2010
Section 08 – Aquarius Ground System Overview and Data ProcessingMission Operations Review
• Overview and Status• Key Requirements• Previous Reviews• System Interfaces• Key Processes
– Data Acquisition and Ingest– Science Data Processing– Algorithm Integration, Testing and Evaluation– Data Archive and Distribution– Data Quality Control– Validation Support– Periodic Reprocessing
• Facilities/System Configuration/Backup Strategy/ IT Security• Integration and Testing• Staffing, Roles and Responsibilities• Documentation• Risks and Issues
Agenda
4 of 71February 8-10, 2010
Section 08 – Aquarius Ground System Overview and Data ProcessingMission Operations Review
Aquarius Data Processing System Overview
• The Aquarius Ground System at NASA/GSFC consists of the Aquarius Data Processing System (ADPS) and the Aquarius Command and Control System (ACCS - next presentation)
• The ADPS will support:– Acquisition and ingest of Aquarius/SAC-D data from CONAE.– Acquisition of dynamic ancillary required to support the science processing.– Processing of Aquarius data to standard product levels using algorithms provided
by the Science Team.– Quality control of data products using methods specified by the Science Team.– Match-up of Aquarius data with in situ salinity measurements.– Archive and distribution of all data products during the active life of the mission to
the science team and general science community.– Delivery of source data (Level-1a) and “validated” Level-2 and Level-3 derived
data products to the PO.DAAC.
• The ADPS will be supported within the existing infrastructure of the Ocean Biology Processing Group (OBPG).
5 of 71February 8-10, 2010
Section 08 – Aquarius Ground System Overview and Data ProcessingMission Operations Review
Ocean Biology Processing Group
http://oceancolor.gsfc.nasa.gov
• Located: Goddard Space Flight Center
• Current Archive size: > 900 TBytes
• Distribution (> 2/04): 53 million files
● All missions are supported within the framework and facilities of the current NASA Ocean Data Processing System (ODPS) which has been successfully supporting operational, satellite-based remote-sensing missions since 1996, and its capabilities continue to evolve and expand to meet the demands and challenges of future missions
MISSIONS Currently SUPPORTED
• SeaWiFS : 1997 - present• MODIS (Terra and Aqua) : 2000 -
present• MERIS (US Coastal Waters) Feb 2009 -
present • CZCS / Nimbus-7 : 1978 - 1986• OCTS / ADEOS-I : 1996 - 1997• Glory data system prototype : 2010
launch• Aquarius / SAC-D : 2010 launch• VIIRS / NPP : 2011 launch• Community Processing & Analysis
Software SeaDAS (1991- present)
6 of 71February 8-10, 2010
Section 08 – Aquarius Ground System Overview and Data ProcessingMission Operations Review
Status - Science Code only
• If Aquarius were launched today and downlink files in the format expected showed up on the CONAE data server, we could Operationally :– Detect, transfer and ingest the files
– Reformat the files eliminating duplicate records
– Create temporary Level-1A data files and perform telemetry monitoring
– When subsequent downlinks are received, complete Level-1A orbit files produced
– Ingest and preprocess current set of required ancillary data
– Process to Level-2 salinity products and make products (Level-1A & 2) available via the existing web browse, download and order tool
– Process Level-2 files to Level-3 maps and make products available via web browse/download tool.
However
7 of 71February 8-10, 2010
Section 08 – Aquarius Ground System Overview and Data ProcessingMission Operations Review
• From now until launch we need to:
– Obtain and incorporate sample SAC-D telemetry files from CONAE into the Level-1A file structure (TBD)
– Receive updated Level-1b and Level-2 processing code from Frank Wentz (November 2009)
– Receive Scatterometer processing code from JPL (November 2009)
– Review and verify delivered code and merge simulated data sets (December 2009)
– Integrate Radiometer and Scatterometer code into data processing system (Jan/Feb 2010)
– Develop ingest procedures for additional ancillary data required by new code (Jan/Feb 2010)
– Verify output products with algorithm providers (February 2010)
– Process one year simulated data set and make available to science community (March 2010)
– Begin real-time mission simulation (March/April 2010 --> Launch)
– Verify Downlink file and SAC-D Housekeeping telemetry format/content (TBD)
– Participate in mission rehearsals as scheduled
Status - Science Code TBD
8 of 71February 8-10, 2010
Section 08 – Aquarius Ground System Overview and Data ProcessingMission Operations Review
• Overview and Status• Key Requirements• Previous Reviews• System Interfaces• Key Processes
– Data Acquisition and Ingest– Science Data Processing– Algorithm Integration, Testing and Evaluation– Data Archive and Distribution– Data Quality Control– Validation Support– Periodic Reprocessing
• Facilities/System Configuration/Backup Strategy/ IT Security• Integration and Testing• Staffing, Roles and Responsibilities• Documentation• Risks and Issues
Agenda
9 of 71February 8-10, 2010
Section 08 – Aquarius Ground System Overview and Data ProcessingMission Operations Review
ADPS Key Requirements
• The ADPS shall receive, log, process and archive:– Level 0 raw science data– Aquarius HKT– Service platform HKT (including attitude data)– Ephemeris data
• The ADPS shall acquire and utilize ancillary data products to process Aquarius science data.
• The ADPS shall process science data to Level 1, 2 and 3 products.• The ADPS shall deliver Aquarius data products to the Science Team and
other users.• The ADPS shall deliver data products to the JPL PO.DAAC.• The ADPS shall support Aquarius data product validation by the Aquarius
Validation Data System (AVDS).• The Aquarius Ground System shall utilize and expand upon the GSFC
OBPG facilities as required.• The ADPS shall support reprocessing of the full Aquarius mission in six
months or less.• The ADPS shall support the archive of the full Aquarius mission data set.
10 of 71February 8-10, 2010
Section 08 – Aquarius Ground System Overview and Data ProcessingMission Operations Review
Previous Reviews
• Aquarius Preliminary Design Review – June 2005• Aquarius Science/Ground System Pre-CDR – July 2006• Aquarius 1st Science Algorithm Workshop – March 2007• Aquarius 2nd Science Algorithm Workshop – March 2008• ACCS/ADPS Peer Review – March 2008• Aquarius/SAC-D Ground Segment CDR – July 2008• Aquarius Data Processing System Readiness Review – July 2009• Joint CONAE/NASA Mission Operations Working Group – Sept,
Dec 2009• ACCS/ADPS Peer Review – January 2010
All RFAs from previous reviews have been closed.
11 of 71February 8-10, 2010
Section 08 – Aquarius Ground System Overview and Data ProcessingMission Operations Review
• Overview and Status• Key Requirements• Previous Reviews• System Interfaces• Key Processes
– Data Acquisition and Ingest– Science Data Processing– Algorithm Integration, Testing and Evaluation– Data Archive and Distribution– Data Quality Control– Validation Support– Periodic Reprocessing
• Facilities/System Configuration/Backup Strategy/ IT Security• Integration and Testing• Staffing, Roles and Responsibilities• Documentation• Risks and Issues
Agenda
12 of 71February 8-10, 2010
Section 08 – Aquarius Ground System Overview and Data ProcessingMission Operations Review
Ground System Interfaces
AQUARIUS Ground System
13 of 71February 8-10, 2010
Section 08 – Aquarius Ground System Overview and Data ProcessingMission Operations Review
Ground System Interfaces
ADPS ACCSAquarius AquariusData Command &Processing ControlSystem System
14 of 71February 8-10, 2010
Section 08 – Aquarius Ground System Overview and Data ProcessingMission Operations Review
Ground System Interfaces
ADPS ACCSAquarius AquariusData Command &Processing ControlSystem System
CONAEData Mission RFArchive Operations Services
CommandCenterAquarius
Data, SAC-D Telemetry and OrbitProducts
CUSS CODS
1- SAC-D Ground System to Aquarius Ground System ICD
15 of 71February 8-10, 2010
Section 08 – Aquarius Ground System Overview and Data ProcessingMission Operations Review
Ground System Interfaces
ADPS ACCSAquarius AquariusData Command &Processing ControlSystem System
CONAEData Mission RFArchive Operations Services
CommandCenter
Aquarius Command SchedulesAnd Reports
Schedule ConfirmationAnd Reports
1- SAC-D Ground System to Aquarius Ground System ICD
Aquarius Data, SAC-D Telemetry and OrbitProducts
CUSS CODS
16 of 71February 8-10, 2010
Section 08 – Aquarius Ground System Overview and Data ProcessingMission Operations Review
Ground System Interfaces
ADPS ACCSAquarius AquariusData Command &Processing ControlSystem System
CONAEData Mission RFArchive Operations Services
CommandCenter
Aquarius Command SchedulesAnd Reports
S-Band
X-Band
Wallops Svalbard Alaska McMurdo
NASA Ground Network
S-Band
CommandsReal-TimeTelemetry &Coordination
Schedule ConfirmationAnd Reports
X-Band
ASI Matera
Cordoba
ASI Malindi
1- SAC-D Ground System to Aquarius Ground System ICD2- SAC-D / NASA PSLA, Network Requirements Document & Scheduling Interface3- SD - Observatory Ground IF Specification Document
Aquarius Data, SAC-D Telemetry and OrbitProducts
CUSS CODS
17 of 71February 8-10, 2010
Section 08 – Aquarius Ground System Overview and Data ProcessingMission Operations Review
Ground System Interfaces
ADPS ACCSAquarius AquariusData Command &Processing ControlSystem System
CONAEData Mission RFArchive Operations Services
CommandCenter
Aquarius Command SchedulesAnd Reports
S-Band
X-Band
Wallops Svalbard Alaska McMurdo
NASA Ground Network
S-Band
CommandsReal-TimeTelemetry &Coordination
Schedule ConfirmationAnd Reports
X-Band
ASI Matera
Cordoba
Principal Investigator andScience Planning Team
Calibration &Instrument
ConfigurationChange Requests
InstrumentEngineering TeamAquarius Instrument Testbed
Flight Software Updates
ASI Malindi
1- SAC-D Ground System to Aquarius Ground System ICD2- SAC-D / NASA PSLA, Network Requirements Document & Scheduling Interface3- SD - Observatory Ground IF Specification Document4- Aquarius Science Team to Aquarius Ground System ICD5- AQ - Aquarius Ground System – Aquarius Instrument Team Anomaly ICD and Aquarius Test bed User Manual
Aquarius Data, SAC-D Telemetry and OrbitProducts
CUSS CODS
18 of 71February 8-10, 2010
Section 08 – Aquarius Ground System Overview and Data ProcessingMission Operations Review
Ground System Interfaces
ADPS ACCSAquarius AquariusData Command &Processing ControlSystem System
CONAEData Mission RFArchive Operations Services
CommandCenter
Aquarius Command SchedulesAnd Reports
S-Band
X-Band
Wallops Svalbard Alaska McMurdo
NASA Ground Network
Science Team andAlgorithm Providers
S-Band
Algorithms &Science Code
Evaluation Products
Ancillary Data Sources
CommandsReal-TimeTelemetry &Coordination
Schedule ConfirmationAnd Reports
X-Band
ASI Matera
Cordoba
Principal Investigator andScience Planning Team
Calibration &Instrument
ConfigurationChange Requests
InstrumentEngineering TeamAquarius Instrument Testbed
Flight Software Updates
ASI Malindi
1- SAC-D Ground System to Aquarius Ground System ICD2- SAC-D / NASA PSLA, Network Requirements Document & Scheduling Interface3- SD - Observatory Ground IF Specification Document4- Aquarius Science Team to Aquarius Ground System ICD5- AQ - Aquarius Ground System – Aquarius Instrument Team Anomaly ICD and Aquarius Test bed User Manual6- Aquarius Data Product User Guide (PO.DAAC) & Format Specification (ADPS)7- Ocean Data Processing System Data and Software Management Plan, ATBD’s and Software User’s Guides
Aquarius Data, SAC-D Telemetry and OrbitProducts
CUSS CODS
19 of 71February 8-10, 2010
Section 08 – Aquarius Ground System Overview and Data ProcessingMission Operations Review
Ground System Interfaces
ADPS ACCSAquarius AquariusData Command &Processing ControlSystem System
CONAEData Mission RFArchive Operations Services
CommandCenter
Aquarius Command SchedulesAnd Reports
S-Band
X-Band
Wallops Svalbard Alaska McMurdo
NASA Ground Network
Science Team andAlgorithm Providers
S-Band
Algorithms &Science Code
Evaluation Products
Ancillary Data Sources
CommandsReal-TimeTelemetry &Coordination
Schedule ConfirmationAnd Reports
AVDS Aquarius ValidationData System
Insitu DataBias Adjustments
Aquarius Match-up Data
X-Band
ASI Matera
Cordoba
Principal Investigator andScience Planning Team
Calibration &Instrument
ConfigurationChange Requests
InstrumentEngineering TeamAquarius Instrument Testbed
Flight Software Updates
ASI Malindi
1- SAC-D Ground System to Aquarius Ground System ICD2- SAC-D / NASA PSLA, Network Requirements Document & Scheduling Interface3- SD - Observatory Ground IF Specification Document4- Aquarius Science Team to Aquarius Ground System ICD5- AQ - Aquarius Ground System – Aquarius Instrument Team Anomaly ICD and Aquarius Test bed User Manual6- Aquarius Data Product User Guide (PO.DAAC) & Format Specification (ADPS)7- Ocean Data Processing System Data and Software Management Plan, ATBD’s and Software User’s Guides8- A V D S to A D P S ICD
Aquarius Data, SAC-D Telemetry and OrbitProducts
CUSS CODS
20 of 71February 8-10, 2010
Section 08 – Aquarius Ground System Overview and Data ProcessingMission Operations Review
Ground System Interfaces
ADPS ACCSAquarius AquariusData Command &Processing ControlSystem System
CONAEData Mission RFArchive Operations Services
CommandCenter
Aquarius Command SchedulesAnd Reports
S-Band
X-Band
Wallops Svalbard Alaska McMurdo
NASA Ground Network
Science Team andAlgorithm Providers
S-Band
Algorithms &Science Code
Evaluation Products
Ancillary Data Sources
CommandsReal-TimeTelemetry &Coordination
Schedule ConfirmationAnd Reports
AVDS Aquarius ValidationData System PO.DAAC
Level 1, 2, 3Aquarius Data
Data Users andScience Team
Aquarius L1, 2, 3 DataAncillary DataFeedback
SeaDASSoftware
X-Band
ASI Matera
Cordoba
Principal Investigator andScience Planning Team
Calibration &Instrument
ConfigurationChange Requests
InstrumentEngineering TeamAquarius Instrument Testbed
Flight Software Updates
ASI Malindi
1- SAC-D Ground System to Aquarius Ground System ICD2- SAC-D / NASA PSLA, Network Requirements Document & Scheduling Interface3- SD - Observatory Ground IF Specification Document4- Aquarius Science Team to Aquarius Ground System ICD5- AQ - Aquarius Ground System – Aquarius Instrument Team Anomaly ICD and Aquarius Test bed User Manual6- Aquarius Data Product User Guide (PO.DAAC) & Format Specification (ADPS)7- Ocean Data Processing System Data and Software Management Plan, ATBD’s and Software User’s Guides8- A V D S to A D P S ICD9- Aquarius Ground System to PO.DAAC ICD
Insitu DataBias Adjustments
Aquarius Match-up Data
Aquarius Data, SAC-D Telemetry and OrbitProducts
CUSS CODS
21 of 71February 8-10, 2010
Section 08 – Aquarius Ground System Overview and Data ProcessingMission Operations Review
• Overview and Status• Key Requirements• Previous Reviews• System Interfaces• Key Processes
– Data Acquisition and Ingest– Science Data Processing– Algorithm Integration, Testing and Evaluation– Data Archive and Distribution– Data Quality Control– Validation Support– Periodic Reprocessing
• Facilities/System Configuration/Backup Strategy/ IT Security• Integration and Testing• Staffing, Roles and Responsibilities• Documentation• Risks and Issues
Agenda
22 of 71February 8-10, 2010
Section 08 – Aquarius Ground System Overview and Data ProcessingMission Operations Review
Data Acquisition and Ingest
• Aquarius/SAC-D data downlink strategy comprises multiple, overlapping downlinks each day of Aquarius science high-rate data (HRD) and Aquarius and SAC-D housekeeping telemetry (HKT).– 3 to 5 passes per 24 hours over the Cordoba ground station.– 1 pass per week over Matera ground station, which is transferred
to Cordoba after pass.– 6 - 11 minutes per pass.– Downlink full science memory of 110 MB at 4Mbps (just under 4
minutes), repeating throughout pass.
• ADPS automatically acquires downlinks from the CUSS (separate files for Science, SAC-D/HKT) after each pass.
• Orbit Products including Ephemeris files are generated daily by the CODS and acquired (Two-line Elements, Predicted Ephemeris and Orbit Ephemeris, Ground Station Contacts, Ground Station Contacts Details, SAA Passes, Eclipses, Events,
Maneuvers List)• All files are ingested and cataloged automatically after
acquisition, and made available for processing.
23 of 71February 8-10, 2010
Section 08 – Aquarius Ground System Overview and Data ProcessingMission Operations Review
• Overview and Status• Key Requirements• Previous Reviews• System Interfaces• Key Processes
– Data Acquisition and Ingest– Science Data Processing– Algorithm Integration, Testing and Evaluation– Data Archive and Distribution– Data Quality Control– Validation Support– Periodic Reprocessing
• Facilities/System Configuration/Backup Strategy/ IT Security• Integration and Testing• Staffing, Roles and Responsibilities• Documentation• Risks and Issues
Agenda
24 of 71February 8-10, 2010
Section 08 – Aquarius Ground System Overview and Data ProcessingMission Operations Review
Aquarius Data Processing Development
Approach:1- Use of rapid prototypes evolving into ever more realistic full-mission simulations2- Verify that all requirements are being met3- All components of the system are being thoroughly exercised4- use of "real-world" testing of the end-to-end system on a routine basis - don’t just test for success
Philosophy:
1- Maintain Flexibility - things WILL change2- Strive for maintainability of code and documentation3- Testing, Testing and more Testing:
Get the mechanics right so that when data starts to flow, you can focus on the science
Implementation:1- Start with a full end-to-end data flow description2- Break into the component parts3- Use as close to real data and operational software as possible4- Continuously evolve the simulation to add more credible data, software and interfaces as they
become available
Result:1- All the supporting infrastructure, tools and procedures can be developed, tested and refined as the process continues2- Through this modular, flexible approach, the entire system can gradually evolve to a higher and higher level of fidelity until you have as close to a fully tested and functional launch-ready
system as possible prior to launch
25 of 71February 8-10, 2010
Section 08 – Aquarius Ground System Overview and Data ProcessingMission Operations Review
Algorithm Integration, Testing and Evaluation
working code (C, Fortran)
published algorithms
in-house development
written specifications (memos, emails)
prototype code (IDL, MATLAB?)
Science Software Development and
Algorithm Integration
Aquarius Teamalgorithms
issues and test results
Software Repository
Version Controlled Code and Tables
Including ACCS
software updates
ADPS Team
SeaDAS
Software Package
26 of 71February 8-10, 2010
Section 08 – Aquarius Ground System Overview and Data ProcessingMission Operations Review
Science Data Processing
The Aquarius science data processing consists of the following steps:
• Level-0 Preprocessor – process each HRD downlink file to produce a single time-ordered set of Aquarius science blocks and removes duplicates– developed by Joel Gales - ADPS.
• Level-0-to-1A – separate Level-0 files into orbits, unpack science data, incorporate overlapping ephemeris and SAC-D HKT and convert Aquarius HKT – developed by Joel Gales - ADPS.
• Telemetry Analysis – HKT data from Level-1A files analyzed for web display on the Aquarius web site – developed by Susie Kennison, Norman Kuring and Liang Hong - ADPS/ACCS.
• Interactive Telemetry Analysis - HKT files analyze using CONAE-provided telemetry viewer.
• Level-1A Merge – consolidate Level-1A files from overlapping downlinks into a single, best-quality full orbit product – Joel Gales - ADPS.
• Level-1A-to-2 – perform calibration, atmospheric correction and salinity retrieval for Aquarius science data – radiometer and scatterometer code delivered by RSS and JPL, respectively; integration and input/output code developed by Joel Gales and Liang Hong - ADPS.
• Level-2-to-3 Binning – Level-2 salinity retrievals for one day geographically projected and collected into equal-area bins – existing OBPG software modified by Joel Gales- ADPS.
• Level-2-to-3 Smoothing – Level-2 salinity retrievals for one day optimally interpolated into equal-area bins – algorithm by Jonathan Lilly/ESR implemented by Joel Gales - ADPS.
• Level-3 Binning and Mapping – Level-3 binned files aggregated to longer time periods (weekly, monthly, etc.) and projected onto map grid – existing OBPG software.
All processing steps are run automatically when all input data sets are available.All software exists in at least preliminary prelaunch form at this time.
27 of 71February 8-10, 2010
Section 08 – Aquarius Ground System Overview and Data ProcessingMission Operations Review
Aquarius Data Processing Overview
Level-0 Preprocessor
Level-0 to 1A
Level-1A to 2
Level-1A Merge
Raw Level-0Downlink
File
Level-1Aorbit
Level-2
Level-3 Map
SAC-DHKT Data
Level-2 to 3Binning
Level-2 to 3Smoothing
Level-3 Binning
Level-3 Mapping
DynamicAncillary
Data
StaticAncillary
Data
Level-1Atemp
Level-0
Level-3 Bin
Level-0Report
Level-1aReport
Automated TelemetryAnalysis Tool
Standard Data ProcessingWeb-Based Telemetry Monitoring Tools
TelemetryReport
PredictedDownlink
Times
In-Situ Match-upProcess
Level-2Match-upAVDSBias Adjustment
Web-Based Browse, Download and Order Tools
CONAE-providedInteractive Telemetry
Analysis Tool
SAC-DHKT Data
28 of 71February 8-10, 2010
Section 08 – Aquarius Ground System Overview and Data ProcessingMission Operations Review
• Overview and Status• Key Requirements• Previous Reviews• System Interfaces• Key Processes
– Data Acquisition and Ingest– Science Data Processing– Algorithm Integration, Testing and Evaluation– Data Archive and Distribution– Data Quality Control– Validation Support– Periodic Reprocessing
• Facilities/System Configuration/Backup Strategy/ IT Security• Integration and Testing• Staffing, Roles and Responsibilities• Documentation• Risks and Issues
Agenda
29 of 71February 8-10, 2010
Section 08 – Aquarius Ground System Overview and Data ProcessingMission Operations Review
Data Archive and Distribution
• Aquarius data product archiving at ADPS will be performed using the existing ODPS archive catalog database and RAID-based data storage system (> 1 PetaByte of current storage for SeaWiFS, MODIS, MERIS, OCTS, CZCS, etc.)
• Aquarius data distribution during the active life of the mission will be performed using the existing ODPS browse and order system (examples to follow).
• All source data and “validated” data products will be delivered to the JPL PO.DAAC for permanent archive, value-added services and long-term distribution (next slides).
30 of 71February 8-10, 2010
Section 08 – Aquarius Ground System Overview and Data ProcessingMission Operations Review
PO.DAAC Support for Aquarius
PO.DAAC (Physical Oceanography Distributed Active Archive Center) responsibilities:
• Long-term (Life of Mission and Beyond) data management, distribution & archive of the Aquarius mission data.
• Ensure data maintenance, usability, access, provenance and security.• Ensure integrity of Aquarius data through quality, quantity, continuity and latency
audits.• Ensure all Aquarius products conform to NASA’s file and metadata standards.• Ensure all Aquarius metadata are delivered to the NASA metadata registry (ECHO).• Implement and maintain PO.DAAC Aquarius ftp site and web pages.• Maintain data Aquarius data access tools including product/granule search,
OpenDAP, L2 and L3 subsetters.• Develop and maintain Aquarius product user guides and read software.• Answer user questions concerning Aquarius data.• Configure and implement the PO.DAAC Data Management & Archive System
(DMAS) to conform to the interface specification and product ingest requirements.• Report data access metrics.
31 of 71February 8-10, 2010
Section 08 – Aquarius Ground System Overview and Data ProcessingMission Operations Review
PO.DAAC Delivery Status
• Draft Interface Control Document (Completed September 19, 2008)• Draft Data Management Plan (Completed October 18, 2008)• Final Interface Control Document (February 2010)• Draft User Guide (February 2010)• Final Data Management Plan (May, 2010 2 months prior to
Operational Readiness Review)• Final User Guide (May, 2010)• DMAS adaptation (Ingest, Archive and Distribution tools) (March
2010 for Operations Readiness Test)• Aquarius FTP and web site (including read software) (June 2010 for
Operations Readiness Test)• Data Access tools (July 2010)
32 of 71February 8-10, 2010
Section 08 – Aquarius Ground System Overview and Data ProcessingMission Operations Review
Data Archive and Distribution
• Overview of the currently operational web-based browse, order and
distribution tool that is available to support the distribution of
Aquarius data within the Aquarius Data Processing System that also
serves as a data quality monitoring tool for both internal and
external use.
33 of 71February 8-10, 2010
Section 08 – Aquarius Ground System Overview and Data ProcessingMission Operations Review
Aquarius Level-1-2 Data Browserhttp://oceancolor.gsfc.nasa.gov/cgi/aquarius
34 of 71February 8-10, 2010
Section 08 – Aquarius Ground System Overview and Data ProcessingMission Operations Review
Aquarius Level-1-2 Data Browserhttp://oceancolor.gsfc.nasa.gov/cgi/aquarius
35 of 71February 8-10, 2010
Section 08 – Aquarius Ground System Overview and Data ProcessingMission Operations Review
Aquarius Level-1-2 Data Browserhttp://oceancolor.gsfc.nasa.gov/cgi/aquarius
36 of 71February 8-10, 2010
Section 08 – Aquarius Ground System Overview and Data ProcessingMission Operations Review
Aquarius Level-3 Data Browser
http://oceancolor.gsfc.nasa.gov/cgi/l3
37 of 71February 8-10, 2010
Section 08 – Aquarius Ground System Overview and Data ProcessingMission Operations Review
Beam 1 - daily
Aquarius Level-3 Data Browser
http://oceancolor.gsfc.nasa.gov/cgi/l3
38 of 71February 8-10, 2010
Section 08 – Aquarius Ground System Overview and Data ProcessingMission Operations Review
All Beams - daily
Aquarius Level-3 Data Browser
http://oceancolor.gsfc.nasa.gov/cgi/l3
39 of 71February 8-10, 2010
Section 08 – Aquarius Ground System Overview and Data ProcessingMission Operations Review
all beams-weekly
all beams-monthly
Aquarius Level-3 Data Browser
http://oceancolor.gsfc.nasa.gov/cgi/l3
40 of 71February 8-10, 2010
Section 08 – Aquarius Ground System Overview and Data ProcessingMission Operations Review
• Overview and Status• Key Requirements• Previous Reviews• System Interfaces• Key Processes
– Data Acquisition and Ingest– Science Data Processing– Algorithm Integration, Testing and Evaluation– Data Archive and Distribution– Data Quality Control– Validation Support– Periodic Reprocessing
• Facilities/System Configuration/Backup Strategy/ IT Security• Integration and Testing• Staffing, Roles and Responsibilities• Documentation• Risks and Issues
Agenda
41 of 71February 8-10, 2010
Section 08 – Aquarius Ground System Overview and Data ProcessingMission Operations Review
Data Quality Control
• Automated telemetry monitoring, alarm notification and web tools updated on every downlink
• Interactive telemetry monitoring tools
• Automated format and science data quality checks performed on each file as defined by science team
• Visual inspection via web tools for Level-2 and Level-3 data products
• Level-3 Long-term statistics monitoring tool
• AVDS data analysis by Science Team
42 of 71February 8-10, 2010
Section 08 – Aquarius Ground System Overview and Data ProcessingMission Operations Review
• Overview and Status• Key Requirements• Previous Reviews• System Interfaces• Key Processes
– Data Acquisition and Ingest– Science Data Processing– Algorithm Integration, Testing and Evaluation– Data Archive and Distribution– Data Quality Control– Validation Support– Periodic Reprocessing
• Facilities/System Configuration/Backup Strategy/ IT Security• Integration and Testing• Staffing, Roles and Responsibilities• Documentation• Risks and Issues
Agenda
43 of 71February 8-10, 2010
Section 08 – Aquarius Ground System Overview and Data ProcessingMission Operations Review
Validation Support
• The Aquarius Validation Data System (AVDS) will statistically evaluate the Aquarius SSS data product using a global set of in situ salinity measurements (e.g., Argos buoys)
• The ADPS will support the validation activity by acquiring the in situ data from AVDS, performing matchups with the Aquarius Level-2 data products, and uploading the matchup files to AVDS.
• The rationale for the data exchange is to streamline this activity by using preexisting ADPS hardware, software, and satellite data processing facilities, thereby eliminating the redundant effort of developing parallel facilities.
• A successful data exchange simulation was accomplished using in situ water temperature measurements and the MODIS-Aqua 11-micron SST data product.
• The AVDS support testing will be included in the mission simulation described under Integration and Testing.
44 of 71February 8-10, 2010
Section 08 – Aquarius Ground System Overview and Data ProcessingMission Operations Review
AVDS-ADPS Data Exchange
in situ data uploaded (daily)
ftp://ftp.esr.org/pub/datasets/avds
ftp://samoa.gsfc.nasa.gov/aquarius
FTP SITESin situ data downloaded daily
limited to new files
in situ data ingested into database
satellite files located & extracted
level-2 hdf files generated
browse images & maps generated
seabass:/data3/adps
DATA STORAGEhttp://seabass.gsfc.nasa.gov/eval/adps.cgi
WEB SITE
satellite data uploaded & posted online
satellite data removed after 1-week
AVDS RESPONSIBILITY
45 of 71February 8-10, 2010
Section 08 – Aquarius Ground System Overview and Data ProcessingMission Operations Review
• Overview and Status• Key Requirements• Previous Reviews• System Interfaces• Key Processes
– Data Acquisition and Ingest– Science Data Processing– Algorithm Integration, Testing and Evaluation– Data Archive and Distribution– Data Quality Control– Validation Support– Periodic Reprocessing
• Facilities/System Configuration/Backup Strategy/ IT Security• Integration and Testing• Staffing, Roles and Responsibilities• Documentation• Risks and Issues
Agenda
46 of 71February 8-10, 2010
Section 08 – Aquarius Ground System Overview and Data ProcessingMission Operations Review
Periodic Reprocessing
• Level-1 Requirement: No later than twelve (12) months after the end of the IOC period, the Aquarius Project shall record, process, archive, and deliver the first release of Level 1A and Level 3 data products to a NASA Distributed Active Archive Center (DAAC),
• Level-1 Requirement: After the first release of Aquarius data, the Aquarius Level 1A and Level 3 data products shall be delivered to the DAAC no later than six months after measurements are taken in orbit.
• Level-1 Requirement: During the mission life, all products shall be available to the science community from the Aquarius Data Processing Segment (ADPS) and the final data products produced by the Aquarius Project shall be delivered to the DAAC within six (6) months after the end of the prime mission
47 of 71February 8-10, 2010
Section 08 – Aquarius Ground System Overview and Data ProcessingMission Operations Review
Periodic Reprocessing
• Development and test environments allow for multiple algorithm versions to be run on the entire mission data set to test the impacts of calibration and algorithm improvements.
• The results of these “reprocessings” are made immediately available to the science team members via web and ftp mechanisms
• Using SeaWiFS as a benchmark, the most recent reprocessing of the entire mission-long (ten year) data set required 10 hours or 1 year/hour. Aquarius is ½ the data volume of SeaWiFS so data storage and system throughput are NOT a concern. A better estimate of the actual processing throughput will be derived once we process the year-long Aquarius simulated data set later this month
• Once the calibration/algorithm changes have been validated and approved by the Science Team, a mission-long reprocessing will be initiated and the new products and documentation will be made available to the science community and PO.DAAC.
48 of 71February 8-10, 2010
Section 08 – Aquarius Ground System Overview and Data ProcessingMission Operations Review
• Overview and Status• Key Requirements• Previous Reviews• System Interfaces• Key Processes
– Data Acquisition and Ingest– Science Data Processing– Algorithm Integration, Testing and Evaluation– Data Archive and Distribution– Data Quality Control– Validation Support– Periodic Reprocessing
• Facilities/System Configuration/Backup Strategy/ IT Security• Integration and Testing• Staffing, Roles and Responsibilities• Documentation• Risks and Issues
Agenda
49 of 71February 8-10, 2010
Section 08 – Aquarius Ground System Overview and Data ProcessingMission Operations Review
OBPG Computing Facility
NASA Goddard Space Flight CenterGreenbelt, Maryland
Server location: Building 28 Room W220
- 900 square feet dedicated to the project
- Raised computer floor with under-floor A/C
- Primary and backup A/C units
- 80 KVA Powerware UPS unit with dual 50 KVA PDUs, additional 5 KVA rack-mounted UPS units
- Connections to GSFC high speed networks: SEN
50 of 71February 8-10, 2010
Section 08 – Aquarius Ground System Overview and Data ProcessingMission Operations Review
OBPG System Hardware
Processing Cluster46 processing nodes 15 TB
4 storage nodes 10.9 TB
Ingest Servers2 SeaSpace workstations 78
GB1 storage node 5.1 TB
Distribution Servers (ftp)
8 storage nodes 49.3 TB1 server 250GB
Distribution and Storage Servers (web)
Production: 3 processing nodes75 storage nodes 1.03 PB
Testing Cluster
Network Support Servers
Backup Servers2 storage nodes 19.2 TB
Extreme NetworksBlack Diamond 8810
Gigabit Ethernet switch480 ports
Cal/Val & QC Systems
Mission Operations Systems
Development Servers2 superstore nodes 10 TB
User Desktops
Subversion CM Servers2 superstore nodes 586 GB
Database Servers2 superstore nodes 16.4 TB
AQOPS Servers (web)Production: 2 processing nodes
51 of 71February 8-10, 2010
Section 08 – Aquarius Ground System Overview and Data ProcessingMission Operations Review
Aquarius Backup Strategy
• Data• On-site
• Downlinks, L0, L1a data files• primary copy on RAID6 storage node• mirror copy on separate RAID6 storage node, at file creation
• Science Products• primary copy on RAID6 storage node• mirror copy if desired, or regenerated from L1a
• Off-site• Downlinks: stored at CONAE• L1A archived at PO.DAAC
• Source Code• On-site
• primary copy on subversion server, RAID6• mirror copy on backup Subversion server, RAID6, nightly mirror
• Off-site• Subversion data copied to external disk, taken off-site every 1 month• Multiple copies, given to Project Manager, SE, Processing Manager
• Other• On-site
• AQOPS, Forum: database mirrored to backup server nightly• Telemetry Viewer, other: image backups periodically, manual
• Off-site• AQOPS, Forum: databases included in source code off-line backups• Telemetry Viewer, other: image backups stored off-line like source code
52 of 71February 8-10, 2010
Section 08 – Aquarius Ground System Overview and Data ProcessingMission Operations Review
IT Security Plan
• ADPS and ACCS: covered under the same plan as other Ocean Biology Processing Group (OBPG) systems.
• NASA IT Security Plan Identification– System name: Sciences and Exploration Directorate (SED) Multi-
Program/Project (MPP) IT Science Systems– System Abbreviation: SED MPP– System Unique Identifier: CD-014-L-GSF-6004– Information System Type: General Support System– Information System Security Categorization: Low (transitioning to
medium within the year)– Information System Operational Status: Operational
• Effective 5/31/2007. Passed Certification & Accreditation in Summer 2007 - in process of being upgraded
• Plan designated Sensitive But Unclassified by NASA
• Implementation questions: addressed off-line
53 of 71February 8-10, 2010
Section 08 – Aquarius Ground System Overview and Data ProcessingMission Operations Review
• Overview and Status• Key Requirements• Previous Reviews• System Interfaces• Key Processes
– Data Acquisition and Ingest– Science Data Processing– Algorithm Integration, Testing and Evaluation– Data Archive and Distribution– Data Quality Control– Validation Support– Periodic Reprocessing
• Facilities/System Configuration/Backup Strategy/ IT Security• Integration and Testing• Staffing, Roles and Responsibilities• Documentation• Risks and Issues
Agenda
54 of 71February 8-10, 2010
Section 08 – Aquarius Ground System Overview and Data ProcessingMission Operations Review
Integration and Testing
• ODPS Testing Philosophy:
– As described in the Aquarius Data Processing Development section
• Phased approach to testing: get one thing working and move on to the next step until entire end-to-end stream is operational.
• Testing Aquarius support to be done within operational ODPS environment, with products clearly identified for separation from operational product stream.
• Testing stages:– Interfaces
– Internal functionality (ingest, processing, evaluation, distribution)
– End-to-end (Mission level testing, simulations and rehearsals)
• Test data sources:– Mission simulation
– Aquarius I&T data
55 of 71February 8-10, 2010
Section 08 – Aquarius Ground System Overview and Data ProcessingMission Operations Review
Functional Testing
• Functional testing has been completed for most OBPG developed software using simulated Aquarius data:– Level-0 Preprocessor
– Level-0-to-1A*
– Level-1A Merge
– Telemetry Analysis
– Level 2-to-3 Binning**
– Level 2-to-3 Smoothing**
– In situ Matchup**
• Functional testing of Telemetry Analysis software is in progress using I&T files from Bariloche.
• Functional testing for updated Level-1A-to-2 software will begin March 2010
* Need to re-test when final SAC-D HKT format is implemented and made available.
** Need to re-test with new file formats once updated code has been implemented (March 2010)
56 of 71February 8-10, 2010
Section 08 – Aquarius Ground System Overview and Data ProcessingMission Operations Review
Interface Testing – Current
• The CONAE MOC interface has been tested using one month of simulated downlink files staged on interim SFTP site, acquired and ingested by ADPS. Recent tests have also been conducted on the CONAE CUSS data server. Anticipate testing CODS interface and products along with actual SAC-D HKT telemetry file in March.
• The Science Team interface has been tested via algorithm code deliveries and distribution of data products generated from simulated data.
• The AVDS interface was tested using SST in situ and MODIS data in 2006 and is awaiting update to processing software to resume.
• The Ancillary Data interfaces for current data sources are exercised routinely as part of ongoing data processing activities.
57 of 71February 8-10, 2010
Section 08 – Aquarius Ground System Overview and Data ProcessingMission Operations Review
Interface Testing – Planned
• The CONAE MOC interface using the CUSS and CODS will be tested according to the CONAE I&T schedule:
– Simulated Aquarius downlink files, HKT and ephemeris staged by the Aquarius GS (ongoing).
– Sample SAC-D orbit products and HKT files provided by CONAE (March)
– Aquarius collected during I&T (ongoing) .
– The X-band end-to-end test proposed by NASA (next page).
• This interface, and the Aquarius processing software, will also be exercised extensively during the Mission Simulation (following pages).
• The Ancillary Data interfaces for new sources identified as part of the recent Science Code delivery will be tested in parallel with the functional testing of the integrated software.
• The PO.DAAC interface will be tested starting in March 2010 once final data products with complete formats are generated.
58 of 71February 8-10, 2010
Section 08 – Aquarius Ground System Overview and Data ProcessingMission Operations Review
X-band End-to-end Test Configuration
Goal is to verify the downlink file
format containing actual Aquarius
instrument data as produced and
delivered by the CONAE MOC. This
test is absolutely CRITICAL to verify
the actual downlink file format upon
which all of our subsequent
software is based.
Currently scheduled for April 2010
which provides enough time to
resolve any differences
59 of 71February 8-10, 2010
Section 08 – Aquarius Ground System Overview and Data ProcessingMission Operations Review
Aquarius Mission Simulation
• The Aquarius Mission Simulation will be a running, day-by-day exercise of the Aquarius/CONAE MOC interface and the ADPS.
• The simulation will follow this daily sequence of events:– The Radiometer (RSS) and Scatterometer (JPL) algorithm teams will
deliver a day’s worth of simulated instrument data to the Aquarius GS.
– The GS will generate simulated downlink (HRD and HKT) files for a typical pass schedule and a simulated daily ephemeris file.
– The simulated files will be staged on the CONAE servers.
– The ADPS will automatically detect, acquire and ingest the files, and process them through all product levels.
– The products will be archived, cataloged, and made available for distribution through the Browse and Order interface and to the PO.DAAC.
– The Aquarius Science Team will use this simulation to test their tools, data quality monitoring processes and anomaly detection.
• The Mission Simulation will commence in early March 2010 and will continue until the Aquarius/SAC-D launch.
60 of 71February 8-10, 2010
Section 08 – Aquarius Ground System Overview and Data ProcessingMission Operations Review
Aquarius Simulated Data Set
The Mission Simulation starts with simulated sensor data generated by members of the Aquarius Science Team, using assumed salinity fields and spacecraft orbit data, actual ancillary data and models of the instrument response
61 of 71February 8-10, 2010
Section 08 – Aquarius Ground System Overview and Data ProcessingMission Operations Review
Aquarius Simulated Data Set
Once the simulated sensor files are created, these are formatted into Aquarius science blocks and packaged into realistic downlink files based on predicted contact times and durations which are as close to the actual downlink data format as possible given the currently available information and sample telemetry
62 of 71February 8-10, 2010
Section 08 – Aquarius Ground System Overview and Data ProcessingMission Operations Review
Integration and Test Schedule
• Most of the interfaces defined in slide #20 have already been tested with first generation data products.
• The Aquarius Data Processing System will be ready to test all interfaces and participate in all Mission Level tests with at-launch data products in March 2010
• Subsequent tests and schedules are presented as part of the Mission Readiness Testing Section 17 presentation
63 of 71February 8-10, 2010
Section 08 – Aquarius Ground System Overview and Data ProcessingMission Operations Review
• Overview and Status• Key Requirements• Previous Reviews• System Interfaces• Key Processes
– Data Acquisition and Ingest– Science Data Processing– Algorithm Integration, Testing and Evaluation– Data Archive and Distribution– Data Quality Control– Validation Support– Periodic Reprocessing
• Facilities/System Configuration/Backup Strategy/ IT Security• Integration and Testing• Staffing, Roles and Responsibilities• Documentation• Risks and Issues
Agenda
64 of 71February 8-10, 2010
Section 08 – Aquarius Ground System Overview and Data ProcessingMission Operations Review
Ground System Staffing
NASA Management - Gene Feldman: Oversight of the overall effort, communication/coordination with other NASA organizations and centers and CONAE; oversight of science support, algorithm integration, QC, validation and in situ data support.
Contractor Management - Fred Patt: Oversight of contract support staff, including hiring, tasking, review of deliverables and schedules; ADPS interface definition and product format specification.
Mission Operations - Susie Kennison: Technical lead for instrument operations and commanding support; interface to instrument engineers and CONAE mission operations.
Data Systems Engineering - John Wilding: Technical lead for the data system integration effort; overall system design, implementation and testing.
Software Engineering – Joel Gales: Technical lead for the level conversion / product generation software, including development of the Level 0-to-1 software and integration/packaging of the science (Level 2 and 3) processing software to be provided by the algorithm developers.
Data Systems Support - Dan Knowles: Integrate the product generation software into the existing automated processing system, develop ingest modules, develop database tables for archive products, develop schedules and recipes for data acquisition, production and migration.
Systems Administration - Paul Smith: Specify and acquire new processing and data storage hardware, integrate into existing data processing facility, verify/upgrade network capability for external data transfers.
Quality Control and Validation - Liang Hong: Working with the Science Team, develop methodologies, tools, procedures and reports for operational QC; specify data and develop algorithms and tools for data and science algorithm validation.
In Situ Data Support - Jeremy Werdell: Work closely with the Aquarius Validation Data System (AVDS) to develop methods and tools for data analysis and quality control, building upon the existing SeaWiFS Bio-optical Archive and Storage System (SeaBASS).
Archive and Distribution - Norman Kuring: Incorporate Aquarius product browse, search, order and distribution into existing web-based Ocean Color capability and work with JPL PO.DAAC regarding “institutional” archive support requirements.
65 of 71February 8-10, 2010
Section 08 – Aquarius Ground System Overview and Data ProcessingMission Operations Review
• Overview and Status• Key Requirements• Previous Reviews• System Interfaces• Key Processes
– Data Acquisition and Ingest– Science Data Processing– Algorithm Integration, Testing and Evaluation– Data Archive and Distribution– Data Quality Control– Validation Support– Periodic Reprocessing
• Facilities/System Configuration/Backup Strategy/ IT Security• Integration and Testing• Staffing, Roles and Responsibilities• Documentation• Risks and Issues
Agenda
66 of 71February 8-10, 2010
Section 08 – Aquarius Ground System Overview and Data ProcessingMission Operations Review
ADPS Documentation
Name Status Version Date
Aquarius Ground System Level-3 Requirements (AQ-326-0152) Baselined
L3 Aquarius Science Algorithm Requirements (AQ-322-135) Baselined March 30, 2008
Aquarius JPL to GSFC Operations Transition Plan (AQ-211-0261)
Preliminary November 25, 2009
NASA/GSFC Ocean Data Processing System Operations and Data Management Plan
Approved April 2, 2009
Aquarius Ground System Integration Test & Training Plan (AQ-336-0244)
Revision
SAC-D Ground System to Aquarius Ground System ICD (AS-336-0151b)
Review July 14, 2008
AVDS to ADPS ICD (AQ-332-0331) Signed December 18, 2006
Aquarius Science Team to Aquarius Ground System ICD (AQ-336-0243)
Review June 17, 2009
Aquarius Ground System to PO.DAAC ICD (AQ-336-0151d) Review January 4, 2010
Level-1A Product Format Specification (AQ-376-0238) Final September 22, 2009
Level-2 Product Format Specification (AQ-376-0238) Preliminary October 14, 2009
Level-3 Product Format Specification (AQ-376-0238) Review May 26, 2009
67 of 71February 8-10, 2010
Section 08 – Aquarius Ground System Overview and Data ProcessingMission Operations Review
Aquarius Website
Consolidated Data access,
Information,
Services and
Community Feedback
68 of 71February 8-10, 2010
Section 08 – Aquarius Ground System Overview and Data ProcessingMission Operations Review
Aquarius Wiki
Online Documentation and Software Descriptions available online at:
http://aquarius.nasa.gov/
Under Data and Documents Link labeled “Aquarius Ground System and Mission Operations Wiki Documentation”
Major WIKI documentation update planned in March/April 2010
69 of 71February 8-10, 2010
Section 08 – Aquarius Ground System Overview and Data ProcessingMission Operations Review
• Overview and Status• Key Requirements• Previous Reviews• System Interfaces• Key Processes
– Data Acquisition and Ingest– Science Data Processing– Algorithm Integration, Testing and Evaluation– Data Archive and Distribution– Data Quality Control– Validation Support– Periodic Reprocessing
• Facilities/System Configuration/Backup Strategy/ IT Security• Integration and Testing• Staffing, Roles and Responsibilities• Documentation• Concerns and Summary
Agenda
70 of 71February 8-10, 2010
Section 08 – Aquarius Ground System Overview and Data ProcessingMission Operations Review
Key Concerns
• Awaiting delivery of actual CONAE-produced data products (Aquarius science data in downlink format, SAC-D HKT data file and CODS-produced orbit products) to verify formats and contents and resolve any discrepancies or modify our software as required. Anticipated delivery of preliminary versions in March 2010 and key Aquarius data test in April 2010)
• Mission rehearsals are not yet clearly defined in scope, participation and success criteria and overall test coordination needs to be clarified .
71 of 71February 8-10, 2010
Section 08 – Aquarius Ground System Overview and Data ProcessingMission Operations Review
Summary
• Aquarius Data Processing System is being built within an operational and already proven and well-tested end to end data system serving multiple missions in a modular, flexible and easily expandable manner.
• Implementation of major components of the science data processing system are in place from ingest to data distribution.
• Smooth and seamless integration of improved algorithms into the development, test and operational environments as they become available.