Colleagues:

Attached is a marked up copy of the doc we walked through this morning,
also appended below as plain text.

Chip



LQCD I/O Design

This summarizes the results of our morning session on Lattice QCD I/O. 
Points of consensus are labeled AGREED. Points needing additional work 
are labeled TBD (to be determined).

1. Multiple file layout
a. AGREED: single or multi-level meta-data file(s) for a multi-file 
dataset (i.e. the common meta-data)
+ AGREED: pure XML (no additional record structure as used for binary)
b. AGREED: mixed files (binary + meta-data) for the data itself 
+ single file, not separated into 2 or more files of metadata, binary data
c. pointers / references / links (should files reference each other)
+ AGREED: all files should have a unique ID (divorced from project 
organization), like a global inode, which never changes
+ AGREED: navigate from metadata files to data? YES, by external entity
+ AGREED: navigate from data to metadata? YES, by external entity
+ AGREED: all files shoul have absolute global file name (GFN)
e.g. /MILC/physicsA/sub-project7/file37
may change if project is restructured
mapping exists between global ID GID and GFN

2. Mixed file formats
a. AGREED: structure is multiple records << 1,000,000
b. AGREED: DIME as the carrier for mixed data
+ TBD: recursion to be determined
c. AGREED: some records carry pure XML meta-data,
+ e.g. 1st record might contain UID, ... physics XML
+ this type of content standardization TBD
d. TBD: for binary data...
+ separate pure binary record (with data-type info in separate record)
+ or recursive structure with one part data-type info, one part binary
e. AGREED: use subset of BINX as the starting point for XML schema for 
data-type description, but dont track (as a standard) BINXs evolution
f. TBD: extended data-type info (schema) to include QDP object type 
(e.g. gauge field)

3. Programmers view of the data (the API)
a. AGREED: meta-data I/O
+ AGREED: will provide namelist style reader / writer (convenience 
routines)  hide XML, exact API TBD
+ AGREED: full blown XML parser (dom, sax) is always possible, but not 
encouraged
b. TBD: binary buffer I/O
+ something like: write (type-info, buffer, nbytes)
+ needs further discussion
c. QDP object I/O
+ AGREED: one line of code to write out a QDP object (C or C++)
+ TBD: c++: stream oriented, where special stream provides both 
type-info, bytes
d. TBD: parallel file system
+ N nodes to 1 writer
+ N writers
+ N nodes to M writers
Goal: choice is made at open time, not I/O time

-- 
Chip Watson
High Performance Computing Group
Thomas Jefferson National Accelerator Facility
Tel: (757) 269-7101
http://www.jlab.org/~watson