Subject: Draft QIO/C-0.1 Hi Rich, Attached is a draft proposal for the binary file format, the QDP I/O API and for the underlying implementation components. While I called it QIO/C-0.1, I hope that most of the implementation components can be shared between the C and C++ interfaces. Most of what is here arises from discussions by the I/O subcommittee (Edwards, Hetrick, Osborn, Simone, and me), which began before the All Hands meeting. However, I have taken the liberty to enlarge and add more details, which have not been scrutinized by the subcommittee. Nonetheless, we felt that this approach would help focus the plenary discussion on Thursday. What is missing from this document are details of the XML management API. This discussion is naturally separate from the present proposal. Jim Simone has done most of the work here, and we will circulate some elements of his proposals next week. The draft proposal is organized with an overview at the top, followed by the QDP API, followed by details of the underlying implementation API's. Regards, Carleton ---------------------------------------------------------------------- Draft SciDAC I/O Implementation March 25, 2003 This document enlarges on the discussions of the I/O subcommittee on March 18 and 25 (Brower, Edwards, Simone, DeTar, Osborn). It attempts an overview of the components needed to implement the I/O system. The goal is to share these components between the C and C++ implementation wherever possible. ---------------------------------------------------------------------- File format (sketch) Logical file layout: all components are logical records with DIME packaging. Chunking is possible. Private file XML Reserved for the QIO system File XML Application information Private record XML Reserved for the QIO system Record XML BinX These five map to a single QDP field Binary data or array of fields. They may be Checksum XML repeated ad lib. The binary data is arranged in site-major order - i.e. the bytes for the datum on one site are contiguous. ---------------------------------------------------------------------- Software heirarchy: 0. QDP | 1. QIO -----> XML | \ | \--("QDPstream")-> 1b DWL --> "factory fcn" ("stream") | v 2. LRL | | 3. OS The XML parsing and manipulation functionality is separate and not discussed here. 0. QDP Top Level API. This is just a wrapper layer to create what we had been calling the "QIO-Lite" API. 1. QIO layer This layer reads and writes QDP/QLA objects and reads and writes XML data. It manages the file format and orchestrates the data flow between the DWL and LRL. It doesn't know about the data layout. 1b. DWL Data wrangling layer This layer knows the layout and the volume format. Its principal function is to get/put one site's worth of data in the correct file site order on the correct node. It uses QMP to move data among nodes. The "factory" function delivers a QLA datum for a specified site on a node. 2. LRL Logical record layer This layer runs on a single node and accesses its own file (or portion of a single file). It has no knowledge of the data layout. It the architecture supports it, the same file can be accessed by more than one node, but that is handled by the OS. This layer includes the DIME parser. It presents data to the next layer up as a series of logical records consisting of bytes. No DIME information - e.g. chunking - is needed above this layer. The upward interface consists of open, seek, read, write, close. This functionality is needed both for files and records. Seek is needed only for multinode access to the same file, and then only on opening a record to position the file pointer at the correct starting byte for that node. 3. OS layer (the usual system calls for open, seek, read, write, close including parallel opens if supported by the architecture.) What follows are some details for the API's in layers 0, 1, 1b, and 2. ---------------------------------------------------------------------- QDP/C API -- this is what the application programmer uses QDP_Reader *QDP_open_read(XML_MetaData *fileXML, char *filename, int serialparallel) QDP_Writer *QDP_open_write(XML_MetaData *fileXML, char *filename, int serialparallel, int siteorder, int mode) API Opens a single file for writing or reading and writes/extracts the fileXML information. serialparallel is "QDP_SERIAL" or "QDP_PARALLEL". If the architecture does not support parallel reads/writes the mode reverts to serial. siteorder is "QDP_LEX_ORDER" or "QDP_NAT_ORDER" for lexicographic or native order. This specification applies only when writing the file. mode is "QDP_CREATE", "QDP_APPEND", or "QDP_TRUNCATE" This specification applies only when writing the file. Implementation Note: On the QCDSP the file may be locally distributed across multiple disks with a subset of nodes assigned to each disk. The staging process would take care of assembling/disassembling the single external file, but this is invisible to the application. QDP_Reader *QDP_open_read_multidump(XML_MetaData *fileXML, char *filename) QDP_Writer *QDP_open_write_multidump(XML_MetaData *fileXML, char *filename) API Opens one file per node for writing. This format is intended for rapid reading/writing of intermediate files that are not needed after the job is done. Depending on the implementation, the actual file names may vary from node-to-node, for example, by appending the node number to the filename. Implementation Note: On the QCDSP the number of files may be less than the number of nodes. The difference from above is that the staging system would not be asked to move the file offline. Some of the file XML is required in order to interpret the volume format: e.g. ordered site list, format (multidump or single). One way to enforce this is to append this private XML to the user-supplied file XML upon writing the file and to strip it when reading the file. The site list is obtained from the DWL layer. int QDP_write_T(QDP_Writer *fd, XML_MetaData *recXML, QDP_Type *field); The suffix T matches the type QDP_Type Writes a physics record containing a single QDP field. Returns status. int QDP_vwrite_T(QDP_Writer *fd, XML_MetaData *recXML, QDP_Type *field, int n); Writes an array of n fields in site-major order - i.e. at each site we regard the data as consisting of n QLA values. int QDP_write_t(QDP_Writer *fd, XML_MetaData *recXML, QLA_Type *data); Writes a scalar. int QDP_vwrite_t(QDP_Writer *fd, XML_MetaData *recXML, QLA_Type *data, int n); Writes an array of scalars. int QDP_read_T(QDP_Reader *fd, XML_MetaData *recXML, QDP_Type *field); int QDP_vread_T(QDP_Reader *fd, XML_MetaData *recXML, QDP_Type *field, int n); int QDP_read_t(QDP_Reader *fd, XML_MetaData *recXML, QLA_Type *data); int QDP_vread_t(QDP_Reader *fd, XML_MetaData *recXML, QLA_Type *data, int n); int QDP_peek_metadata(QDP_Reader *fd, XML_MetaData *recXML); API For peeking at the physics record metadata, but not reading the binary data. Implementation Reads the record XML through an LRL call and then backs up to the beginning of the record. int QDP_skip_record(QDP_Reader *infile, int n); API Skip n records from the current position. Implementation Calls LRL to seek +5n DIME records. int QDP_seek_record(QDP_Reader *infile, int n); API Go to the n th record (forwards or backwards) Implementation Calls LRL to seek the 5n + 2nd record (private XML and file XML are 1st & 2nd records) ---------------------------------------------------------------------- QIO level function calls QIO_Reader *QIO_open_read(XML_MetaData *fileXML, char *filename, int serialparallel) QIO_Writer *QIO_open_write(XML_MetaData *fileXML, char *filename, int serialparallel, int siteorder, int mode) QIO_Reader *QIO_open_read_multidump(XML_MetaData *fileXML, char *filename) QIO_Writer *QIO_open_write_multidump(XML_MetaData *fileXML, char *filename) (see the corresponding QDP calls) int QIO_write(QIO_Writer *fd, XML_MetaData *recXML, QLA_factory *f); API Writes a group of logical records as a single unit. These consist of the private record XML, the physics record XML, a BinX record, the binary payload, and an XML checksum record. f a structure with member function void (*get)(char *buf, int coords[]) and member integer int size so that f.get(buf, coords) copies all the needed data for the site at the specified coordinate coords to the buffer "buf" of size "size". The data has to be presented in the standard file layout order for the resulting "QLA" datatype. Implementation The structure f is set up by the wrapper functions "QDP_write_T" and "QDP_vwrite_T". It also encodes the number of array elements for "vector" writes, i.e. writing n fields with data organized as an array of the corresponding QLA types on each site. For scalars the factory structure simply would rearrange the QLA datum into the standard file byte layout. Within QIO, we must also generate these records BinX For now the BinX field is a prestored constant for each field type. Calls the LRL to write the record. Binary data Calls the DWL site-by-site to collect the data and assemble the record in a buffer. Calls the LRL to write the record. Checksum Asks the DWL for the checksum. Calls the LRL to write. int QIO_read(QIO_Reader *fd, XML_MetaData *recXML, QLA_factory *f); f Has a member function void (*put)(char *buf, int coords[]) and member integer size put is a callback function for copying data from "buf" of size "size" to the QDP site specified by "coords". Implementation The inverse of write. Here it is necessary to read the private XML to detect the volume format and site order. int QIO_peek_metadata(QIO_Reader *fd, XML_MetaData *recXML); int QIO_skip_record(QIO_Reader *infile, int n); int QIO_seek_record(QIO_Reader *infile, int n); (see the corresponding QDP calls) ---------------------------------------------------------------------- Function calls into the Data Wrangling Layer This layer needs much more thought. A toy model testing the concept is needed. DWL = Data Wrangling Layer. "Constructors" DWL_get_stream *DWL_init_get(QLA_factory *f, int sp, int vf) DWL_put_stream *DWL_init_put(QLA_factory *f, int sp, int vf) DWL_get_stream *DWL_init_get_scalar(QLA_factory *f) DWL_put_stream *DWL_init_put_scalar(QLA_factory *f) DWL_get_stream *DWL_vinit_get(QLA_factory *f, int sp, int vf, int n) DWL_put_stream *DWL_vinit_put(QLA_factory *f, int sp, int vf, int n) DWL_get_stream *DWL_vinit_get_scalar(QLA_factory *f, int sp, int vf, int n) DWL_put_stream *DWL_vinit_put_scalar(QLA_factory *f, int sp, int vf, int n) sp for serial or parallel file I/O vf for multidump or single file I/O scalars are always written serial, single file f (see QIO_read, QIO_write above) API Initialization call. vinit is for arrays of length n. int DWL_number_of_sites(DWL_stream *ds) API Returns the max number of sites any node has to write/read to/from Implementation Calls the QDP_layout utility functions to get layout information. Uses QDP globals to get the lattice dimensions. void DWL_node_site_list(DWL_stream *ds, site *list) API Returns the list of site coordinates to be written by this node. The site coordinates are linearized according to a convention. "site" is an "int". This might be useful for consistency checking with the multidump format. Implementation Same calls as above. void DWL_full_site_list(DWL_stream *ds, site *list) API Returns the list of site coordinates in the order written by all nodes. This information is inserted into the private record XML for nonlexicographic site ordering. int DWL_get_next(DWL_stream *ds, char *buf) int DWL_put_next(DWL_stream *ds, char *buf) API The "QDPstream" functions. Transfers data between the buffer and the field. "Next" refers to the order in which data is arranged on the current node. Return indicates status - e.g. end of data. Implementation We have in mind that the nodes call get/put in more-or-less lockstep whether or not a node has any data to move between the LRL and the DWL. Communication via QDP may be necessary to produce the next field. On a grid-based machine with a fixed layout, this operation may be implemented through a series of shifts. int DWL_close_get(DWL_get_stream *ds, DWL_Checksum *checksum) int DWL_close_put(DWL_put_stream *ds, DWL_Checksum *checksum) API Generate the global checksum for the data transferred. Cleans up. Checks that no data was left unprocessed. Status returned as int. Implementation QMP global XOR call may be needed to generate the checksum. Function calls into the Logical Record Layer LRL = Logical Record Layer. Choose your own favorite namespace! Because of the logical organization of the file, we have two types of open calls -- one for the whole file and one for a single record. LRL_fReader * LRL_open_read_file(char *filename) LRL_fWriter * LRL_open_write_file(char *filename, int mode) API where mode = LRL_CREATE, LRL_APPEND, LRL_TRUNCATE LRL opens the file, reads the DIME header, and stops before the first DIME record (the file XML record) LRL_rReader * LRL_open_read_record(LRL_fReader *fd, DIME_tag tag); LRL_rWriter * LRL_open_write_record(LRL_fWriter *fd, size_t size, DIME_tag tag); size Number of bytes in this record (for DIME) tag We might want to use this tag for consistency checking. API LRL prepares to read/write a record. Returns a "record descriptor". Implementation This descriptor can point to the parent fd. int LRL_seek_record(LRL_fReader *fd, int nrec, int seek_set); API LRL seeks to record number "nrec". Offset is relative or absolute, depending on seek_set. int LRL_seek_read_bytes(LRL_rReader *rd, size_t nseek, int seek_set); int LRL_seek_write_bytes(LRL_rWriter *rd, size_t nseek, int seek_set); API LRL seeks to byte number "nseek". Offset is relative or absolute, depending on seek_set. size_t LRL_read_bytes(LRL_rReader *rd, char *buf, size_t size); API LRL reads the next size bytes into the buffer from the specified record. Returns the number of bytes read. LRL does not read past the end of the record. Multinode reads are simply passed through to the OS, so they are supported if the OS supports it. size_t LRL_write_bytes(LRL_rWriter *rd, char *buf, size_t size); API See read. int LRL_close_read_record(LRL_rd *rReader); int LRL_close_write_record(LRL_rd *rWriter); API Close record int LRL_close_read_file(LRL_fd *fReader); int LRL_close_write_file(LRL_fd *fWriter); API Close file ----------------------------------------------------------------------