Software Coordinating Committee Conference Call I/O Subcomittee February 27, 2003 3:00 PM EST Recorder: C. DeTar Present: DeTar, Brower, Hetrick, Osborn, Watson, Holmgren, Edwards, Simone, Neff, Pochinsky Absent: Mendes, Richards, Petrov, Mawhinney, Jung, Gregory ** Action items ===================================================================== 1.) LQCD IO We need to take our general outline from the workshop on IO/file_formats/ data_archiving and turn them into a plan. All of the presentations are posted at the usual place as minutes: http://physics.bu.edu/~brower/SciDAC/scc.html In particular see Chip's "Agreements on LQCD I/O" doc 2.) QIO lite I want first get the basic I/O instructions defined and implemented in a C and C++ along the lines in Carleton's note (appended below and on my webpage of course). Simone: For C++ Balint has sent around his idea of a DIME wrapper. Uses streams. I have tried to do the same in C. Just need a method for C++ that returns the next n bytes. All C fields should have a corresponding stream "factory" function. This would provide a connection between the DIME encoder and C. DeTar: Would the stream approach allow seeking to a given record in a file? Edwards: If this is an important use case, we could support it. I have never found the need for it. DeTar: The MILC code has this capability and we have used it on occasion. Osborn: C could have a stream abstraction. Edwards: [Discusses the IO Interface diagram] Watson: How do streams handle a two-headed object? Edwards: QDP serializes itself as a single DIME object containing both XML and binary Watson: Not a good idea to have the QDP stream know about DIME. Simone: Agree. Watson: Need to investigate further. [Edwards and Watson after some discussion, came to some agreement.] Brower: Where do we go from here? DeTar: Can we agree on "QIO Lite" or "QFS lite" in Robert's language as a statement of the minimal user functionality? What are the controversial points? Watson: Our list of agreed points dealt with binary file structure and not interface. Edwards: Agree with all of these, but would schedule them so serial reads and writes would come in the first edition and parallel reads and writes come later. [Some discussion about whether it is a good idea to wrap C on C++] Pochinski: Even if we use EXTERN C and present the user with a binary library, the C++ runtime library has to be there, so there must be a C++ compiler. Edwards: With the stream model g++ > 3.0 is required. The alpha compilers can't compile QDP++ and won't compile the I/O layer we are proposing, for example. Osborn: Would it be possible to write C++ as a wrapper on C engines? Simone: The XML part is more naturally implemented in C++. Edwards: We need the MetadataReader class in any language. Simone: I am working on a toy C++ version. Brower: I would like to have something ready before the next ILDG meeting. Edwards: Balint is ready to go. There is already a stream feature in QDP++ for binary. Just need the XML feature to finish it. DeTar: Returning to C vs C++ XML parsing, what about expat? Simone: All the XML parsers use C. I use Xerxes. What if we looked at expat and xerxes and developed an abstract interface that both C and C++ would access? Edwards: I was trying to avoid that. Osborn: What about sax2 parsers? Simone: It may not be completely standard. DeTar: We shouldn't be trying to support the most general XML. Let's be more modest. Edwards: What would it take to get a minimal parser? Simone: Not hard in C++. Brower: When can we have it? ** Simone: Two weeks. But my toy isn't well designed at this point. Brower: Let's get something going. Then perhaps James could convert it to C if it isn't so difficult. We can decide then where we should go from there. Brower: Subcommittee should discuss among themselves if need be before next week. 3.) Do we need a separate "subcommittee" to work on the common features of our LQCD runtime environment? (Don H, Bob M and Chip W?) ---------------------------------------------------------------------- Opaque types: QIO_MetaData *fileXML, *recXML; (single set of metadata) QIO_FILE *infile, *outfile; (plays the role of a file pointer) - ---------------------------------------------------------------------- Metadata manipulation 1. Creating, destroying QIO_MetaData *QIO_create(void); void QIO_destroy(QIO_MetaData *); 2. Creating a new subpath path is a string as in Jim Simone's "/gauge/flavor0/mass" void QIO_MD_addpath(QIO_MetaData *, char *path, char *newname) 3. Setting a value (path includes the name of the data item) void QIO_MD_insert_real(QIO_MetaData *, char *path, double value); void QIO_MD_insert_int(QIO_MetaData *, char *path, int value); void QIO_MD_insert_string(QIO_MetaData *, char *path, char *value); (Do we need complex?) 4. Extracting values double QIO_MD_extract_real(QIO_MetaData *, char *path); (sets value) int QIO_MD_extract_int(QIO_MetaData *, char *path); (sets value) char * QIO_MD_extract_string(QIO_MetaData *, char *path); (sets pointer) 5. Inspecting: counting (ls -1 path | wc -l) Count entries in path int QIO_MD_entries(QIO_MetaData *, char *path); 6. Inspecting: listing (ls -l path) char *names[dim]; char *types[dim]; char *values[dim]; int QIO_MD_list(QIO_MetaData *, char *path, char *names[], char* types[], char* values[], int dim); Returns the list of data and corresponding object names under the specified path. The types encode real, int, string, and "object" for subpaths. (We could also encode the name and type pair in a structure.) 7. Deleting (rm -r path) int QIO_MD_delete(QIO_MetaData *, char *path); Returns status 8. Copying (cp -r srcpath destpath) int QIO_MD_copy_metadata(QIO_MetaData *dest, char *destpath, QIO_MetaData *src, char *srcpath); Returns status (Do we allow clobbers?) 9. Reading and writing pure XML files int QIO_MD_read_metadata(QIO_MetaData *, char *string); int QIO_MD_write_metadata(QIO_MetaData *, char *string); Opens a pure metadata file specified by string and reads (writes) the metadata. (Could be used for the project metadata file or startup parameters.) "read" implies broadcast and "write" implies only the master node. ---------------------------------------------------------------------- Binary file I/O 1. Open a file for writing or reading QIO_FILE *QIO_open(QIO_MetaData *fileXML,char *string, char *mode) where string identifies the file (URL?? filename??) and mode is "r" or "w" or "a". If reading, fileXML is set to contain the file metadata If writing, fileXML is metadata to be written at the head of the file Somehow the I/O system has to ensure that the datatype T is encoded automatically in the metadata. Perhaps we need a separate open for data distributed across multiple files: QIO_FILE *QIO_open_dump_T(QIO_MetaData *fileXML,*string) This format is unsuitable for archiving, so standardization is not important. Filenames would encode the node number. 2. Close a file QIO_close(QIO_FILE *file); (No need to specify datatype.) 3. Write record - various modes - single QDP field Returns status integer. Only metadata from the master node is written. a. Master node writes all int QIO_write_serial_T(QIO_FILE *outfile, QIO_MetaData *recXML, QDP_Type *field); b. All nodes write - lexicographic order int QIO_write_parallel_T(QIO_FILE *outfile, QIO_MetaData *recXML, QDP_Type *field); c. All nodes write - native order int QIO_write_checkpoint_T(QIO_FILE *outfile, QIO_MetaData *recXML, QDP_Type *field); d. Multiple file int QIO_write_multidump_T(QIO_FILE *outfile, QIO_MetaData *recXML, QDP_Type *field); where The QDP_Type matches T. The recXML is the record metadata to be written at the head of the record. 4. Write record various modes - array of QDP fields In this case the data is ordered as though it is an array on each site with the array index varying more rapidly - the site index varying slowest. This would be commonly used for a gauge field where the array index would be the spatial tensor index for the field. Returns status integer. Only metadata from the master node is written. a. Master node writes all int QIO_vwrite_serial_T(QIO_FILE *outfile, QIO_MetaData *recXML, QDP_Type *field, int n); b. All nodes write - lexicographic order int QIO_vwrite_parallel_T(QIO_FILE *outfile, QIO_MetaData *recXML, QDP_Type *field, int n); c. All nodes write - native order int QIO_vwrite_checkpoint_T(QIO_FILE *outfile, QIO_MetaData *recXML, QDP_Type *field, int n); d. Multiple file int QIO_vwrite_multidump_T(QIO_FILE *outfile, QIO_MetaData *recXML, QDP_Type *field, int n); 5. Read various modes - single QDP field Returns status integer. Broadcasts metadata. a. Master node reads all int QIO_read_serial_T(QIO_FILE *infile, QIO_MetaData *recXML, QDP_Type *field); b. All nodes read - any order int QIO_read_parallel_T(QIO_FILE *infile, QIO_MetaData *recXML, QDP_Type *field); c. Multiple file int QIO_read_multidump_T(QIO_FILE *infile, QIO_MetaData *recXML, QDP_Type *field); (If recXML = NULL, ignore the metadata.) 6. Read various modes - array of QDP fields Returns status integer. Broadcasts metadata. a. Master node reads all int QIO_vread_serial_T(QIO_FILE *infile, QIO_MetaData *recXML, QDP_Type *field, int n); b. All nodes read - any order (autodetect) int QIO_vread_parallel_T(QIO_FILE *infile, QIO_MetaData *recXML, QDP_Type *field, int n); c. Multiple file int QIO_vread_multidump_T(QIO_FILE *infile, QIO_MetaData *recXML, QDP_Type *field, int n); Reads n values. Truncates if n is smaller than the number in the record. If n is greater, fills the leading array components. 7. Read only the record XML for the current record int QIO_read_metadata(QIO_FILE *infile, QIO_MetaData *recXML); Returns status integer. Broadcasts the metadata. (File is repositioned at the beginning of the record.) 8. Skip to the next record int QIO_skip_record(QIO_FILE *infile); Returns status integer. 8. Seek to the nth record int QIO_seek_record(QIO_FILE *infile, int n); Returns status integer. ---------------------------------------------------------------------- ASCII global scalar I/O 1. stdin broadcast to all nodes int QIO_scan_int(int *); int QIO_scan_real(double *); int QIO_scan_string(char *); 2. Master writes its values to stdout or file int QIO_printf(char *,,,); int QIO_fprintf(FILE *,char *,,,); 3. Free-for-all Standard printf. - ---------------------------------------------------------------------- Legacy files Reading MILC format and NERSC archive formats -- we could have a compatibility mode that processed MILC ".info" data to get the file XML and nothing for the NERSC archive format, but responded sensibly to the rest. ---------------------------------------------------------------------- Conference concluded at 5:00 PM EST. Next week we meet at FNAL. Next plenary call Mar 6 at 3 PM EST.