Software Coordinating Committee Conference Call January 2, 2003 3:00 PM EST Recorder: C. DeTar Present: DeTar, Osborn, Holmgren, Brower, Mawhinney, Simone, Edwards, Chulwoo Jung, Hetrick, Watson, Richards Absent: Pochinsky, Gregory, Mendes, Neff, Petrov ** Action items ===================================================================== 1.) Report on by Jim Simone and Jim Hetrick on Edinburgh Data-Grid workshop. What are they doing that we can use now and/or in the future. 2.) Status of Jan 1, 2003 benchmarks & Milestones: (SEE: http://physics.bu.edu/~brower/tests ) Gottlieb doing MILC benchmarks at FNAL Edwards to have numbers soon Mawhinney: See Postscript file, just sent. Results for MILC naive staggered C on 2^4 Soon to have results for 4^4 and also Naik. Asqtad force term. Communications channel is closed and reopened for each parallel transport, so bad for performance - may need coarser granularity to hide this latency -- but can't test 4^4 well on simulator. Could minor changes in the MILC code improve performance? Jung: Could there something like a dslash_special for the parallel transport in Asqtad? ** DeTar: Will look at it and get back to Columbia. 3.) Status of QDP with C binding (James Osborn) and C++ binding (Robert Edwards). QDP++ Edwards: First version of specification 0.4 was out on Dec 9. Modified version ready to go out. Working serial version is on my web site. QDP/C DeTar: QLA 1.1 is nearly finished and the first QDP version will also be ready soon. 4.) Plans to complete the definition of the QDP I/O Syntax and the Data Ffle formats/headers by end of February. It would be good to have this interface in reasonable published form by the all hands meeting since this is clearly an area where feedback from the larger community of users would be useful. ILDG Middleware working group: Watson and Hetrick ILDG Metadata working group: Simone (our rep) and Edwards Discussion of file formats and metadata Simone: See BinX document DeTar: Sent e-mail "Metadata Philosophy" earlier this week. Watson: SciDAC should decide on a format and use BinX to describe it, rather than generating a multitude of documents. DeTar: What if we go ahead with our own format and plan to have the archiving process capable of delivering a binary file in BinX format if the user wants it? Edwards: UKQCD wants to write large metadata schema automatically from the application code. Edwards: SAX and DOM do data binding and there are tools for generating Java and C++ from XML. C++ version is rough. DeTar: If people agree with the proposal to have a minimal subset of metadata in each file and the rest in a dataset description file, then we need to define that minimal set. Simone: See minimal file encoding in my proposal. DeTar: We need a way to read the XML data whether encoded separately in a file or in the dataset description file. Edwards: We should be careful that the dataset description file is correlated with the component files. We could construct a hash based on the minimal parameter set and include the same hash in the dataset description file. Watson: We could simply repeat all the file metadata for each file in the dataset description file. Edwards: We also need to have a way to associate a derived file, say a propagator, with its parent lattice. DeTar: Perhaps this could be done by putting the parent lattice file name in the propagator file metadata. Or at least a sequence number. Edwards: It would be good to have a private part of the archive for each user while the files were being created and analyzed. DeTar: Then when the files are ready to be published the rest of the metadata would be filled in for the catalog. Discussion of QDP I/O functionality Edwards: The C++ version would use the stream metaphor for file I/O. Richards: UKQCD finds it useful to read a single time slice. Edwards: We have a subset concept already that does that. DeTar: The application (QDP) also needs a facility for generating the minimal metadata set required for each file that is to be written. Edwards: This part of the process goes above QDP in our coding hierarchy. Watson: We need a QMD (?) for handling metadata - separate from QDP. It could deal with the larger set of metadata. Then the I/O interface would call into QMD and QDP. DeTar: We have talked about multirecord files, which would be useful for, say, writing multiple eigenvectors for a lattice. Do we want the flexibility to write identifying information for separate records in a file? Simone: That isn't in the current strawman proposal. Edwards: In QDP++ the current plan is that the quantum would be the whole file. The binary data in a file is a single data object. DeTar: But shouldn't we arrange so we can read a single record from a multirecord file? So then we might want identifying information for that record other than just a sequence number. Richards: Agree this is useful - e.g. QCDOC generates a really large file and we want to read portions on a smaller machine. Edwards: That could be considered a "slice" in STL language. Brower: Could we arrange so the file would be read, but only part of the data stored? ** Brower: Jim Simone and Chip Watson to put together a proposal for minimal metadata. Watson: How does UKQCD plan to store physics metadata under ** BinX? Can we find out? Brower: Will be away Jan 9, but devote audiobridge time to I/O discussion. ---------------------------------------------------------------------- Conference concluded at 5:00 PM EST. Next plenary call Jan 9 at 3 PM EST.