From detar@nova.physics.utah.edu Thu May 22 17:49:08 2003 Date: Thu, 22 May 2003 15:49:07 -0600 (MDT) From: Carleton DeTar To: petrov@physics.columbia.edu CC: brower@lns.mit.edu, edwards@jlab.org, simone@fnal.gov, detar@physics.utah.edu In-reply-to: <19816033384.20030522160437@physics.columbia.edu> (message from Konstantin Petrov on Thu, 22 May 2003 16:04:37 -0400) Subject: Re: QIO and QCDOC Hi Kostya, The details of the C wrapper are not completely ironed out, so it is the functionality that is most important. We want to be sure that if a user writes what he/she thinks is a single file, it appears at the other end of the pipeline as a single file without requiring further intervention at the level of the user code. So if the local implementation first writes it in mutliple parts, the OS would have to take care of reassembling it on whatever file system is available. Perhaps we could agree to a job script command to the OS for doing this, but certainly not a modification of the portable code. Similarly, if a user writes a multidump file, the resulting file would be in multiple parts, as requested. We did not specify how many parts. The intention is to make multidump writing and reading be as fast as possible, so it functions like memory swapping. But I would hope these files still encode the sitelist, so, if need be, we could reread multidump volumes to any new layout with suitable data redistribution across the communications network. This capability has not been agreed upon, yet. We do intend, of course, for single archival files to be written and read from a parallel file system if it is available. The multidump files are more likely destined for local scratch disks, which I would not regard as a parallel file system. For the clusters we had in mind that it made sense to write files either in serial mode or parallel mode. I didn't get the impression we had completely settled on making this an option. Here the distinction is not whether or not the target filesystem is parallel, but whether all processors participate in writing or one processor does the writing. We had in mind that if a user requested parallel writing, but the file system did not permit it, the mode would automatically revert to serial writing. If the QCDOC can only do parallel writes, one could imagine a similar policy of always promoting serial requests to parallel. Best regards, Carleton Date: Thu, 22 May 2003 16:04:37 -0400 From: Konstantin Petrov Reply-To: Konstantin Petrov Organization: CU X-Priority: 3 (Normal) X-MailScanner: Found to be clean Hi Rich&Carlton, I went through the QIO specification/implementation and we do not seem to have strong objections so far. I am slightly puzzled by the mixed use of struct and raw variables for passing lattice parameters, but this is of no major concern for the following reason - all actual underlying QCDOC software is going to be C++, so all C API's have to be wrapped around it. So the relevant _implementation_ is what, as I understand, Robert is writing - and, again - which is work in progress so that no final/semi-final specification/implementation has been released. However I do not expect major complications in adopting it. One thing of course is that we are going to use _parallel_ filesystem for _archival_ configurations output, not only as scratch space (as you do, right?), and OS will be responsible for linking/archiving it. We will thus likely port QIO/C++ as leading order approximation and extend _implementation_ to reflect _transparently_ the above mentioned parallelism. Thus, for Qio/C to function properly somebody need to ensure that specification of QIO/C++ allows QIO/C to be wrapped around it. then we are all-set. Kostya ps. Sorry for not getting back to you earlier, but we were trying to produce a first working version of the OS - and did it, though there are many things to figure out still. From petrov@physics.columbia.edu Thu May 22 21:50:56 2003 MIME-Version: 1.0 Date: Thu, 22 May 2003 21:49:18 -0400 From: Konstantin Petrov Reply-To: Kobstantin Petrov Organization: CU X-Priority: 3 (Normal) To: Carleton DeTar CC: brower@lns.mit.edu, edwards@jlab.org, simone@fnal.gov Subject: Re[2]: QIO and QCDOC In-Reply-To: <200305222149.h4MLn7s04962@dirac.physics.utah.edu> Hi Carleton, CD> If the QCDOC can only do CD> parallel writes, one could imagine a similar policy of always CD> promoting serial requests to parallel. QCDOC will be able to do both. However due to bandwidth being significantly different we will have to develop certain policy for serial/parallel writes, and I am not sure if the user will be able to control what is actually happening - and of course user will not have to modify portable code. I am not sure how we are going to do this - we didn't write that piece of OS yes, but we will start to work on authorization and other aspects of run-time interaction next week. Kostya ps. If you have strong opinion on whether user should be allowed to control usage of parallell/serial writes - please comment on this, I will bring it up for discussion at developer meeting then. From detar@nova.physics.utah.edu Fri May 23 10:16:56 2003 Date: Fri, 23 May 2003 08:16:54 -0600 (MDT) From: Carleton DeTar To: petrov@physics.columbia.edu CC: brower@lns.mit.edu, edwards@jlab.org, simone@fnal.gov In-reply-to: <9445175518.20030522214918@physics.columbia.edu> (message from Konstantin Petrov on Thu, 22 May 2003 21:49:18 -0400) Subject: Re: QIO and QCDOC Hi Kostya, The main reason I was advocating user control over serial/parallel I/O modes is that on some systems it may not be possible for the implementation software to detect whether a file system is capable of parallel operation. We will presumably find ways to get that system information from our clusters, but we want our code to be portable to proprietary machines as well, where the necessary system calls may not be available. For example the IBM GPFS is a parallel file system, but IBM SP code must be built on MPI-2 with MPI-I/O calls to access it. So if you ask whether the hardware can handle parallel operations, the answer is yes, but then you have to check for MPI-2 compilation. It seems much simpler to have the user choose. In MILC code we select the I/O mode in our startup parameter file. Since the QCDOC implementation is special to the QCDOC, except possibly for debugging purposes, I see no problem choosing an optimum standard (presumably parallel) and simply ignoring user requests for nonoptimum modes for writing lattice field data. Does anyone else see a problem with this? Regards, Carleton