All, Today's call. Dates:Thursday-March 3, 2005 Time:1:00PM- 3:00PM Number of Participants:(12) Number to Call: 617-324-7520 Agenda ======= March Madness ====== 1.) QCDOC I/O (Carleton) -- SEE attached note on UK plans from Carleton, Balint and Peter 2.) Disk Space needs for production on US QCDOC. --Status of storage arrangements at BNL and grid transfer to JLab & FNAL 3.) Progress on Asqtad Level 3 tests/interface to MILC? (Dru, Chulwoo, Carleton and James) 4.) Chroma on QCDOC (postpone to next most likely) 5.) et al Cheers, Rich -- Prof. Richard C. Brower Physics Department --Boston University 590 Commonwealth Ave. Boston, MA 02215 Tele: 617 353 6052 (BU) 617 253 2599 (MIT) 617 497 8296 (HOME) Fax: 617 353 9393 Email: brower@bu.edu brower@lns.mit.edu ========================================================================== Hi Don and Jim and Bob, I have been having some useful exchanges with Balint Joo and Peter Boyle about QCDOC I/O. I wanted you to see them in case you (Don and Jim) have any comments about their I/O system. We could discuss the issues tomorrow. Bob, especially, I want to know if the BNL machine will be configured in the same way with the same standard operating procedures. The exchange begins with an I/O benchmark report by Balint. I followed up with a QIO analysis and questions about how UK is going to do large files. For brevity I left out some of the intermediate exchanges and deleted discussion of other topics. The upshot is that the UK folks plan to stage their large working files on the /host file system and *not* /pfs as we had been told we would need to do on the BNL machine. And they want to do fully parallel reads and writes with a single file to/from several thousand nodes with some throttling of concurrency if need be. Peter thinks he can make it work. Regards, Carleton - ---------------------------------------------------------------------- Message 1 - ---------------------------------------------------------------------- Date: Tue, 01 Mar 2005 15:58:50 +0000 From: Balint Joo X-Accept-Language: en-us, en To: Robert Edwards , Peter Boyle , Carleton DeTar Subject: QIO Rack tests X-Edinburgh-Scanned: at lawnmarket.ucs.ed.ac.uk with MIMEDefang 2.33, Sophie 3.04rc1, Sophos Anti-Virus 3.90 X-Scanned-By: MIMEDefang 2.43 X-Scanned-By: MIMEDefang 2.33 (www . roaringpenguin . com / mimedefang) This is a multi-part message in MIME format. --------------090202030207050602030802 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Dear All, Attached are the benchmarks from QIO run on a 1024 node rack. Best, Balint -- ------------------------------------------------------------------- Dr Balint Joo Post Doctoral Research Fellow School of Physics University of Edinburgh Mayfield Road, Edinburgh EH9 3JZ Scotland UK Tel: 0131 650 6469 (from UK) +44-131-650-6469 (from outwith UK) Fax: 0131 650 5902 (from UK) +44-131-650-5902 (from outwith UK) email: bj@ph.ed.ac.uk bj@phys.columbia.edu WWW : http://www.ph.ed.ac.uk/~bj ------------------------------------------------------------------- --------------090202030207050602030802 Content-Type: text/plain; name="RACK_IO_TESTS" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="RACK_IO_TESTS" QIO: I/O tests run on Rack 42 (1024) Nodes Machine wired as 4x4x8x2x2x2 torus 16x16x16x32 tests run with Machine Topology 4 4 8 8 => Local Vol 4 4 2 4 Set with: qpartition_remap -X0 -Y1 -Z2 -T345 24x24x24x32 tests run with Machine Topology 4 4 4 16 => Local Vol: 6 6 6 2 24x24x24x64 tests run with Machine Tolology 4 4 4 16 => Local Vol: 6 6 6 4 Set with: qpartition_remap -X0 -Y1 -Z34 -T25 16x32x32x32 tests run with Machine Topology 4 4 8 8 => Local Vol: 4 8 4 4 Set with: qpartition_remap -X0 -Y1 -Z2 -T345 Timings: ======= --------------------------------------------------------------------- Volume Size Singlefile Singlefile Multifile Multifile (Global) (Mb) Write Time Read Time Write Time Read Time --------------------------------------------------------------------- 16x16x16x32 72Mb 100.178s 124.378s 10.6086s 3.07285s 24x24x24x32 243Mb 370.267s 458.616s 12.8844s 34.9078s 24x24x24x64 486Mb 742.873s 918.138s 32.1632s 87.6804s Transfer Rates (in Mbytes/s) ============================ ----------------------------------------------------------- Volume Singlefile Singlefile Multifile Multifile (Global) Write Rate Read Rate Write Rate Read Rate ----------------------------------------------------------- 16x16x16x32 0.7187 0.5789 6.7869 23.4311 24x24x24x32 0.6563 0.5299 18.8601 6.9612 24x24x24x64 0.6542 0.529 15.1104 5.5429 Notes: i) The Singlefile times are rather bad: 100s = 1.7 min 124s = 2.1 min 370s = 6.2 min 458s = 7.6 min 724s = 12 min 918s = 15.3 min The multifile times are acceptable... ii) There are large fluctuations in the Multifile read and write times. This may (or may not) be to do with the performance of the NFS server being written to/read from. --------------090202030207050602030802-- - ---------------------------------------------------------------------- Message 2 - ---------------------------------------------------------------------- X-Authentication-Warning: snort.ph.ed.ac.uk: paboyle owned process doing -bs Date: Wed, 2 Mar 2005 14:33:03 +0000 (GMT) From: Peter Boyle To: Carleton DeTar cc: bj@ph.ed.ac.uk Subject: Re: QIO Rack tests In-Reply-To: <200503020027.RAA00934@dirac.physics.utah.edu> X-Scanned-By: MIMEDefang 2.43 Hi Carleton, [irrelevancies deleted] [ quoted from my reply... ] > My expectation is that with really big projects running on several K > nodes, we will have to use PARTFILE. SINGLEFILE (serial or parallel) > will be impractical. For small projects (few hundred nodes) any > format is probably OK. Our experience has been that SINGLEFILE if accessed in parallel can be reasonable on even thousands of nodes, depending on whether you call 25MB/s for I estimate 20s-1minute delays of compute reasonable. If you need more, writing to /pfs will give it, but distribute the directories on a per node basis. We're not losing much time to the optimised SINGLEFILE method of writing here. [irrelevancies deleted] Thanks, Peter > Best regards, > Carleton > - ---------------------------------------------------------------------- Message 3 - ---------------------------------------------------------------------- Date: Wed, 2 Mar 2005 08:51:20 -0700 (MST) From: Carleton DeTar To: paboyle@ph.ed.ac.uk CC: bj@ph.ed.ac.uk, detar@physics.utah.edu In-reply-to: (message from Peter Boyle on Wed, 2 Mar 2005 14:33:03 +0000 (GMT)) Subject: Re: QIO Rack tests X-Scanned-By: MIMEDefang 2.43 Hi Peter, > Our experience has been that SINGLEFILE if accessed in parallel can > be reasonable on even thousands of nodes, depending on whether you > call 25MB/s for I estimate 20s-1minute delays of compute reasonable. > If you need more, writing to /pfs will give it, but distribute the > directories on a per node basis. We're not losing much time to the > optimised SINGLEFILE method of writing here. I am not sure I understand your satisfaction with SINGLEFILE. I do not consider more than 5 minutes an acceptable read/write time for a smallish lattice (24^3 x 64). What are our I/O requirements? Plans in the first year are to run lattices as big as 48^3 x 96 = 3 GB for the gauge configuration and 6 GB for the corresponding Wilson propagator files. If we do heavy-light form factor calculations, a single lattice may have several such propagator files. Future lattices are sure to be larger. So ideally on several thousand nodes we would have a file system with an aggregate data transfer rate of the order 100 MB/s to transfer such files within minutes. How do these requirements fit Balint's benchmarks? The local volume for 48^3 x 96 on 8K processors would be 6^4, which corresponds approximately to Balint's 24^3 x 64 on 1K nodes. Balint's MULTIFILE numbers of the order 10 MB/s to 1000 nodes scales to about 80 MB/s on 8K nodes, close to our goal. Not bad for starters. But Balint's numbers for SINGLEFILE are under 1 MB/s, more than two orders of magnitude too small. I don't know how much better a clever routing algorithm could do, but even if routing were not a bottleneck, could you get 100 MB/s through a single NFS connection? So unless you are running small lattices, we need PARTFILE. Regards, Carleton - ---------------------------------------------------------------------- Message 4 - ---------------------------------------------------------------------- X-Authentication-Warning: snort.ph.ed.ac.uk: paboyle owned process doing -bs Date: Wed, 2 Mar 2005 16:03:41 +0000 (GMT) From: Peter Boyle To: Balint Joo cc: Carleton DeTar Subject: Re: QIO Rack tests In-Reply-To: <4225E2A7.8010404@ph.ed.ac.uk> X-Scanned-By: MIMEDefang 2.43 Thanks Balint, > > I am not sure I understand your satisfaction with SINGLEFILE. I do not > > consider more than 5 minutes an acceptable read/write time for a > > smallish lattice (24^3 x 64). > > Here Pete was mentioning SINGLEFILE accessed in parallel. Ie > more than one I/O node writes, each to its offets. So this is the old > parallel IO to single file mode again, not the QIO SINGLEFILE. We should > give this mode a name. It is essentially PARTFILE/MULTIFILE with no > host conversion needed. As you can see we all get confused with the > various labels. Yes, I did indeed mean the SINGLEFILE whose implementation would be parallel access. The call and end result would be the same but the parallel access would result in a much faster (20x) implementation. I would only be satisfied with this performance. Thanks, Peter > All the best, > Balint > > > - ---------------------------------------------------------------------- Message 5 - ---------------------------------------------------------------------- Date: Wed, 2 Mar 2005 12:14:49 -0700 (MST) From: Carleton DeTar To: paboyle@ph.ed.ac.uk CC: bj@ph.ed.ac.uk, detar@physics.utah.edu In-reply-to: (message from Peter Boyle on Wed, 2 Mar 2005 16:03:41 +0000 (GMT)) Subject: Re: QIO Rack tests X-Scanned-By: MIMEDefang 2.43 Hi Peter, Thanks for the clarification. So the idea is to have 8K to 10K nodes writing to and reading from a single file. You would need to provide NFS mounts from all nodes to all RAID boxes. Where does this file reside? Are you striping across multiple RAID boxes without regard to the distribution of lattice sites? Or do you expect a single RAID box to manage all 8K - 10K NFS mounts? FNAL has questioned whether that is possible because when you exceed the throughput capacity of the RAID system, it can be overwhelmed, resulting in a huge performance hit. I would want a demonstration that it works. As for terminology, the specifications we had originally planned for QIO are volume format = {SINGLEFILE, MULTIFILE, PARTFILE} and I/O mode = {SERIAL, PARALLEL}. The current placeholder for the I/O mode parameter is the last QIO_ioflag argument to QIO_open_write and QIO_open_read. So I would call the alternatives SINGLEFILE (parallel mode) and SINGLEFILE (serial mode). (But of course there is no QIO implementation of the parallel mode at present. To make it work requires some coordination with LIME, which is built on a serial streaming model.) Regards, Carleton - ---------------------------------------------------------------------- Message 6 - ---------------------------------------------------------------------- X-Authentication-Warning: snort.ph.ed.ac.uk: paboyle owned process doing -bs Date: Wed, 2 Mar 2005 20:41:44 +0000 (GMT) From: Peter Boyle To: Carleton DeTar cc: bj@ph.ed.ac.uk Subject: Re: QIO Rack tests In-Reply-To: <200503021914.MAA01420@dirac.physics.utah.edu> X-Scanned-By: MIMEDefang 2.43 Hi Carleton, 1. So far we haven't been using more than 1k nodes in anger for concurrent writing but have demonstrated good bandwidth with 1k nodes concurrently writing to a filesystem that exports and supports 4k mounts due to being shared amongst 4 racks. This has been entirely successful, in part through tuning Linux kernel parameters and in part because I wrote the NFS client and have complete control of all timeouts and retransmit rates. How careful has FNAL's tuning been? I'd guess (but do not know) 4k simultaneous mounts and 1k simultaneous writing nodes exceeds Fermilab's experience, but agree that the 8k extrapolation is indeed an extrapolation. The single file will reside on a disk on the IBM Pseries, which *should* be beefier than Linux, but your demonstration will have to wait until have 8k nodes cabled together and working. We have been unable to do so because of problems with our Fibre Channel SAN. 2. in practice getting 8k nodes for a single job in my estimation is unlikely. It is after all a shared facility. 3. we need to distinguish carefully the globally shared /host file system, which is a directly accessed disk logical volume group (or in our case FibreChannel SAN) on the IBM frontend, and the linux based Raid servers which will ultimately provide /pfs. Writing a single file will necessarily be to this /host file system, since that is the only way to obtain a single file. Thus striping across RAID boxes is not possible in this mode. When writing to the PFS, this is necessarily multi-file mode. In this mode I will stripe in a way that two consecutive motherboards are on the same RAID, but each rack mounts 8 RAIDS, with every group of four RACKS sharing the same 8 RAIDS. 4. Columbia have implemented the option of continuously throttling down the number of _concurrent_ writers to be anything between ALL and ONE. As I said, this is an implementation issue. Note that ALL mount the disk simultaneously, and indeed ALL write. The question is when, and again compared with the naive single file, this is an implementation issue and performance issue. With this continuously variable throttling, I could virtually guarantee you no issues with 8k compared with the 1k we have demonstrated, since an NFS server is by design stateless and maintains no growing "state" whether it has dolled out 8k mounts or 1k mounts. The load is controlled only by the number of writers. Just because one could implement the parallel write to single file in a spectacularly agressive way doesn't mean that one has to. Adding a placeholder "hint" to QIO for concurrency in the SINGLEFILE case would be wise. Subsequently the two extreme cases use a value of either 1 or ALL, however strictly the intermediate cases can either use CONCURRENCY I/O nodes and pass through the comms network, or can have all nodes do their own I/O but schedule the access. Again an implementation issue. I would finish by reemphasizing that our existing setup actually runs with 4 racks on one LinuxRAID disk, i.e. we typically have 4k simultaneous mounts, but that since the running jobs rarely write out simultaneously the demonstration is only really valid at the 1k level. Best wishes, Peter - ----------------------------------------------------------------------