Software Coordinating Committee Conference Call March 3, 2005 1:00 PM EST Recorder: C. DeTar Present: DeTar, Maresca, Brower, Holmgren, Simone, Eric Blum, Watson, Edwards Renner, Pochinsky, Osborn, Mawhinney, Zhang, Jung Absent: Petrov ** action items ===================================================================== 0.) Bug tracking Brower: Bob, can we get the bug tracking system started. ** Mawhinney: I will work with Stratos 1.) QCDOC I/O (Carleton) [ Reviewed the information from Balint and Peter attached below ] Question for Don: Any comments from FNAL? Holmgren: We had actual data corruption with many nodes. Any RAID system can have high performance with a couple of I/O streams. Multiple streams would require coordination. We throttle them so only 2 - 3 simultantous streams. Unless writes are sector-aligned, you have problems. Watson: We found TCP/NFS much more reliable than UDP/NFS. Mawhinney: 8-processor front end with its own disks. 500 GB on /host, which we are calling /qdata. Mawhinney: We have also been running of the /host file system, all doing reading and writing to and from a single file with 100's to 1000's of writers Question for Bob: DeTar: What should be the policy? Will users be free to use the /host file system, or will they be urged to split files and use the /pfs system? File splitting is a hassle. Mawhinney: It is to our advantage to use /pfs To get 10 - 25 MB/s to and from the host. To get that they had to turn off the acknowledgements. The biggest reason to stay with /pfs is that they are more scalable. Stratos is finding sustainable rates of 10 MB/s to and from 1000 nodes on a single RAID box. Stratos is using Gigabit links and getting about 8% of the bandwidth. Edwards: What is the rate to a single node? Mawhinney: I don't know. Probably low. UDP fragments are limited in size and the link is only 100 Mbit. Watson: The cheap RAIDs can deliver 30 MB/s. Mawhinney: There is no reason to think the cheap RAID boxes are vastly inferior to IBM. Edwards: I'd like to understand how much of the bottleneck is hardware and how much software. With SINGLEFILE mode, it would seem we are nowhere near the hardware limit. Mawhinney: So the consensus is that we will continue with our plan to use the /pfs in preference to /qdata. 2.) Disk Space needs for production on US QCDOC. --Status of storage arrangements at BNL and grid transfer to JLab & FNAL Brower: At some point it would be useful to discuss storage capacity. Mawhinney: 5 - 10 TB storage on the front end system for offline staging plus the /pfs RAID boxes. ** I'll send you the numbers. We are not prepared to do archival storage at BNL. Watson: I think there is too much allocated to /pfs and and not enough on staging. I would think we should separate the band-width issue from capacity. It is possible to get different numbers of controllers in a box. Mawhinney: There is a committee that is supposed to be advising us on storage. Watson: I am collecting storage needs from our users. I recommend this approach. Brower: Negele just sent me a note estimating LHPC needs 100 TB in a year. Watson: That certainly pushes our own requirements. 3.) Progress on Asqtad Level 3 tests/interface to MILC? (Dru, Chulwoo, Carleton and James) Jung: I had to debug my test code. I have now updated the library with only the double precision code. Dru, you should be able to figure out the interface from my test code. Renner: I'll fetch it and try it out. 4.) Chroma on QCDOC (postpone to next most likely) Brower: James is about to get the Bagelized version of the Chroma code for the alpha testers. ===================================================================== Committee conference concluded at 3:00 PM EST. Next plenary call March 10 at 1:00 PM EST ========================================================================== Hi Don and Jim and Bob, I have been having some useful exchanges with Balint Joo and Peter Boyle about QCDOC I/O. I wanted you to see them in case you (Don and Jim) have any comments about their I/O system. We could discuss the issues tomorrow. Bob, especially, I want to know if the BNL machine will be configured in the same way with the same standard operating procedures. The exchange begins with an I/O benchmark report by Balint. I followed up with a QIO analysis and questions about how UK is going to do large files. For brevity I left out some of the intermediate exchanges and deleted discussion of other topics. The upshot is that the UK folks plan to stage their large working files on the /host file system and *not* /pfs as we had been told we would need to do on the BNL machine. And they want to do fully parallel reads and writes with a single file to/from several thousand nodes with some throttling of concurrency if need be. Peter thinks he can make it work. Regards, Carleton - ---------------------------------------------------------------------- Message 1 - ---------------------------------------------------------------------- Date: Tue, 01 Mar 2005 15:58:50 +0000 From: Balint Joo X-Accept-Language: en-us, en To: Robert Edwards , Peter Boyle , Carleton DeTar Subject: QIO Rack tests X-Edinburgh-Scanned: at lawnmarket.ucs.ed.ac.uk with MIMEDefang 2.33, Sophie 3.04rc1, Sophos Anti-Virus 3.90 X-Scanned-By: MIMEDefang 2.43 X-Scanned-By: MIMEDefang 2.33 (www . roaringpenguin . com / mimedefang) This is a multi-part message in MIME format. --------------090202030207050602030802 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Dear All, Attached are the benchmarks from QIO run on a 1024 node rack. Best, Balint -- ------------------------------------------------------------------- Dr Balint Joo Post Doctoral Research Fellow School of Physics University of Edinburgh Mayfield Road, Edinburgh EH9 3JZ Scotland UK Tel: 0131 650 6469 (from UK) +44-131-650-6469 (from outwith UK) Fax: 0131 650 5902 (from UK) +44-131-650-5902 (from outwith UK) email: bj@ph.ed.ac.uk bj@phys.columbia.edu WWW : http://www.ph.ed.ac.uk/~bj ------------------------------------------------------------------- --------------090202030207050602030802 Content-Type: text/plain; name="RACK_IO_TESTS" Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="RACK_IO_TESTS" QIO: I/O tests run on Rack 42 (1024) Nodes Machine wired as 4x4x8x2x2x2 torus 16x16x16x32 tests run with Machine Topology 4 4 8 8 => Local Vol 4 4 2 4 Set with: qpartition_remap -X0 -Y1 -Z2 -T345 24x24x24x32 tests run with Machine Topology 4 4 4 16 => Local Vol: 6 6 6 2 24x24x24x64 tests run with Machine Tolology 4 4 4 16 => Local Vol: 6 6 6 4 Set with: qpartition_remap -X0 -Y1 -Z34 -T25 16x32x32x32 tests run with Machine Topology 4 4 8 8 => Local Vol: 4 8 4 4 Set with: qpartition_remap -X0 -Y1 -Z2 -T345 Timings: ======= --------------------------------------------------------------------- Volume Size Singlefile Singlefile Multifile Multifile (Global) (Mb) Write Time Read Time Write Time Read Time --------------------------------------------------------------------- 16x16x16x32 72Mb 100.178s 124.378s 10.6086s 3.07285s 24x24x24x32 243Mb 370.267s 458.616s 12.8844s 34.9078s 24x24x24x64 486Mb 742.873s 918.138s 32.1632s 87.6804s Transfer Rates (in Mbytes/s) ============================ ----------------------------------------------------------- Volume Singlefile Singlefile Multifile Multifile (Global) Write Rate Read Rate Write Rate Read Rate ----------------------------------------------------------- 16x16x16x32 0.7187 0.5789 6.7869 23.4311 24x24x24x32 0.6563 0.5299 18.8601 6.9612 24x24x24x64 0.6542 0.529 15.1104 5.5429 Notes: i) The Singlefile times are rather bad: 100s = 1.7 min 124s = 2.1 min 370s = 6.2 min 458s = 7.6 min 724s = 12 min 918s = 15.3 min The multifile times are acceptable... ii) There are large fluctuations in the Multifile read and write times. This may (or may not) be to do with the performance of the NFS server being written to/read from. --------------090202030207050602030802-- - ---------------------------------------------------------------------- Message 2 - ---------------------------------------------------------------------- X-Authentication-Warning: snort.ph.ed.ac.uk: paboyle owned process doing -bs Date: Wed, 2 Mar 2005 14:33:03 +0000 (GMT) From: Peter Boyle To: Carleton DeTar cc: bj@ph.ed.ac.uk Subject: Re: QIO Rack tests In-Reply-To: <200503020027.RAA00934@dirac.physics.utah.edu> X-Scanned-By: MIMEDefang 2.43 Hi Carleton, [irrelevancies deleted] [ quoted from my reply... ] > My expectation is that with really big projects running on several K > nodes, we will have to use PARTFILE. SINGLEFILE (serial or parallel) > will be impractical. For small projects (few hundred nodes) any > format is probably OK. Our experience has been that SINGLEFILE if accessed in parallel can be reasonable on even thousands of nodes, depending on whether you call 25MB/s for I estimate 20s-1minute delays of compute reasonable. If you need more, writing to /pfs will give it, but distribute the directories on a per node basis. We're not losing much time to the optimised SINGLEFILE method of writing here. [irrelevancies deleted] Thanks, Peter > Best regards, > Carleton > - ---------------------------------------------------------------------- Message 3 - ---------------------------------------------------------------------- Date: Wed, 2 Mar 2005 08:51:20 -0700 (MST) From: Carleton DeTar To: paboyle@ph.ed.ac.uk CC: bj@ph.ed.ac.uk, detar@physics.utah.edu In-reply-to: (message from Peter Boyle on Wed, 2 Mar 2005 14:33:03 +0000 (GMT)) Subject: Re: QIO Rack tests X-Scanned-By: MIMEDefang 2.43 Hi Peter, > Our experience has been that SINGLEFILE if accessed in parallel can > be reasonable on even thousands of nodes, depending on whether you > call 25MB/s for I estimate 20s-1minute delays of compute reasonable. > If you need more, writing to /pfs will give it, but distribute the > directories on a per node basis. We're not losing much time to the > optimised SINGLEFILE method of writing here. I am not sure I understand your satisfaction with SINGLEFILE. I do not consider more than 5 minutes an acceptable read/write time for a smallish lattice (24^3 x 64). What are our I/O requirements? Plans in the first year are to run lattices as big as 48^3 x 96 = 3 GB for the gauge configuration and 6 GB for the corresponding Wilson propagator files. If we do heavy-light form factor calculations, a single lattice may have several such propagator files. Future lattices are sure to be larger. So ideally on several thousand nodes we would have a file system with an aggregate data transfer rate of the order 100 MB/s to transfer such files within minutes. How do these requirements fit Balint's benchmarks? The local volume for 48^3 x 96 on 8K processors would be 6^4, which corresponds approximately to Balint's 24^3 x 64 on 1K nodes. Balint's MULTIFILE numbers of the order 10 MB/s to 1000 nodes scales to about 80 MB/s on 8K nodes, close to our goal. Not bad for starters. But Balint's numbers for SINGLEFILE are under 1 MB/s, more than two orders of magnitude too small. I don't know how much better a clever routing algorithm could do, but even if routing were not a bottleneck, could you get 100 MB/s through a single NFS connection? So unless you are running small lattices, we need PARTFILE. Regards, Carleton - ---------------------------------------------------------------------- Message 4 - ---------------------------------------------------------------------- X-Authentication-Warning: snort.ph.ed.ac.uk: paboyle owned process doing -bs Date: Wed, 2 Mar 2005 16:03:41 +0000 (GMT) From: Peter Boyle To: Balint Joo cc: Carleton DeTar Subject: Re: QIO Rack tests In-Reply-To: <4225E2A7.8010404@ph.ed.ac.uk> X-Scanned-By: MIMEDefang 2.43 Thanks Balint, > > I am not sure I understand your satisfaction with SINGLEFILE. I do not > > consider more than 5 minutes an acceptable read/write time for a > > smallish lattice (24^3 x 64). > > Here Pete was mentioning SINGLEFILE accessed in parallel. Ie > more than one I/O node writes, each to its offets. So this is the old > parallel IO to single file mode again, not the QIO SINGLEFILE. We should > give this mode a name. It is essentially PARTFILE/MULTIFILE with no > host conversion needed. As you can see we all get confused with the > various labels. Yes, I did indeed mean the SINGLEFILE whose implementation would be parallel access. The call and end result would be the same but the parallel access would result in a much faster (20x) implementation. I would only be satisfied with this performance. Thanks, Peter > All the best, > Balint > > > - ---------------------------------------------------------------------- Message 5 - ---------------------------------------------------------------------- Date: Wed, 2 Mar 2005 12:14:49 -0700 (MST) From: Carleton DeTar To: paboyle@ph.ed.ac.uk CC: bj@ph.ed.ac.uk, detar@physics.utah.edu In-reply-to: (message from Peter Boyle on Wed, 2 Mar 2005 16:03:41 +0000 (GMT)) Subject: Re: QIO Rack tests X-Scanned-By: MIMEDefang 2.43 Hi Peter, Thanks for the clarification. So the idea is to have 8K to 10K nodes writing to and reading from a single file. You would need to provide NFS mounts from all nodes to all RAID boxes. Where does this file reside? Are you striping across multiple RAID boxes without regard to the distribution of lattice sites? Or do you expect a single RAID box to manage all 8K - 10K NFS mounts? FNAL has questioned whether that is possible because when you exceed the throughput capacity of the RAID system, it can be overwhelmed, resulting in a huge performance hit. I would want a demonstration that it works. As for terminology, the specifications we had originally planned for QIO are volume format = {SINGLEFILE, MULTIFILE, PARTFILE} and I/O mode = {SERIAL, PARALLEL}. The current placeholder for the I/O mode parameter is the last QIO_ioflag argument to QIO_open_write and QIO_open_read. So I would call the alternatives SINGLEFILE (parallel mode) and SINGLEFILE (serial mode). (But of course there is no QIO implementation of the parallel mode at present. To make it work requires some coordination with LIME, which is built on a serial streaming model.) Regards, Carleton - ---------------------------------------------------------------------- Message 6 - ---------------------------------------------------------------------- X-Authentication-Warning: snort.ph.ed.ac.uk: paboyle owned process doing -bs Date: Wed, 2 Mar 2005 20:41:44 +0000 (GMT) From: Peter Boyle To: Carleton DeTar cc: bj@ph.ed.ac.uk Subject: Re: QIO Rack tests In-Reply-To: <200503021914.MAA01420@dirac.physics.utah.edu> X-Scanned-By: MIMEDefang 2.43 Hi Carleton, 1. So far we haven't been using more than 1k nodes in anger for concurrent writing but have demonstrated good bandwidth with 1k nodes concurrently writing to a filesystem that exports and supports 4k mounts due to being shared amongst 4 racks. This has been entirely successful, in part through tuning Linux kernel parameters and in part because I wrote the NFS client and have complete control of all timeouts and retransmit rates. How careful has FNAL's tuning been? I'd guess (but do not know) 4k simultaneous mounts and 1k simultaneous writing nodes exceeds Fermilab's experience, but agree that the 8k extrapolation is indeed an extrapolation. The single file will reside on a disk on the IBM Pseries, which *should* be beefier than Linux, but your demonstration will have to wait until have 8k nodes cabled together and working. We have been unable to do so because of problems with our Fibre Channel SAN. 2. in practice getting 8k nodes for a single job in my estimation is unlikely. It is after all a shared facility. 3. we need to distinguish carefully the globally shared /host file system, which is a directly accessed disk logical volume group (or in our case FibreChannel SAN) on the IBM frontend, and the linux based Raid servers which will ultimately provide /pfs. Writing a single file will necessarily be to this /host file system, since that is the only way to obtain a single file. Thus striping across RAID boxes is not possible in this mode. When writing to the PFS, this is necessarily multi-file mode. In this mode I will stripe in a way that two consecutive motherboards are on the same RAID, but each rack mounts 8 RAIDS, with every group of four RACKS sharing the same 8 RAIDS. 4. Columbia have implemented the option of continuously throttling down the number of _concurrent_ writers to be anything between ALL and ONE. As I said, this is an implementation issue. Note that ALL mount the disk simultaneously, and indeed ALL write. The question is when, and again compared with the naive single file, this is an implementation issue and performance issue. With this continuously variable throttling, I could virtually guarantee you no issues with 8k compared with the 1k we have demonstrated, since an NFS server is by design stateless and maintains no growing "state" whether it has dolled out 8k mounts or 1k mounts. The load is controlled only by the number of writers. Just because one could implement the parallel write to single file in a spectacularly agressive way doesn't mean that one has to. Adding a placeholder "hint" to QIO for concurrency in the SINGLEFILE case would be wise. Subsequently the two extreme cases use a value of either 1 or ALL, however strictly the intermediate cases can either use CONCURRENCY I/O nodes and pass through the comms network, or can have all nodes do their own I/O but schedule the access. Again an implementation issue. I would finish by reemphasizing that our existing setup actually runs with 4 racks on one LinuxRAID disk, i.e. we typically have 4k simultaneous mounts, but that since the running jobs rarely write out simultaneously the demonstration is only really valid at the 1k level. Best wishes, Peter - ----------------------------------------------------------------------