Software Coordinating Committee Conference Call
		       August 25, 2005 1:00 PM EST
			  Recorder: C. DeTar

Present: 

   DeTar, Osborn, Brower, Blum, Watson, Mawhinney, Zhang, Holmgren, Simone, 
   Pochinsky, Gottlieb, Renner, Edwards, Jung

Absent:
   
   Efstathiadis, Sommervoll
   
    
** action items
=====================================================================

0.) CRC errors reported by the MILC QMP communications layer. 
             (Carleton and James  -- see note appended.)

    Holmgren:  Gottlieb is generating 24^3 x 64 lattices and 
               was getting CRC communication errors on the FNAL 
               Infiniband cluster.  They are repeatable.
               On Myrinet, gm gave a different error.
               With VMI he was getting a message not completed error
               but not when the number of time steps was reduced.
    DeTar:     This sounds like a problem that Steve and I should
               tackle.  
    DeTar:     Dru is seeing occasional CRC errors on the QCDOC with
               essentially the same software.  We wondered why
               the message passed the underlying tests but failed
               on the MILC test.
    Mawhinney: Every 64 bits sent on a link is parity checked, and
               single-bit errors are corrected.  Then every 64 bits 
               are XORed in software and kept until the end of the run.
               The qhdrware check command should tell if those sending
               and receiving node checksums mismatch.
               I suppose an ASIC error is possible, but we have never 
               seen one.
    Jung:      Would the lattice still be reproducible if we ignored
               the CRC32 error?
    DeTar:     Let's tackle the checksum error on the cluster.
               In the meantime let's see if not aborting on the QCDOC
               but just reporting the error still leads to the same lattice.
    Watson:    Do you routinely run a test that exchanges random data
               and verifies integrity?
    Mawhinney: Yes, not really random. Our test runs the same QCD simulation
               on each node and exchanges results and compares.

1.) Status of US QCDOC Hardware Partitions. 
       Particularly hardware reliability and file I/O
             (Bob, Chulwoo, Kostia)

    Mawhinney: Rack 19 is now available to Robert for DWF
               pfs is working.
               Chulwoo is currently running MILC code on the 4K to
               shake down problems.
    Dru:       I have run some 7 trajectories on the 2K node,
               counting reruns.  So far so good.
    Edwards:   Rack 16 is taking a lot of my time now,
               so someone else should go ahead and use Rack 19 .

               Show stoppers:
               1. Max file handles is 64 under 2.6.0
                  Not a problem for us on clusters.
                  We will recompile to use fewer for now.
               2. I/O time. On /pfs it takes 1000 sec to write.
                  a propagator (2.4 GB).  I may need to
                  switch to memory instead of disk.
               3. Memory limitation.  Can't fit Ls 12 or 16
                  but can fit Ls 8. 2.6.0 is supposed to
                  give us 112 MB.  I haven't looked to
                  see what I actually got.  I am in the process
                  of constructing a memory map to check for fragmentation.

               Smaller problems:
               4. Barrier call in QIO was calling problems.
                  Pint needed to be much higher.
                  I changed it to a global sum and it worked.
               5. NDIO error inside qreset_boot.
               6. qrun takes some 14 sec to start

   Mawhinney:
               2. Chulwoo's tests suggests 100 Mbit/s
                  should be achievable.on a given RAID.
                  We have seen a lot of Ethernet resets
                  which might explain degradation of performance.
                  Some NFS tuning might help.  We are looking 
                  into it.
               3. You should have 112 MB * 800 = 100 GB
                  of memory.
               4. I worry about using a barrier technique that could hang
                  with no explanation.
               6. There is a Gbit link from the host to all the 
                  compute nodes.  Perhaps you are saturating it.

   Jung:       5. Such an error usually requires power cycling

   Brower:     What can we do to speed things up?
   Jung:       Problems look a lot worse because we are doing "short term"
               testing. Doing qrun and qreset_boot frequently causes
               more frequent problems.
   Mawhinney:  For powercycling requests, use CTS.  Try qreset_sys first.
               Eric and I are trying to find a way to set this up
               and not violate BNL cybersecurity so a very small
               number of alpha users can do it.
   Jung:       Don't run qreset_boot too frequently.
   DeTar:      Shouldn't we get to the point that power cycling isn't
               necessary?
   Mawhinney:  Since hardware problems with the ethernet system and other 
               components will very likely persist, we won't ever 
               reach that point, but we can hope the need becomes
               rare.
   Mawhinney:  It is really important to be using this machine
               for science.  We have done so with UKQCD and RBRC
               and we are close with the DOE machine now.
   Brower:     What % of the time are the machines being used?
   Mawhinney:  60-70% for my project on RBRC.  We are running with
               10% repetition and haven't seen problems.

Committee conference concluded at 3:00 PM EST.  
Next plenary call Sept 1 at 1:00 PM EST