From: Celso Mendes
Subject: QMP tests


As I had promised, I'm sending below my status on the QMP
testing issue, and a summary of my changes to QMP that
I had mentioned in the latest telecon. Because I will be
on travel during all day this Wed., I won't have a chance
to check my e-mail until later in the evening, from
N.Carolina.

First, in regard to J-Lab tests: I got my account setup
this Tuesday afternoon, was able to login to the main
gateway, but couldn't yet find how to reach the 128-node
cluster (I'm assuming that's where I should run tests).
Chip/Robert: cold you please help me on this? The site
I was given (http://www.jlab.org/hpc/home) has a picture,
but I could not find any names for the clusters.

Second, for the test suite: in the last few days, I have
done additional tests at Fermilab, using both the mpich-gm
version and the gm version of QMP. I'm observing some
unexpected behavior with the gm version. While all of the
tests pass when I use the mpich-gm version, three of the
tests fail under gm, with error messages related to
'time-out'. The test reports can be found under

   http://bugle.cs.uiuc.edu/~cmendes/QMP/gm_report.ps
   http://bugle.cs.uiuc.edu/~cmendes/QMP/mpich_report.ps

I might be missing something in the form of running gm
codes at Fermilab; even some of the original JLab example
programs (e.g. QMP_grid_test.c), linked against Fermilab's
QMP-gm library, produced similar errors. I really need to
run these at JLab, to see what will be the behavior there.

Finally, in regard to my changes to QMP: I'm sending a
list with the required changes. There were two classes of
changes: (a) those required to enable the test-suite to
work, and (b) those required by the QMP profiling support.
Basically, the changes in (a) were required because of
printf's in QMP that are not under control of QMP_verbose,
or because the machine name is printed, which breaks the
reproducibility required by the tests.

-Celso

=============================================================
CHANGES TO QMP IMPLEMENTATION:
-----------------------------

(A) Changes required by the test-suite:

A-1) File QMP_grid.c: comment-out lines 415/.../428
     These lines cotain a bunch of 'printf'

A-2) File QMP_init_mpi.c: comment-out line 371
     This line has a 'printf' 

A-3) Files QMP_util_mpi.c and QMP_util_gm.c:
     Change function QMP_printf() such that the hostname is
     *not* printed


(B) Changes required by the profiling support:

B-1) Files QMP_P_MPI.h and QMP_P_GM.h:
     Insert, near the top, a line with   #include "QMP_profiling.h"

B-2) Add to the set of header files the file "QMP_profiling.h" which
     has contents such as:

     #pragma weak QMP_start = PQMP_start
     #undef QMP_start
     #define QMP_start PQMP_start

     This should be repeated for each QMP function of interest.
     It will enable profiling tools to define 'QMP_start' and let
     the tools do whatever they want to do before calling the real
     library function, redefined as 'PQMP_start'

B-3) Add to the library the dummy function "int QMP_profcontrol (int level)"


==============EARLIER COMMENTS: Celso, Chuwloo and James Osborn ===========

From cmendes@cs.uiuc.edu  Sat Feb 21 23:58:10 2004
Subject: Re: QMP msgmem

>  1)  I see we have a function
>  QMP_msgmem_t QMP_declare_strided_array_msgmem (void ** base, QMP_u32_t*
>    blksize, QMP_u32_t *nblocks, QMP_u32_t* stride);
>  which I guess allocates several strided msgmem together.  It needs to know
>  the length of the input arrays like
>  QMP_msgmem_t QMP_declare_strided_array_msgmem (void ** base, int* blksize,
>    int* nblocks, int* stride, int n );

Yes, I agree. Other than this, one possibility to keep the
original call would be a convention such that the **base would
contain at some point a null pointer base[j], meaning that there
were 'j' valid elements in the set. I prefer James' suggestion too.
Either way, it should be defined in the specs.

>  (Incidentally the API now seems to be littered with QMP_u32_t's instead
>  of int's.  We never discussed this and I think it's better to leave
>  everything as int's.)

Well, these have been in the specs for a while. James: what makes
you think the explicit form (e.g. 'int') is better?

>  2)  Are there any potential constraints on how much 'aligned memory'
>  there is?  It seems that to allow communications directly out of fields
>  they should all be allocated with QMP_allocate_aligned_memory.  Does this
>  ever come out of a special pool of memory that could be exhausted before
>  the total system memory?  If so could it then just return regular memory
>  as long as that is available?
>  
>  
>  3)  It is possible (likely) that someone will have many msgmem's that
>  contain overlapping memory.  Doing concurrent sends from overlapping
>  memory regions should be allowed.  Concurrent receives are undefined.
>  This is consistent with MPI and I think is the policy we should adopt.
>  Is there any problem with this?

I don't see a problem in leaving it 'undefined'. And I agree with
keeping it consistent with MPI, since changing that could make it
very confusing for users already familiar with MPI-style coding.

-Celso


From chulwoo@phys.columbia.edu  Fri Feb 20 15:24:13 2004
Dear James and all,

I added reponses for some of the questions James had.
On Fri, 20 Feb 2004, James Osborn wrote:

> 
> Hi All,
> 
> I've been looking over the definitions for the msgmem functions and had a
> few comments/questions.
> 
> 1)  I see we have a function
> QMP_msgmem_t QMP_declare_strided_array_msgmem (void ** base, QMP_u32_t*
>   blksize, QMP_u32_t *nblocks, QMP_u32_t* stride);
> which I guess allocates several strided msgmem together.  It needs to know
> the length of the input arrays like
> QMP_msgmem_t QMP_declare_strided_array_msgmem (void ** base, int* blksize,
>   int* nblocks, int* stride, int n );
> 

It should be. It is, in fact, how it is declared in QCDOC implementation.
This never came up since this was never used so far. We should have done a
better job at checking.

> (Incidentally the API now seems to be littered with QMP_u32_t's instead
> of int's.  We never discussed this and I think it's better to leave
> everything as int's.)
> 
> 
> 2)  Are there any potential constraints on how much 'aligned memory'
> there is?  It seems that to allow communications directly out of fields
> they should all be allocated with QMP_allocate_aligned_memory.  Does this
> ever come out of a special pool of memory that could be exhausted before
> the total system memory?  If so could it then just return regular memory
> as long as that is available?
> 

There is a limitation on QCDOC, where the amount of faster Embedded DRAM
is 4MB. It can be easily implemented to do what James suggested,
allocating external DRAM if internal one is exhausted.
However,  adding a flag would give user better control of the memory
allocation.

> 
> 3)  It is possible (likely) that someone will have many msgmem's that
> contain overlapping memory.  Doing concurrent sends from overlapping
> memory regions should be allowed.  Concurrent receives are undefined.
> This is consistent with MPI and I think is the policy we should adopt.
> Is there any problem with this?
> 

Allowing overlapping sends are not problematic for QCDOC. There are,
however, cases where cache coherency becomes an issue, such as a strided
transfer is matched with modification on the other parts of the memory.
I don't believe we have such occurences from existing platforms (MILC,
Chroma, QDP, CPS) and we should specify that a concurrent modification and
transfer within a memory block allocated by a single
QMP_allocate_aligned_memory is undefined.


> 
> -James Osborn
> 
> 
>