A virtual message/geometry layer and some additions on physical layer
[R. Edwards, June 26, 2001]

Considerations:

  For a grid based problem (QCD), need code for laying out grid on underlying
machine. For a switch based machine, need to set neighbor table for each
direction. Must choose physical grid lengths. On a grid-based machine, must
map virtual grid (the problem size) onto physical hardware.  Neighbor table
must be determined. Separate above physical communication calls from
overlying grid. However, must set physical grid on non-grid based
architectures, so require some feedback mechanism.


Physical:

int initMachine(void)   or possibly  int initMachine(int argc, char
**argv)
int shutdownMachine(void)

int sumInt(int x)
bool globalOr(bool b)   // Requested by Carleton. This I would do with
sumInt

// Return info on the physical connectivity
//   0 for switch, 1 for grid, 2 for fat-tree/whatever,..
int getMachineType()

// Create a physical geometry
// Only called for a switch, fat-tree
// Error for a fixed grid machine (???)
void declareTopology(int *length, int rank)

// Check if physical grid declared?
// Only defined after initMachine()
//
// True on a grid-based machine
// Initially false on a switch, fat-tree, etc.
// NOTE: returns true if declareTopology() called successfully!!!
int isTopologyDeclared(void)

// Return physical number of communication nodes
// Only defined after initMachine()
int getPhysicalNumNodes(void)

// Return physical machine size
// Only defined after initMachine and possibly declareTopology()
void getPhysicalSize(int *length)

// Return physical dimension
// Only defined after initMachine and possibly declareTopology()
int getPhysicalDimension(void)


Virtual:

Virtual geometry object accessible by the user contains:
  Virtual problem size and dimension
  Axes ordering (mapping of virtual to physical axes)
  Virtual nodeid
  Virtual subgrid size and volume
  Access to physical grid info. Note, physical grid dimension could be
     less than virtual grid.
  Info on block-size, stride, num-blocks all in units of sites
     for each direction. Used for sending a subgrid face


// Create a geometry with fixed ordering. Ordering is mapping of
// virtual to physical axes,  ordering[0]=1, ordering[1]=0,
ordering[2]=2,etc.
//
// If isTopologyDeclared() false, will call declareTopolopy
//   so physical grid fits virtual grid
// If isTopologyDeclared() true (say 3d and want a 4d problem), will
//   layout virtual grid as requested on physical grid
geometry createDetailedGeometry(int *length, int rank, int *ordering)

// More automated geometry constructor.
// If isTopologyDeclared() false, ordering is trivial and call detailed
//    geometry constructor.
// If isTopologyDeclared() true, layout virtual grid optimally on
physical
//    grid (e.g., minimize surface area)
geometry createGeometry(int *length, int rank)

// Grid based send in virtual direction (must go through axis ordering)
// Possible macro call implementation
void * declareSendVirtual (geometry *geom, int nbytes, int
relativeVirtualHost,
     int direction, int remoteBuffer);
  where relativeVirtualHost is an integer, 0 [x],1 [y],2 [z],etc.
direction is +-1
// Logically implemented as
void * declareSendVirtual (int nbytes, int
geometry->ordering[relativeVirtualHost], int direction, int
remoteBuffer);


Code examples:
// On a 8 node switch and want automatic layout
// This may create a physical size of   1x2x2x2
int dim = 4;
int length[] = {4,8,12,16};
geometry geom = createGeometry(length, dim);

// On a switch but want to choose physical layout
if (! isTopologyDeclared())
{
  int phys[] = {1,1,1,8};
  int ordering[] = {0,1,2,3};
  declareTopology(phys, dim);
  createdetailedGeometry(length, dim, ordering);
}