Structured AMR Interoperability Effort

Data Aliasing

• Problem
  • The largest impediment to interoperability of various Structured AMR efforts within the DOE (and beyond) are related to data description.

• Goal
  • Provide a community-standard interface for describing Structured AMR data sets and associated data structures.
  • Enable different community members to focus efforts on component development without suffering from data-conversion losses (performance and memory).
  • examples: Chombo Elliptic Solver Component, AMR-HDF5 File IO Component.
• Approach
  • A API written in ANSI-C for client components to describe their AMR data sets to server components.
  • AMRData

Client Component functions

void AMR_Create(int MPI_communicator,
                int components,
                int dimension,
                int centering,
                int datatype,
                int ordering,
                int* levelhandle);

void AMR_addPatchFloat( int levelhandle,
                        int* loRange, int* hiRange,
                        int* loValid, int* hiValid,
                        float* dataPtr);

void AMR_Close(int levelhandle);

void AMR_Delete(int levelhandle);

Server Component functions

void AMR_GetInfo(int levelhandle,
                 int* MPI_communicator,
                 int* nComponents,
                 int* dimension,
                 int* centering,
                 int* datatype,
                 int* ordering,
                 int* localcount);

void AMR_GetPatchInfo(   int levelhandle, 
                         int** loRange, int** hiRange,
                         int** loValid, int** hiValid);
void AMR_GetPatchFloat(  int levelhandle, float**  dataPtr);
void AMR_NextPatchInfo(   int levelhandle, 
                          int** loRange, int** hiRange,
                          int** loValid, int** hiValid);
void AMR_NextPatchFloat(  int levelhandle, float**  dataPtr);

Example: Writing a Server Component

extern "C" {
  int helm3_(  double dx, double helmsoeff, double scale, 
               double tolerance, int maxIter, int minIter,
               int* domain_lo, int* domain_hi, int phi,  int rhs  ) 
{
  int p_comps, p_dimension,p_centering,p_type,p_ordering,p_count;
  AMR_GetInfo(phi,&p_comps, &p_dimension,&p_centering, 
                  &p_type,&p_ordering,&p_count);
//---perform checks on input data here.... return codes, etc. omitted---
  AliasDataFactory phiFactory(phi);   
  AliasDataFactory rhsFactory(rhs);
  levelData<FArrayBox> p(phiFactory); 
  levelData<FArrayBox> r(rhsFactory);
  
  ProblemDomain pd(Box(domain_lo, domain_hi));
  HelmHoltzOp hop;                
  LevelSolver solver;  
  hop.setHelmCoeff(helmcoeff);    
  hop.scaleHelmCoeff(scale);  
  solver.setTolerace(tolerance);  
  solver.setMaxIter(maxIter);
  solver.define(p.getBoxes(), NULL, pd, dx, 1, &hop, p_comps);
  solver.levelSolveH(p, r);
  return 0;
  }
}

Example: C++ Client Using a Server Component

void ChomboHelmHoltz3(XArray3<Grid3<double> >& phi,
            XArray3<Grid3<double> >& rhs,
            double dx, double helmcoeff, double scale, 
            double tolerance, int maxIter, int minIter,
            const Region& domain)
{
  int p, r, i;
  AMR_Create( 1, 3, 1, 3, 0, &p);  
  AMR_Create( 1, 3, 1, 3, 0, &r);

  for (nodeIterator ni(phi); ni; ++ni) 
  {
    i = ni();   
    double *p_ptr(phi.ptr(i)),  *r_ptr(rhs.ptr(i));
    Region reg = phi.floorplan(i);  
    Region valid(reg);  
    valid.grow(-1);
    AMR_addPatchDouble(p, reg.lower()(), reg.upper()()
             valid.lower()(), valid.upper()(), p_ptr);
    AMR_addPatchDouble(r, reg.lower()(), reg.upper()()
             valid.lower()(), valid.upper()(), r_ptr); 
  }  
  AMR_close(p);  
  AMR_close(r); 
  helm3_(dx, helmcoeff,scale, tolerance, maxIter, 
        minIter,domain.lower()(), domain.upper()(), p, r);
  
  AMR_Delete(p);
  AMR_Delete(r);
}

Example: Fortran Client Using a Server Component

This example is an atem

  subroutine CHhelm3(dx, helmcoeff, scale, tolerance, maxIter, minIter)
    integer maxIter, minIter, phi_handle, rhs_handle, offset, lo(3), hi(3)
    real*8 dx(3),  helmcoeff, scale, tolerance)
    DBASEMODULE()
    call amrcreate( 1, 3, 1, 3, 0, phi_handle)
    call amrcreate( 1, 3, 1, 3, 0, rhs_handle)

    do this_block = 1, level_blocks
      call dBaseGetIntCoords( "phi", this_block, lo, hi)
      call dBaseGetDataOffset("phi", this_block, offset)
      call amraddpatchdouble(phi_handle, lo, hi, lo, hi, DATA(offset))        
      call dBaseGetDataOffset("rhs", this_block, offset)
      call amraddpatchdouble(rhs_handle, lo, hi, lo, hi, DATA(offset))
    end do
    call amrclose(phi_handle)
    call amrclose(rhs_handle)
    call helm3(dx, helmcoeff,scale, tolerance, maxIter, 
  &     minIter,dl, dh , phi_handle, rhs_handle)
    return
    end

Pros and Cons

• Pros
  • Portable.
  • All computer languages currently provide mechanisms for calling C in a portable manner.
  • Integer opaque handles and fundamental type interfaces make binding to Fortran and other languages simpler.
  • fundamental type interfaces remove some namespace and datatype collisions
• Cons
  • Using integer opaque handles thwarts type-safety.
  • Most usage errors become runtime errors instead of compile/link errors.
  • Thsi requires people to drop to the ugly details of C for their interoperability.
  • Needs to leverage the MPI-2 functions MPI_Comm_c2f and MPI_Comm_f2c to create language independent fundamental types for MPI Communicators.

AMR-HDF5 Interface

• HDF5
  • portable binary files.
  • supports additional meta data for specific application domains.
  • hyper-slab. (out-of-core tools). Ability to read in specific component of a specific box.
  • parallel I/O support on most platforms.
• readable/(writable) by ChomboVis
• File organization
  • component names
  • grid spacing
  • centerings
  • dimensions
  • data organized as levels of grids
  • parallel neutral (but contains processor assignments from when the file was written)

Do I have to learn HDF5 ?

• HDF5 is a large, powerful, and complicated API.
• Make a Server Component that provides the capabilities needed to write such files. A user would alias their AMR Data using the AMRData API, then invoke the C interface to AMR-HDF5.

    AMR_OpenHDF5(char *filename, int MPI_Communicator, int mode, int *hdf5)

    AMR_WriteData(int hdf5, int numLevels, int *levelData, int ncomp,
		       char **compNames, int *refineFactor, 
               double *spacing, double time, double dt, 
               CONST int *domain_lo, CONST int *domain_hi)

    AMR_ReadMetaData(int hdf5, int *numLevels, int *ncomp, 
			int *ndim, double *spacingCoarse, 
			int *domain_lo_valid, int * domain_hi_valid,
			double *time, double* dtCoarse) 

    AMR_ReadLevel(int hdf5, int level, int levelDatahandle, 
               int compBegin, int ncomp)

• AMR-HDF5.h

Current Status

• AMRdata.[h,c] : written and preliminary testing.
• AMR-HDF5.[h,cpp] : written using Chombo as the implementation. Can be downloaded and used without needing to install Chombo.
• Common file format specification under review.
• Next:
  • implement new file format with independent HDF5 c code. AMR-HDF5 as standalone product for community.
  • work with community to determining what features of Chombo should be presented as Server Components.
  • Consider where AMRdata interface might need extension or improvement.
  • Add more functions to AMR-HDF5 to enable finer control of the writing process. The current model has a monolothic write function, and a more database-like reading API. Users would like to have a more incremental approach to writing as well.