Coding Standards

A Flash-X unit is expected to be fully encapsulated, with all of its data being private to it. The only global data accessible to a unit is in the form of included foo.h files, two of which are available to every unit and every application instance. These are Simulation.h, which is generated by the setup tool and constants.h which contains all the global constants in the code. Of these Simulation.h does not exist in the source tree, whereas constanst.h lives in the Simulation unit. Each unit is required to have a file Unitname_data.F90 that declares all the unit scope data.

The Unit_interface.F90 file for the unit is to be decorated with annotations in the form of comments to inform the packet code generator about the data that needs offloading. The data are in one of the following categories:

device resident data – This is the data that is assumed to be available on demand while computing. If the computation is offloaded to a device, this data is sent over once at the begining and remains on the device

device resident space allocation – This indicates needed scratch space to be used during computation. A good practice is to allocate before beginning the computation and deallocate once the computation is done.

sent not returned – This is the data that gets updated on the host and needs to be sent over to the device at every iteration of the computation. But once used it does not need to be updated, and therefore returned.

sent and returned – This is the data that needs to be returned to the host after computation is done.

returned only – This is the data that is generated during computation and needs to be returned to the host, but there is no corresponding sending to the device.

Note that the data could be in the form of scalars, derived types, or any other kind of data structures that are compatible with the target devices.

If a unit intends to be compatible with offloading options then it must not have any scratch space allocation interspersed with the computation. Need for scratch space must either be registered with the framework through SCRATCH keyword in Config file, or declared in the Unit_data.F90 file or be included in at least one of the unit’s API interfaces.

Structure of a Unit

Each unit has its own top level directory Unitname where capitalization of the first letter indicates that the directory name is a namespace. A unit can have subunits that show up as subdirectories named as UnitnameSubunitname. Each unit has at least one subunit which is named UnitnameMain. Except UnitnameMain, each subunit must be encapsulated in its own right such that an application instance can opt to exclude it if it wants to. Each unit publishes its API through which other units can interact with it. API can be subdivided among subunits such that the union of all API functions is the unit API, but the intersection set of API functions implemented by different subunits is NULL. Thus every API function is implemented by exactly one subunit. Within the subunit any API function can have multiple alternative implementations. Alternative implementations reside under different subdirectories of the subunit. This division into alternative implementations can be applied to arbitrary depth in the directory structure, the only constraint is that when a directory is included in the application every file (not subdirectories) in that directory is included. The figure below illustrates the unit structure with the example of Particles unit. This unit has three subunits, each subunit supports two alternative implementations of the full subunit. The MeshOwned implementation assumes that the data structure is owned by the mesh, while the ParticlesOwned implementation assumes that the data structure is owned by the Particles unit. Currently MeshOwned implementation works with AMReX while ParticlesOwned implementation works with Paramesh. In future ParticlesOwned method will also be compatible with AMReX. Here time integration of particles is implemented in ParticlesMain, with three different implementations for ParticlesOwned version. Note that passive is an organizational directory here because active particles will be brought over from FLASH in future. Different time integration schemes relevant to active particles will be clustered under the corresponding organizational directory. The figure is does not show all the directories present in the unit for clarity of presentation.

Organization of Particles unit as an example of unit structure with subunits.

Generally, the alternative implementations are mutually exclusive, however, in certain circumstances more than one may need to be included. For example if work is to be divided between CPU and GPU from the same unit, both variants will need to be included. The mechanism described in Architecture chapter explains how this is done.

Placement of functions in the directories affects how inheritance works. A common rule of thumb is, if there are multiple alternative implementations beneath a directory foo with some common functions then those common functions should be placed in foo. For example in the Grid unit, the AMR implementation has two variants, AMReX and Paramesh. The files that contain common implementations are kept in AMR directory, while files that have code unique to each variant reside in the respective subdirectories. Following the inheritance rules implemented by the Setup tool, all files in AMR will be inherited by, and therefore included with whichever variant is included.

Application Programming Interface (API)

A unit’s API is the set of interfaces through which other units can interact with it. Every interface in the API has a null implementation that resides in the top level namespace directory for the unit. There is no limit on the number of interfaces a unit’s API can have. A good rule of thumb is to expose only the functionality that other units really need to access, and keep everything else private. The Makefile snippet in the top level unit directory includes the names of all API member files. Files that have the null implementations of the interfaces also provide documentation about the use of the interface, which can be viewed on the API pages of the Flash-X website.

In addition to unit level API, there is a provision for local API in every unit. A unit needs to have a subdirectory named localAPI under the top level unit directory only if it has more than one subunit, and at least one of the subunits has interfaces that are private to the unit but not to the subunit. For example Grid unit calls gr_initSolvers from its own initialization routine. However, gr_initSolvers interface resides in GridSolvers subunit. localAPI subdirectory contains null implementations of all such interfaces, and the accompanying Makefile snippet includes the names of all the files present in the subdirectory. Thus if a subunit is excluded in an application instance its null implementation gets included. The Config file of the unit must use the keyword CHILDORDER to specify that localAPI must be scanned before any other subdirectory to correctly implement the inheritance.

Good Practices for Maintenance

Use macros to avoid code duplication

For maintenance and extensibility it is important to minimize code duplication. In FLASH it wasn’t always possible because the granularity of alternative implementations was at the level of a function/subroutine. However, introduction of macros has made it possible to minimize code duplication, and they should be used as often as needed. It is critical to follow naming conventions in defining macros to avoid possibility of naming conflict because their scoping cannot be enforced otherwise.

Use easy to search and replace names

Since naming conventions are not fully finalized it is safer to adopt some local convention in private functions that is easy to search and replace with scripts in case there is a need to change.

Add inline comments about design choices

They come in handy for those who were not the original developers of the code to understand and maintain it. They are also useful for refactoring if needed.

Use keyword UNOFFICIAL for not ready code

There is a provision to add new code to the main branch of the repository before it is blessed for production. Such code should have keyword UNOFFICIAL in its Config file, so that the Setup tool knows to abort if the corresponding code is included in an application. Setup command line option -with-unofficial overrides the abort.