The Flash-X configuration

The configuration toolchain effectively implements the Flash-X software architecture. The encapsulation and inheritance of the code is implemented and enforced by this tool. It relies upon Flash-X’s domain specific configuration language (DSCL) that encodes meta-information about the components in the accompanying Config files. A Config file is unique to a unix directory and has all the meta-information about files in that directory. It can also include information about how to traverse subdirectories of the corresponding component. The Config files are parsed by the setup tool.

Config files come in two syntactic flavors: static text and python. In static mode, configuration keywords are listed as lines in a plain text file. This mode is the most readable and intuitive of the two, but it lacks flexibility. The python mode has been introduced to circumvent this inflexibility by allowing the configuration file author to specify the configuration directives as a function of the setup variables with a python procedure. This allows the content of each directive and the number of directives in total to be amenable to general programming.

The rule the setup script uses for deciding which flavor of configuration file it’s dealing with is simple. Python configuration files have as their first line ##python:genLines. If the first line does not match this string, then static mode is assumed and each line of the file is interpreted verbatim as a directive.

If python mode is triggered, then the entire file is considered as valid python source code (as if it were a .py). From this python code, a function of the form def genLines(setupvars) is located and executed to generate the configuration directives as an array (or any iterable collection) of strings. The sole argument to genLines is a dictionary that maps setup variable names to their corresponding string values.

As an example, here is a configuration file in python mode that registers runtime parameters named indexed_parameter_x where x ranges from 1 to NP and NP is a setup line variable.

##python:genLines

# We define genLines as a generator with the very friendly “yield” syntax. # Alternatively, we could have genLines return an array of strings or even # one huge multiline string. def genLines(setupvars): # emit some directives that dont depend on any setup variables yield “”” REQUIRES Driver REQUIRES physics/Hydro REQUIRES physics/Eos “”” # read a setup variable value from the dictionary np = int(setupvars(“NP”)) # must be converted from a string # loop from 0 to np-1 for x in xrange(np): yield “PARAMETER indexed_parameter_%d REAL 0.” % (x+1)

When setting up a problem with NP=5 on the setup command line, the following directives will be processed:

REQUIRES Driver REQUIRES physics/Hydro REQUIRES physics/Eos PARAMETER indexed_parameter_1 REAL 0. PARAMETER indexed_parameter_2 REAL 0. PARAMETER indexed_parameter_3 REAL 0. PARAMETER indexed_parameter_4 REAL 0. PARAMETER indexed_parameter_5 REAL 0.

Config file Keywords and syntax

The syntax of the configuration directives is described here. Arbitrarily many spaces and/or tabs may be used, but all keywords must be in uppercase. Lines not matching an ad missible pattern will raise an error when running setup. The syntax of DSCL includes two types of keywords. The directive keywords define actions to be taken by the setup tool, while the non directive keywords encode information about the directives. The non directive keywords are associated with specific directives keyworks.

# comment

A comment. Can appear as a separate line or at the end of a line.
DEFAULT sub-component

Any component can designate one component (which is effectively a subdirectory) to be the “default”, with this keyword. If no specific sub-component is selected by the application, the designated default implementation gets included.
EXCLUSIVE component…

Specifies a list of components that cannot be included together.
CONFLICTS unit1[/sub-unit[/component…]] …

Specifies that the current component is not compatible with

the list of components that follow. setup issues an error if
the user attempts a conflicting configuration.
REQUIRES unit[/sub-unit[/component…]] [ OR unit[/sub-unit…]]…

Specifies a component that is required. The component can be

specified with partially specified or full specified path. If partial path is specified then full path is generated by following the “DEFAULT” options of the subcomponents.
REQUESTS unit[/sub-unit[/component…]]

Is similar to REQUIRES except that it is a soft

dependency. It can be overridden through either a commandline specification or through dependencies specified in higher priority locations in the source tree.

VARIANTS variantname variantname …

|Indicates that multiple alternative

implemetations of a component are to be included in the setup where each implementation gets its interface and routine/function name appended with the corresponding variant name. For example, one may want to use both CPU and GPU to compute. A component that has valid implementations for both can specify variants as shown below.

VARIANTS CPU GPU

Then if the setup options specify inclusion of both implementations, the names of the relevant files and function/subroutine interfaces are modified to have unique names. If, on the other hand, only one of the implementations is to be included, the default is Null and all the names remain unmodified.

SUGGEST component component …

Unlike REQUIRES, this keyword suggests that the current component be used along with one of the specified component. The setup script will print details of the suggestions which have been ignored. This is useful in catching inadvertently omitted code components before the run starts, thus avoiding a waste of computing resources.
UNOFFICIAL

This keyword suggests that the current component is not fully supported. The setup script will abort if a component with this keyword in included. It can be overridden as follows:

–with-unofficial= component

PARAMETER name type [``CONSTANT``] default [range-spec]

Specifies a runtime parameter. Parameter names are unique up to 20 characters and may not contain spaces. Admissible types include REAL, INTEGER, STRING, and BOOLEAN. Default values for REAL and INTEGER parameters must be valid numbers, or the compilation will fail. Default STRING values must be enclosed in double quotes ("). Default BOOLEAN values must be .true. or .false. to avoid compilation errors. Once defined, runtime parameters are available to the entire code. Optionally, any parameter may be specified with the CONSTANT attribute (e.g., PARAMETER foo REAL CONSTANT 2.2). If a user attempts to set a constant parameter via the runtime parameter file, an error will occur.

The range specification is optional and can be used to specify valid ranges for the parameters. The range specification is allowed only for REAL, INTEGER, STRING variables and must be enclosed in ’[]’.

For a STRING variable, the range specification is a comma-separated list of strings (enclosed in quotes). For a INTEGER, REAL variable, the range specification is a comma-separated list of (closed) intervals specified by min ... max, where min and max are the end points of the interval. If min or max is omitted, it is assumed to be \(-\infty\) and \(+\infty\) respectively. Finally val is a shortcut for val ... val. For example

PARAMETER pres REAL 1.0 [ 0.1 … 9.9, 25.0 … ] PARAMETER coords STRING “polar” [“polar”,”cylindrical”,”2d”,”3d”]

indicates that pres is a REAL variable which is allowed to take values between 0.1 and 9.9 or above 25.0. Similarly coords is a string variable which can take one of the four specified values.
D parameter-name comment

Any line in a Config file is considered a parameter comment line if it begins with the token D. The first token after the comment line is taken to be the parameter name. The remaining tokens are taken to be a description of the parameter’s purpose. A token is delineated by one or more white spaces. For example,

D SOME_PARAMETER The purpose of this parameter is whatever

If the parameter comment requires additional lines, the & is used to indicate continuation lines. For example,

D SOME_PARAMETER The purpose of this parameter is whatever D & This is a second line of description

You can also use this to describe other variables, fluxes, species, etc. For example, to describe a species called “xyz”, create a comment for the parameter “xyz_species”. In general the name should be followed by an underscore and then by the lower case name of the keyword used to define the name.

Parameter comment lines are special because they are used by to build a formatted list of commented runtime parameters for a particular problem. This information is generated in the file setup_params in the object directory.
VARIABLE name [``TYPE:`` vartype] [eosmap-spec]

Registers variable with the framework with name name and a variable type defined by vartype. The setup script collects variables from all the included units, and creates a comprehensive list with no duplications. It then assigns defined constants to each variable and calculates the amount of storage required in the data structures for storing these variables. The defined constants and the calculated sizes are written to the file Simulation.h.

The possible types for vartype are as follows:
- PER_VOLUME
  
  This solution variable is represented in conserved form, i.e., it represents the density of a conserved extensive quantity. The prime example is a variable directly representing mass density. Energy densities, momentum densities, and partial mass densities would be other examples (but these quantities are usually represented in PER_MASS form instead).
- PER_MASS
  
  This solution variable is represented in mass-specific form, i.e., it represents quantities whose nature is \(\hbox{extensive quantity}\,\mathop{\mathrm{per}}\,\hbox{mass unit}\). Examples are specific energies, velocities of material (since they are equal to momentum per mass unit), and abundances or mass fractions (partial density divided by density).
- GENERIC
  
  This is the default vartype and need not be specified. This type should be used for any variables that do not clearly belong to one of the previous two categories.
In the current version of the code, the TYPE attribute is only used to determine which variables should be converted to conservative form for certain Grid operations that may require interpolation (i.e., prolongation, guardcell filling, and restriction) when one of the runtime parameters Grid/convertToConsvdForMeshCalls or Grid/convertToConsvdInMeshInterp is set true. Only variables of type PER_MASS are converted: values are multiplied cell-by-cell with the value of the "dens" variable, and potential interpolation results are converted back by cell-by-cell division by "dens" values after interpolation.

Note that therefore
- variable types are irrelevant for uniform grids,
- variable types are irrelevant if neither Grid/convertToConsvdForMeshCalls nor Grid/convertToConsvdInMeshInterp is true, and
- variable types (and conversion to and from conserved form) only take effect if a
  
  VARIABLE dens …
  
  exists.
An eosmap-spec has the syntax ``EOSMAP:`` eos-role \(|\) ( [``EOSMAPIN:`` eos-role] [``EOSMAPOUT:`` eos-role ]), where eos-role stands for a role as defined in Eos_map.h. These roles are used within implementations of the physics/Eos/Eos_wrapped interface, via the subroutines physics/Eos/Eos_getData and physics/Eos/Eos_putData, to map variables from Grid data structures to the eosData array that physics/Eos/Eos understands, and back. For example,

VARIABLE eint TYPE: PER_MASS EOSMAPIN: EINT

means that within Eos_wrapped, the EINT_VAR component of unk will be treated as the grid variable in the “internal energy” role for the purpose of constructing input to physics/Eos/Eos, and

VARIABLE gamc EOSMAPOUT: GAMC

means that within Eos_wrapped, the GAMC_VAR component of unk will be treated as the grid variable in the EOSMAP_GAMC role for the purpose of returning results from calling physics/Eos/Eos to the grid. The specification

VARIABLE pres EOSMAP: PRES

has the same effect as

VARIABLE pres EOSMAPIN: PRES EOSMAPOUT: PRES

Note that not all roles defined in Eos_map.h are necessarily meaningful or actually used in a given Eos implementation. An eosmap-spec for a VARIABLE is only used in an physics/Eos/Eos_wrapped invocation when the optional gridDataStruct argument is absent or has a value of CENTER.
FACEVAR name [eosmap-spec]

This keyword has the same meaning for face-centered variables, that VARIABLE does for cell-centered variables. It allocates space in the grid data structure that contains face-centered physical variables for “name”.

For eosmap-spec, see above under VARIABLE. An eosmap-spec for FACEVAR is only used when physics/Eos/Eos_wrapped is called with an optional gridDataStruct argument of FACEX, FACEY, or FACEZ.
FLUX name

Registers flux variable name with the framework. When using an adaptive mesh, flux conservation is needed at fine-coarse boundaries. Each state variable that needs flux conservation is made known to the framework through this keyword.
MASS_SCALAR name [RENORM: group-name] [eosmap-spec]

If a quantity is defined with keyword MASS_SCALAR, space is created for it in the grid “unk” data structure. It is treated like any other variable by PARAMESH, but the hydrodynamic unit treats it differently. It is advected, but other physical characteristics don’t apply to it. If the optional “RENORM” is given, this mass-scalar will be added to the renormalization group of the accompanying group name. The hydrodynamic solver will renormalize all mass-scalars in a given group, ensuring that all variables in that group will sum to 1 within an individual cell. See

For eosmap-spec, see above under VARIABLE. An eosmap-spec for a MASS_SCALAR may be used in an physics/Eos/Eos_wrapped invocation when the optional gridDataStruct argument is absent or has a value of CENTER.

It is inadvisable to name variables, species, and mass scalars with the same prefix, as post-processing routines have difficulty deciphering the type of data from the output files. For example, don’t create a variable “temp” to hold temperature and a mass scalar “temp” indicating a temporary variable. Although the Simulation.h file can distinguish between these two types of variables, many plotting routines cannot.
PARTICLETYPE particle-type INITMETHOD initialization-method MAPMETHOD map-method ADVMETHOD time-advance-method

This keyword associates a particle type with mapping and initialization sub-units of Particles unit to operate on this particle type during the simulation. Here, map-method describes the method used to map the particle properties to and from the mesh (see ), initialization-method describes the method used to distribute the particles at initialization, and time-advance-method describes the method used to advance the associated particle type in time. This keyword has been introduced to facilitate inclusion of multiple particle types in the same simulation. It imposes certain requirements on the use of the ParticlesMapping and ParticlesInitialization subunits. Particles (of any type, whether called passive or anything else) do not have default methods for initialization, mapping, or time integration, so a PARTICLETYPE directive in a Config file (or an equivalent -particlemethods= setup option, see ) is the only way to specify the appropriate implementations of the Particles subunits to be used. The declaration should be accompanied by appropriate “REQUESTS” or “REQUIRES” directives to specify the paths of the appropriate subunit implementation directories to be included. For clarity, our technique has been to include this information in the simulation directory Config files only. All the currently available mapping and initialization methods have a corresponding identifier in the form of preprocessor definition in Particles.h. The user may select any particle-type name, but the map-method, initialization-method and time-advance-method must correspond to existing identifiers defined in Particles.h. This is necessary to navigate the data structure that stores the particle type and its associated mapping and initialization methods. Users desirous of adding new methods for mapping or initialization should also update the Particles.h file with additional identifiers and their preprocessor definitions. Note, it is possible to use the same methods for different particle types, but each particle type name must only appear once. Finally, the Simulations Config file is also expected to request appropriate components of mapping and initialization subunits using the keyword REQUESTS, since the corresponding Config files do not specify a default component to include. For example, to include passive particle types with Quadratic mapping, Lattice initialization,and Euler for advancing in time the following code segment should appear in the Config file of the Simulations directory.

PARTICLETYPE passive INITMETHOD lattice MAPMETHOD quadratic ADVMETHOD Euler REQUIRES Particles/ParticlesMain REQUESTS Particles/ParticlesMain/passive/Euler REQUESTS Particles/ParticlesMapping/Quadratic REQUESTS Particles/ParticlesInitialization/Lattice
PARTICLEPROP name type

This keyword indicates that the particles data structure will allocate space for a sub-variable “NAME_PART_PROP.” For example if the Config file contains

PARTICLEPROP dens

then the code can directly access this property as

particles(DENS_PART_PROP,1:localNumParticles) = densInitial

type may be REAL or INT, however INT is presently unused. See for more information and examples.
PARTICLEMAP TO partname FROM vartype varname

This keyword maps the value of the particle property partname to the variable varname. vartype can take the values VARIABLE, MASS_SCALAR, SPECIES, FACEX, FACEY, FACEZ, or one of SCRATCH types (SCRATCHVAR/ SCRATCHCENTERVAR, SCRATCHFACEXVAR. SCRATCHFACEYVAR, SCRATCHFACEZVAR) These maps are used to generate Simulation_mapParticlesVar, which takes the particle property partname and returns varname and vartype. For example, to have a particle property tracing density:

PARTICLEPROP dens REAL PARTICLEMAP TO dens FROM VARIABLE dens
SPECIES name [TO number of ions]

An application that uses multiple species uses this keyword to define them. See for more information. The user may also specify an optional number of ions for each element, name. For example, SPECIES o TO 8 creates 9 spaces in unk for Oxygen, that is, a single space for Oxygen and 8 spaces for each of its ions. This is relevant to simulations using the ionize unit. (Omitting the optional TO specifier is equivalent to specifying TO 0).
DATAFILES wildcard

Declares that all files matching the given wildcard in the unit directory should be copied over to the object directory. For example,

DATAFILES *.dat

will copy all the “.dat” files to the object directory.
KERNEL [subdir]

Declares that all subdirectories must be recursively included. This usually marks the end of the high level architecture of a unit. Directories below it may be third party software or a highly optimized solver, and are therefore not required to conform to Flash-X architecture.

Without a subdir, the current directory (i.e., the one containing the Config file with the KERNEL keyword) is marked as a kernel directory, so code from all its subdirectories (with the exception of subdirectories whose name begins with a dot) is included. When a subdir is given, then that subdirectory must exist, and it is treated as a kernel directory in the same way.

Note that currently the setup script can process only one KERNEL directive per Config file.
LIBRARY name

Specifies a library requirement. Different Flash-X units require different libraries, and they must inform setup so it can link the libraries into the executable. Some valid library names are HDF5, MPI. Support for external libraries can be added by modifying the site-specific Makefile.h files to include appropriate Makefile macros. It is possible to use internal libraries, as well as switch libraries at setup time. To use these features, see
LINKIF filename componentname

Specifies that the file filename should be used only when the compenent componentname is included. This keyword allows a unit to have multiple implementations of any part of its functionality, even down to the kernel level, without the necessity of creating children for every alternative. This is especially useful in Simulation setups where users may want to use different implementations of specific functions based upon the units included. For instance, a user may wish to supply his/her own implementation of Grid_markRefineDerefine.F90, instead of using the default one provided by Flash-X. However, this function is aware of the internal workings of Grid, and has different implementations for different grid packages. The user could therefore specify different versions of his/her own file that are intended for use with the different grids. For example, adding

LINKIF Grid_markRefineDerefine.F90.ug Grid/GridMain/UG LINKIF Grid_markRefineDerefine.F90.pmesh Grid/GridMain/paramesh

to the Config file ensures that if the application is built with UG, the file Grid_markRefineDerefine.F90.ug will be linked in as Grid_markRefineDerefine.F90, whereas if it is built with |paramesh|2 or |paramesh|4.0 or |paramesh|4dev, then the file Grid_markRefineDerefine.F90.pmesh will be linked in as Grid_markRefineDerefine.F90. Alternatively, the user may want to provide only one implementation specific to, say, PARAMESH. In this case, adding

LINKIF Grid_markRefineDerefine.F90 Grid/GridMain/paramesh

to the Config file ensures that the user-supplied file is included when using PARAMESH(either version), while the default Flash-X file is included when using UG.
PPDEFINE sym1 sym2 …

Instructs setup to add the PreProcessor symbols SYM1 and SYM2 to the generated Simulation.h. Here SYM1 is sym1 converted to uppercase. These pre-process symbols can be used in the code to distinguish between which units have been used in an application. For example, a Fortran subroutine could include

#ifdef |flashx|_GRID_UG ug specific code #endif

#ifdef |flashx|_GRID_PARAMESH3OR4 pm3+ specific code #endif

By convention, many preprocessor symbols defined in Config files included in the Flash-X code distribution start with the prefix “Flash-X_”.
USESETUPVARS var1, var2, …

This tells setup that the specified “Setup Variables” are being used in this Config file. The variables initialize to an empty string if no values are specified for them. Note that commas are required if listing several variables.
CHILDORDER child1 child2 …

When setup links several implementations of the same function, it ensures that implementations of children override that of the parent. Its method is to lexicographically sort all the names and allow implementations occurring later to override those occurring earlier. This means that if two siblings implement the same code, the names of the siblings determine which implementation wins. Although it is very rare for two siblings to implement the same function, it does occur. This keyword permits the Config file to override the lexicographic order by one preferred by the user. Lexicographic ordering will prevail as usual when deciding among implementations that are not explicitly listed.
GUARDCELLS num

Allows an application to choose the stencil size for updating grid points. The stencil determines the number of guardcells needed. The PPM algorithm requires \(4\) guardcells, hence that is the default value. If an application specifies a smaller value, it will probably not be able to use the default monotonic AMR Grid interpolation; see the -gridinterpolation setup flag for additional information.
SETUPERROR error message

This causes setup to abort with the specified error message. This is usually used only inside a conditional IF/ENDIF block (see below).
IF, ELSEIF, ELSE, ENDIF

A conditional block is of the following form:

IF cond … ELSEIF cond … ELSE … ENDIF

where the ELSEIF and ELSE blocks are optional. There is no limit on the number of ELSEIF blocks. “…” is any sequence of valid Config file syntax. The conditional blocks may be nested. “cond” is any boolean valued Python expression using the setup variables specified in the USESETUPVARS.

The Setup Tool

The setup tool parses and interprets the Config files to assemble an application instance in the specified object directory. It has three primary actions to perform.

Compile a list of source files to be included in the application. This compilation process ends with all the needed files assembled in the object directory. Some may be linked directly from their location in the source tree, while others may have been generated through some additional actions taken by the setup tool described in Figure …

Compile and initialize a list of runtime parameters needed by the simulation

Generate Makefile.Unit for each unit that will be included in the main Makefile used for compiling the source code. Note that the Flash-X build system assumes GNU Make.

The setup tool traverses the Flash-X source tree starting from the directory hosting the specific application definition. This starting directory is essentially the selected implementation of the “Simulation” unit. It accumulates units, subunits and other compoments from where source code is to be collected through recursive traversal of all encountered dependencies. While traversing the source tree the setup tool uses the following inheritance rules to arbitrate on versions of the source code functions, versions of key definitions and initial values of runtime parameters.

All files with valid source extensions (such as .F90, .c, .h) are added to the list and are linked into the object directory
- If a file with identical name in already included in the list the existing link in the object directory is removed and replaced with a link to the newly encountered file
- The source files in the Simulation unit are the last ones to be added to the list, and therefore can override any file from the source tree in the object directory
If a file has a .ini extension it indicates to the setup tool that it contains definitions for the keys. All the keys defined in the file are added to the list of available keys.
- If a key already existed in the list, its defintion is replaced by the most recently encountered definition.
- Similar to source files, a definition in the Simulation unit overrides any existing definition of a key.
An -mc appended to the name of a file indicates to the setup tool that the file is augmented with keys and needs to be translated
- An augmented file foo.F90-mc may have a Novariant directive, in which case the keys are replaced by the corresponding definitions and the emitted code is placed in a file foo.F90 in the object directory. Note that foo.F90 never comes into existence anywhere in the source tree, so foo.F90 in the object directory is a physical file, not a link.
- If the aumented file does not have Novariant directive then the setup tool generates its variants.
  - It looks for paths specified for the variants in the list of REQUIRES.
  - For every variant it temporarily addes the keys defined in the path to list of available keys, which are removed from the list as soon as the translated code is emitted.
  - The emitted code is placed in the object directory with the name foo_thisvariant.F90.
  - It is assumed that the function or subroutine names are defined to be variant dependent in these files. For details on how to do this correctly see …..
  - In Makefile.Unit instances of foo.o are replaced with foo_variants.o for all available variants.
If any file has the Reorder directive it undergoes a code transformation where the order of indices in the arrays is changed as specified. The emitted code is placed in a file with the same name as the original file and the link to the source tree is removed.
Runtime parameters defined in the Config file are added to the list of runtime parameters and initialized with the specified value.
- If a runtime parameter already exists in the list its value is replaced with the most recently encountered value
- The Simulation unit is the last one to be processed, therefore initial values specified in its Config file override other values.
The setup script determines site-dependent configuration information

by looking for a directory sites/<hostname> where <hostname> is the hostname of the machine on which Flash-X is running. [1] Failing this, it looks in sites/Prototypes/ for a directory with the same name as the output of the uname command. The site and operating system type can be overridden with the -site and -ostype command-line options to the setup command. Only one of these options can be used at one time. The directory for each site and operating system type contains a makefile fragment Makefile.h that sets command names, compiler flags, library paths, and any replacement or additional source files needed to compile Flash-X for that specific machine and machine type.

Setup Arguments

The setup script accepts a large number of command line arguments which affect the simulation in various ways. These arguments are divided into three categories:

Setup Options (example: -auto) begin with a dash and are built into the setup script itself. Many of the most commonly used arguments are setup options.
Setup Variables (example: species=air,h2o) are defined by individual units. When writing a Config file for any unit, you can define a setup variable. explains how setup variables can be created and used.
Setup Shortcuts (example: +ug) begin with a plus symbol and are essentially macros which automatically include a set of setup variables and/or setup options. New setup shortcuts can be easily defined, see for more information.

-verbose= instructs the level of verbosity in progress messages printed by the setup tool. Different levels (in order of increasing verbosity) are ERROR, IMPINFO, WARN, INFO, DEBUG}. The default is WARN.

-auto indicates to the setup tool that it should select DEFAULT in every traversed path unless an alternative is explicitly specified either at commandline or in one of the already traversed Config files. In the process a text file containing a list of all traversed paths is created and is placed in the object directory.

*-[123]d specifies the dimensionality of the simulation. The default is 2d.

-maxblocks= is relevant only when using PARAMESH. It lets the AMR know how many blocks to allocate for the state variables. Note that if the number of blocks generated on a process exceeds maxblocks the execution will abort.

*-nxb=/ -nyb= / -nzb= * specify the number of data points (also called cells) along each dimension of the every block in the setup

-debug /-opt /-test are options used to select the level of optimization to used by the compilers. The defauls is -opt. The Makefile.h in the site directory defines the flags associated with each of these options, and the users can customize them as they want.

-objdir=<dir> allows the user to specify the path where they wish the executable to be built. The default is object at the same level as the source directory.

-with-unit=<path> is used to specify to the setup tool that the source code at the specified path is to be included. The specification of the path on commandline can override any dependency in the traversal that had a REQUESTS associated with it. And it can add dependencies that were not included with -auto option.

-without-unit=<path> is used to tell the setup tool not to include the code in the specified path. This option can also override REQUIRES encountered during the traversal.

-geometry=<geometry> is used to specify one of the supported geometries <cartesian, cylindrical, spherical, polar>, the default is cartesian.

-defines=<def>/ causes the specified pre-processor symbols to be defined when the code is being compiled. This is useful for code segments surrounded by preprocessor directives that are to be included only if a particular implementation of a unit is selected.

-nofbs non-fixed-block size mode where nxb, nyb and nzb are not fixed at compile time.

-gridinterpolation=<scheme> is used to select a scheme for Grid interpolation.

-makefile=<extension>} is used when a site can work with more than one compiler suite. In such situations the site can have several Makefile.h’s named as Makefile.h.<extension> and the one specified with this option will be selected

-index-reorder instructs setup order the indices in state variables as specified. This feature is needed because the base data structure in PARAMESH expects the variable index to be first while AMReX needs it to be last. This feature permits the same code base to work in both modes.

-parfile=<filename> causes setup to copy the specified runtime-parameters file in the simulation directory to the object directory and name it flash.par. By default every setup distributed has a flash.par which is copied into the object directory if this option is not used.

*-append-parfiles=[location1]<filename1>,[location2]<filename2> … takes a comma-separated list of names of parameter files and combines them into one flash.par file in the object directory. File names without an absolute path are taken to be relative to the simulation directory, in a way similat to that for the -parfile option.

-portable this option causes setup copy instead of linking source files.

-site=<site> specifies the suddirectory of the “sites” directory from where to fetch Makefile.h

-with-library=<libname>[,args] instructs setup to link in the specified library when building the final executable. The libraty in questoin can be internal or external. If external the user must make sure that the appropriate path to the library is provided in their site’s Makefile.h. An internal library will be built by the setup if it hasn’t already been done so by an earlier invocation.

-without-library=<libname> is used to override the need for a library specified by one the Config files traversed by the setup tool.

-mconly forces the setup tool to only run the macro processor. Processed files are saved to the object directory.

Using Shortcuts

Apart from the various setup options the setup script also allows you to use shortcuts for frequently used combinations of options. For example, instead of typing in

./setup -a Sod -with-unit=Grid/GridMain/UG

you can just type

./setup -a Sod +ug

The +ug or any setup option starting with a ‘+’ is considered as a shortcut. By default, setup looks at bin/setup_shortcuts.txt for a list of declared shortcuts. You can also specify a “:” delimited list of files in the environment variable SETUP_SHORTCUTS and setup will read all the files specified (and ignore those which don’t exist) for shortcut declarations. See for an example file.

# comment line

# each line is of the form # shortcut:arg1:arg2:…: # These shortcuts can refer to each other.

default:–with-library=mpi:-unit=IO/IOMain:-gridinterpolation=monotonic

# io choices noio:–without-unit=IO/IOMain: io:–with-unit=IO/IOMain:

# frequently used geometries cube64:-nxb=64:-nyb=64:-nzb=64:

The shortcuts are replaced by their expansions in place, so options which come after the shortcut override (or conflict with) options implied by the shortcut. A shortcut can also refer to other shortcuts as long as there are no cyclic references.

The “default” shortcut is special. setup always prepends +default to its command line thus making ./setup -a Sod equivalent to ./setup +default -a Sod. Thus changing the default IO to “hdf5/parallel”, is as simple as changing the definition of the “default” shortcut.

Some of the more commonly used shortcuts are described below:

Shortcuts for often-used options
Shortcut	Description
+cartesian	use cartesian geometry
+cylindrical	use cylindrical geometry
+noio	omit IO
+nolog	omit logging
+pm4dev	use the PARAMESH4DEV grid
+polar	use polar geometry
+spherical	use spherical geometry
+ug	use the uniform grid in a fixed block size mode
+nofbs	use the uniform grid in a non-fixed block size mode

Setup Variables and Preprocessing `Config` Files

setup allows you to assign values to “Setup Variables”. These variables can be string-valued, integer-valued, or boolean. A setup call like

./setup -a Sod Foo=Bar Baz=True

sets the variable “Foo” to string “Bar” and “Baz” to boolean True [2]. setup can conditionally include and exclude parts of the Config file it reads based on the values of these variables. For example, the IO/IOMain/hdf5/Config file contains

DEFAULT serial

USESETUPVARS parallelIO

IF parallelIO DEFAULT parallel ENDIF

The code sets IO to its default value of “serial” and then resets it to “parallel” if the setup variable “parallelIO” is True. The USESETUPVARS keyword in the Config file instructs setup that the specified variables must be defined; undefined variables will be set to the empty string.

Through judicious use of setup variables, the user can ensure that specific implementations are included or the simulation is properly configured. For example, the setup line ./setup -a Sod +ug expands to ./setup -a Sod -unit=Grid/GridMain/ Grid=UG. The relevant part of the Grid/GridMain/Config file is given below:

# Requires use of the Grid SetupVariable USESETUPVARS Grid

DEFAULT paramesh

IF Grid==’UG’ DEFAULT UG ENDIF IF Grid==’PM2’ DEFAULT paramesh/|paramesh|2 ENDIF

The Grid/GridMain/Config file defaults to choosing PARAMESH. But when the setup variable Grid is set to “UG” through the shortcut +ug, the default implementation is set to “UG”. The same technique is used to ensure that the right IO unit is automatically included.

See bin/Readme.SetupVars for an exhaustive list of Setup Variables which are used in the various Config files. For example the setup variable nDim can be test to ensure that a simulation is configured with the appropriate dimensionality (see for example Simulation/SimulationMain/unitTest/Eos/Config).

Creating a Site-specific `Makefile`

If setup does not find your hostname in the sites/ directory it picks a default Makefile based on the operating system. This Makefile is not always correct but can be used as a template to create a Makefile for your machine. To create a Makefile specific to your system follow these instructions.

Create the directory sites/<hostname>, where <hostname> is the hostname of your machine.
Start by copying os/<your os>/Makefile.h to sites/<hostname>
Use bin/suggestMakefile.sh to help identify the locations of various libraries on your system. The script scans your system and displays the locations of some libraries. You must note the location of MPI library as well. If your compiler is actually an mpi-wrapper (e.g.mpif90), you must still define LIB_MPI in your site specific Makefile.h as the empty string.
Edit sites/<hostname>/Makefile.h to provide the locations of various libraries on your system.
Edit sites/<hostname>/Makefile.h to specify the FORTRAN and C compilers to be used.

The Makefile.h must include a compiler flag to promote Fortran Reals to Double Precision. Flash-X performs all MPI communication of Fortran Reals using MPI_DOUBLE_PRECISION type, and assumes that Fortran Reals are interoperable with C doubles in the I/O unit.

Files Created During the `setup` Process

The setup tool generates many files in the directory. They fall into three major categories:

Files not required to build the Flash-X executable, but which contain useful information,
Generated or C code, and
Makefiles required to compile the Flash-X executable.

Informational files

These files are generated before compilation by setup. Each of these files begins with the prefix setup_ for easy identification.

`setup_call`	contains the options with which `setup` was called and the command line resulting after shortcut expansion
`setup_libraries`	contains the list of libraries and their arguments used to generate the executable
`setup_units`	contains the list of all units which were included in the current setup
`setup_defines`	contains a list of all pre-process symbols passed to the compiler invocation directly
`setup_flags`	contains the exact compiler and linker flags
`setup_params`	contains the list of runtime parameters defined in the `Config` files processed by `setup`
`setup_vars`	contains the list of variables, fluxes with their descriptions.

Code generated by the `setup` call

These routines are generated by the setup call and provide simulation-specific code.

`setup_buildstamp.F90`	contains code for the subroutine `setup_buildstamp` which returns the setup and build time as well as code for `setup_systemInfo` which returns the uname of the system used to setup the problem
`setup_call.F90`	contains code which returns build statistics including the actual `setup` call as well as the compiler flags used for the build
`setup_getFlashUnits.F90`	contains code to retrieve the number and list of flashUnits used to compile code
	contains code to retrieve the version of Flash-X used for the build
`Simulation.h`	contains simulation specific preprocessor macros, which change based upon setup unlike `constants.h`. It is described in
	contains code to map an index described in `Simulation.h` to a string described in the `Config` file.
`Simulation/Si mulation_mapStrToInt.F90`	contains code to map a string described in the `Config` file to an integer index described in the `Simulation.h` file.
	contains a mapping between particle properties and grid variables. Only generated when particles are included in a simulation.
`Particles/Part icles_specifyMethods.F90`	contains code to make a data structure with information about the mapping and initialization method for each type of particle. Only generated when particles are included in a simulation.

Makefiles generated by `setup`

Apart from the master Makefile, setup generates a makefile for each unit, which is “included” in the master Makefile. This is true even if the unit is not included in the application. These unit makefiles are named Makefile.Unit and are a concatenation of all the Makefiles found in unit hierarchy processed by setup.

For example, if an application uses Grid/GridMain/paramesh/paramesh4/|paramesh|4.0, the file Makefile.Grid will be a concatenation of the Makefiles found in

Grid,
Grid/GridMain,
Grid/GridMain/paramesh,
Grid/GridMain/paramesh/paramesh4, and
Grid/GridMain/paramesh/paramesh4/|paramesh|4.0

As another example, if an application does not use PhysicalConstants, then Makefile.PhysicalConstants is just the contents of PhysicalConstants/Makefile at the API level.

Since the order of concatenation is arbitrary, the behavior of the Makefiles should not depend on the order in which they have been concatenated. The makefiles inside the units contain lines of the form:

Unit += file1.o file2.o …

where Unit is the name of the unit, which was Grid in the example above. Dependency on data modules files need not be specified since the setup process determines this requirement automatically.

Setup a hybrid MPI+|openmp| Flash-X application

The Flash-X multithreading requires a MPI-2 installation built with thread support (building with an MPI-1 installation or an MPI-2 installation without thread support is possible but strongly discouraged). The Flash-X application requests the thread support level MPI_THREAD_SERIALIZED to ensure that the MPI library is thread-safe and that any OpenMP thread can call MPI functions safely. You should also make sure that your compiler provides a version of OpenMP which is compliant with at least the OpenMP 2.5 (200505) standard (older versions may also work but I have not checked).

In order to make use of the multithreaded code you must setup your application with one of the setup variables threadBlockList, threadWithinBlock or threadRayTrace equal to True, e.g.

./setup Sedov -auto threadBlockList=True ./setup Sedov -auto threadBlockList=True +mpi1 (compatible with MPI-1 - unsafe!)

When you do this the setup script will insert USEOPENMP = 1 instead of USEOPENMP = 0 in the generated Makefile. If it is equal to \(1\) the Makefile will prepend an OpenMP variable to the FFLAGS, CFLAGS, LFLAGS variables.

In general you should not define FLAGS, CFLAGS and LFLAGS in your Makefile.h. It is much better to define FFLAGS_OPT, FFLAGS_TEST, FFLAGS_DEBUG, CFLAGS_OPT, CFLAGS_TEST, CFLAGS_DEBUG, LFLAGS_OPT, LFLAGS_TEST and LFLAGS_DEBUG in your Makefile.h. The setup script will then initialize the FFLAGS, CFLAGS and LFLAGS variables in the Makefile appropriately for an optimized, test or debug build.

The OpenMP variables should be defined in your Makefile.h and contain a compiler flag to recognize OpenMP directives. In most cases it is sufficient to define a single variable named OPENMP, but you may encounter special situations when you need to define OPENMP_FORTRAN, OPENMP_C and OPENMP_LINK. If you want to build Flash-X with the GNU Fortran compiler gfortran and the GNU C compiler gcc then your Makefile.h should contain

OPENMP = -fopenmp

If you want to do something more complicated like build Flash-X with the Lahey Fortran compiler lf90 and the GNU C compiler gcc then your Makefile.h should contain

OPENMP_FORTRAN = –openmp -Kpureomp OPENMP_C = -fopenmp OPENMP_LINK = –openmp -Kpureomp

When you run the hybrid Flash-X application it will print the level of thread support provided by the MPI library and the number of OpenMP threads in each parallel region

: Called MPI_Init_thread - requested level 2, given level 2 [Driver_initParallel]: Number of OpenMP threads in each parallel region 4

Note that the Flash-X application will still run if the MPI library does not provide the requested level of thread support, but will print a warning message alerting you to an unsafe level of MPI thread support. There is no guarantee that the program will work! I strongly recommend that you stop using this Flash-X application - you should build a MPI-2 library with thread support and then rebuild Flash-X.

We record extra version and runtime information in the Flash-X log file for a threaded application.

You should not setup a Flash-X application with both threadBlockList and threadWithinBlock equal to True - nested OpenMP parallelism is not supported. For further information about Flash-X multithreaded applications please refer to Chapter [Chp:Multithreaded|flashx|].