.. include:: defs.h .. _`Chp:The |flashx| configuration `: The |flashx| configuration ============================================ The configuration toolchain effectively implements the Flash-X software architecture. The encapsulation and inheritance of the code is implemented and enforced by this tool. It relies upon Flash-X's domain specific configuration language *(DSCL)* that encodes meta-information about the components in the accompanying *Config* files. A Config file is unique to a unix directory and has all the meta-information about files in that directory. It can also include information about how to traverse subdirectories of the corresponding component. The Config files are parsed by the *setup* tool. Config files come in two syntactic flavors: static text and python. In static mode, configuration keywords are listed as lines in a plain text file. This mode is the most readable and intuitive of the two, but it lacks flexibility. The python mode has been introduced to circumvent this inflexibility by allowing the configuration file author to specify the configuration directives as a function of the setup variables with a python procedure. This allows the content of each directive and the number of directives in total to be amenable to general programming. The rule the setup script uses for deciding which flavor of configuration file it’s dealing with is simple. Python configuration files have as their first line ``##python:genLines``. If the first line does not match this string, then static mode is assumed and each line of the file is interpreted verbatim as a directive. If python mode is triggered, then the entire file is considered as valid python source code (as if it were a .py). From this python code, a function of the form ``def genLines(setupvars)`` is located and executed to generate the configuration directives as an array (or any iterable collection) of strings. The sole argument to genLines is a dictionary that maps setup variable names to their corresponding string values. As an example, here is a configuration file in python mode that registers runtime parameters named indexed_parameter_x where x ranges from 1 to NP and NP is a setup line variable. .. container:: fcodeseg ##python:genLines # We define genLines as a generator with the very friendly "yield" syntax. # Alternatively, we could have genLines return an array of strings or even # one huge multiline string. def genLines(setupvars): # emit some directives that dont depend on any setup variables yield """ REQUIRES Driver REQUIRES physics/Hydro REQUIRES physics/Eos """ # read a setup variable value from the dictionary np = int(setupvars("NP")) # must be converted from a string # loop from 0 to np-1 for x in xrange(np): yield "PARAMETER indexed_parameter_%d REAL 0." % (x+1) When setting up a problem with NP=5 on the setup command line, the following directives will be processed: .. container:: fcodeseg REQUIRES Driver REQUIRES physics/Hydro REQUIRES physics/Eos PARAMETER indexed_parameter_1 REAL 0. PARAMETER indexed_parameter_2 REAL 0. PARAMETER indexed_parameter_3 REAL 0. PARAMETER indexed_parameter_4 REAL 0. PARAMETER indexed_parameter_5 REAL 0. .. _`Sec: DSCL Keywords`: Config file Keywords and syntax ~~~~~~~~~~~~~~~~~~~~~~~~ The syntax of the configuration directives is described here. Arbitrarily many spaces and/or tabs may be used, but all keywords must be in uppercase. Lines not matching an ad missible pattern will raise an error when running setup. The syntax of DSCL includes two types of keywords. The directive keywords define actions to be taken by the setup tool, while the non directive keywords encode information about the directives. The non directive keywords are associated with specific directives keyworks. - | ``# comment`` | A comment. Can appear as a separate line or at the end of a line. - | ``DEFAULT`` *sub-component* | Any component can designate one component (which is effectively a subdirectory) to be the “default”, with this keyword. If no specific sub-component is selected by the application, the designated default implementation gets included. - | ``EXCLUSIVE`` *component...* | Specifies a list of components that cannot be included together. - | ``CONFLICTS`` *unit1[/sub-unit[/component...]]* ... | Specifies that the current component is not compatible with the list of components that follow. ``setup`` issues an error if the user attempts a conflicting configuration. - | ``REQUIRES`` *unit[/sub-unit[/component...]] [* ``OR`` *unit[/sub-unit...]]...* | Specifies a component that is required. The component can be specified with partially specified or full specified path. If partial path is specified then full path is generated by following the "DEFAULT" options of the subcomponents. - | ``REQUESTS`` *unit[/sub-unit[/component...]]* | Is similar to REQUIRES except that it is a soft dependency. It can be overridden through either a commandline specification or through dependencies specified in higher priority locations in the source tree. - | ``VARIANTS`` *variantname variantname ...* |Indicates that multiple alternative implemetations of a component are to be included in the setup where each implementation gets its interface and routine/function name appended with the corresponding variant name. For example, one may want to use both CPU and GPU to compute. A component that has valid implementations for both can specify variants as shown below. .. container:: codeseg VARIANTS CPU GPU Then if the setup options specify inclusion of both implementations, the names of the relevant files and function/subroutine interfaces are modified to have unique names. If, on the other hand, only one of the implementations is to be included, the default is Null and all the names remain unmodified. - | ``SUGGEST`` *component component ...* | Unlike ``REQUIRES``, this keyword suggests that the current component be used along with one of the specified component. The setup script will print details of the suggestions which have been ignored. This is useful in catching inadvertently omitted code components before the run starts, thus avoiding a waste of computing resources. - | ``UNOFFICIAL`` | This keyword suggests that the current component is not fully supported. The setup script will abort if a component with this keyword in included. It can be overridden as follows: .. container:: codeseg --with-unofficial= *component* - | ``PARAMETER`` *name type [``CONSTANT``] default* [*range-spec*] | Specifies a runtime parameter. Parameter names are unique up to 20 characters and may not contain spaces. Admissible types include ``REAL``, ``INTEGER``, ``STRING``, and ``BOOLEAN``. Default values for ``REAL`` and ``INTEGER`` parameters must be valid numbers, or the compilation will fail. Default ``STRING`` values must be enclosed in double quotes (``"``). Default ``BOOLEAN`` values must be ``.true.`` or ``.false.`` to avoid compilation errors. Once defined, runtime parameters are available to the entire code. Optionally, any parameter may be specified with the ``CONSTANT`` attribute (*e.g.*, ``PARAMETER foo REAL CONSTANT 2.2``). If a user attempts to set a constant parameter via the runtime parameter file, an error will occur. The range specification is optional and can be used to specify valid ranges for the parameters. The range specification is allowed only for ``REAL, INTEGER, STRING`` variables and must be enclosed in ’[]’. For a ``STRING`` variable, the range specification is a comma-separated list of strings (enclosed in quotes). For a ``INTEGER, REAL`` variable, the range specification is a comma-separated list of (closed) intervals specified by ``min ... max``, where min and max are the end points of the interval. If min or max is omitted, it is assumed to be :math:`-\infty` and :math:`+\infty` respectively. Finally ``val`` is a shortcut for ``val ... val``. For example .. container:: codeseg PARAMETER pres REAL 1.0 [ 0.1 ... 9.9, 25.0 ... ] PARAMETER coords STRING "polar" ["polar","cylindrical","2d","3d"] indicates that ``pres`` is a REAL variable which is allowed to take values between 0.1 and 9.9 or above 25.0. Similarly ``coords`` is a string variable which can take one of the four specified values. - | ``D`` *parameter-name* *comment* | Any line in a ``Config`` file is considered a parameter comment line if it begins with the token ``D``. The first token after the comment line is taken to be the parameter name. The remaining tokens are taken to be a description of the parameter’s purpose. A token is delineated by one or more white spaces. For example, .. container:: codeseg D SOME_PARAMETER The purpose of this parameter is whatever If the parameter comment requires additional lines, the ``&`` is used to indicate continuation lines. For example, .. container:: codeseg D SOME_PARAMETER The purpose of this parameter is whatever D & This is a second line of description You can also use this to describe other variables, fluxes, species, etc. For example, to describe a species called "xyz", create a comment for the parameter “xyz_species”. In general the name should be followed by an underscore and then by the lower case name of the keyword used to define the name. Parameter comment lines are special because they are used by to build a formatted list of commented runtime parameters for a particular problem. This information is generated in the file ``setup_params`` in the ``object`` directory. - | ``VARIABLE`` *name* *[``TYPE:`` vartype]* *[eosmap-spec]* | Registers variable with the framework with name *name* and a variable type defined by *vartype*. The ``setup`` script collects variables from all the included units, and creates a comprehensive list with no duplications. It then assigns defined constants to each variable and calculates the amount of storage required in the data structures for storing these variables. The defined constants and the calculated sizes are written to the file ``Simulation.h``. The possible types for *vartype* are as follows: - | ``PER_VOLUME`` | This solution variable is represented in **conserved** form, *i.e.*, it represents the density of a conserved extensive quantity. The prime example is a variable directly representing mass density. Energy densities, momentum densities, and partial mass densities would be other examples (but these quantities are usually represented in ``PER_MASS`` form instead). - | ``PER_MASS`` | This solution variable is represented in **mass-specific** form, *i.e.*, it represents quantities whose nature is :math:`\hbox{extensive quantity}\,\mathop{\mathrm{per}}\,\hbox{mass unit}`. Examples are specific energies, velocities of material (since they are equal to momentum per mass unit), and abundances or mass fractions (partial density divided by density). - | ``GENERIC`` | This is the default *vartype* and need not be specified. This type should be used for any variables that do not clearly belong to one of the previous two categories. In the current version of the code, the ``TYPE`` attribute is only used to determine which variables should be converted to conservative form for certain ``Grid`` operations that may require interpolation (*i.e.*, prolongation, guardcell filling, and restriction) when one of the runtime parameters ``Grid/convertToConsvdForMeshCalls`` or ``Grid/convertToConsvdInMeshInterp`` is set ``true``. Only variables of type ``PER_MASS`` are converted: values are multiplied cell-by-cell with the value of the ``"dens"`` variable, and potential interpolation results are converted back by cell-by-cell division by ``"dens"`` values after interpolation. Note that therefore - variable types are irrelevant for uniform grids, - variable types are irrelevant if neither ``Grid/convertToConsvdForMeshCalls`` nor ``Grid/convertToConsvdInMeshInterp`` is ``true``, and - variable types (and conversion to and from conserved form) only take effect if a .. container:: codeseg VARIABLE dens ... exists. An *eosmap-spec* has the syntax *``EOSMAP:`` eos-role* :math:`|` *( [``EOSMAPIN:`` eos-role] [``EOSMAPOUT:`` eos-role ])*, where *eos-role* stands for a **role** as defined in ``Eos_map.h``. These roles are used within implementations of the ``physics/Eos/Eos_wrapped`` interface, via the subroutines ``physics/Eos/Eos_getData`` and ``physics/Eos/Eos_putData``, to map variables from ``Grid`` data structures to the ``eosData`` array that ``physics/Eos/Eos`` understands, and back. For example, .. container:: codeseg VARIABLE eint TYPE: PER_MASS EOSMAPIN: EINT means that within ``Eos_wrapped``, the ``EINT_VAR`` component of ``unk`` will be treated as the grid variable in the “internal energy” role for the purpose of constructing input to ``physics/Eos/Eos``, and .. container:: codeseg VARIABLE gamc EOSMAPOUT: GAMC means that within ``Eos_wrapped``, the ``GAMC_VAR`` component of ``unk`` will be treated as the grid variable in the ``EOSMAP_GAMC`` role for the purpose of returning results from calling ``physics/Eos/Eos`` to the grid. The specification .. container:: codeseg VARIABLE pres EOSMAP: PRES has the same effect as .. container:: codeseg VARIABLE pres EOSMAPIN: PRES EOSMAPOUT: PRES Note that not all roles defined in ``Eos_map.h`` are necessarily meaningful or actually used in a given ``Eos`` implementation. An *eosmap-spec* for a ``VARIABLE`` is only used in an ``physics/Eos/Eos_wrapped`` invocation when the optional ``gridDataStruct`` argument is absent or has a value of ``CENTER``. - | ``FACEVAR`` *name* *[eosmap-spec]* | This keyword has the same meaning for face-centered variables, that ``VARIABLE`` does for cell-centered variables. It allocates space in the grid data structure that contains face-centered physical variables for “name”. For *eosmap-spec*, see above under ``VARIABLE``. An *eosmap-spec* for ``FACEVAR`` is only used when ``physics/Eos/Eos_wrapped`` is called with an optional ``gridDataStruct`` argument of ``FACEX``, ``FACEY``, or ``FACEZ``. - | ``FLUX`` *name* | Registers flux variable *name* with the framework. When using an adaptive mesh, flux conservation is needed at fine-coarse boundaries. Each state variable that needs flux conservation is made known to the framework through this keyword. - | ``MASS_SCALAR`` *name* *[RENORM: group-name]* *[eosmap-spec]* | If a quantity is defined with keyword MASS_SCALAR, space is created for it in the grid “unk” data structure. It is treated like any other variable by ``PARAMESH``, but the hydrodynamic unit treats it differently. It is advected, but other physical characteristics don’t apply to it. If the optional “RENORM” is given, this mass-scalar will be added to the renormalization group of the accompanying group name. The hydrodynamic solver will renormalize all mass-scalars in a given group, ensuring that all variables in that group will sum to 1 within an individual cell. See For *eosmap-spec*, see above under ``VARIABLE``. An *eosmap-spec* for a ``MASS_SCALAR`` may be used in an ``physics/Eos/Eos_wrapped`` invocation when the optional ``gridDataStruct`` argument is absent or has a value of ``CENTER``. .. container:: flashtip It is inadvisable to name variables, species, and mass scalars with the same prefix, as post-processing routines have difficulty deciphering the type of data from the output files. For example, don’t create a variable “temp” to hold temperature and a mass scalar “temp” indicating a temporary variable. Although the ``Simulation.h`` file can distinguish between these two types of variables, many plotting routines cannot. - | ``PARTICLETYPE`` *particle-type* ``INITMETHOD`` *initialization-method* ``MAPMETHOD`` *map-method* ``ADVMETHOD`` *time-advance-method* | This keyword associates a **particle type** with mapping and initialization sub-units of ``Particles`` unit to operate on this particle type during the simulation. Here, *map-method* describes the method used to map the particle properties to and from the mesh (see ), *initialization-method* describes the method used to distribute the particles at initialization, and *time-advance-method* describes the method used to advance the associated particle type in time. This keyword has been introduced to facilitate inclusion of multiple particle types in the same simulation. It imposes certain requirements on the use of the ``ParticlesMapping`` and ``ParticlesInitialization`` subunits. Particles (of any type, whether called ``passive`` or anything else) do not have default methods for initialization, mapping, or time integration, so a ``PARTICLETYPE`` directive in a ``Config`` file (or an equivalent ``-particlemethods=`` setup option, see ) is the only way to specify the appropriate implementations of the ``Particles`` subunits to be used. The declaration should be accompanied by appropriate “REQUESTS” or “REQUIRES” directives to specify the paths of the appropriate subunit implementation directories to be included. For clarity, our technique has been to include this information in the simulation directory ``Config`` files only. All the currently available mapping and initialization methods have a corresponding identifier in the form of preprocessor definition in ``Particles.h``. The user may select any *particle-type* name, but the *map-method*, *initialization-method* and *time-advance-method* must correspond to existing identifiers defined in ``Particles.h``. This is necessary to navigate the data structure that stores the particle type and its associated mapping and initialization methods. Users desirous of adding new methods for mapping or initialization should also update the ``Particles.h`` file with additional identifiers and their preprocessor definitions. Note, it is possible to use the same methods for different particle types, but each particle type name must only appear once. Finally, the Simulations ``Config`` file is also expected to request appropriate components of mapping and initialization subunits using the keyword ``REQUESTS``, since the corresponding Config files do not specify a default component to include. For example, to include ``passive`` particle types with ``Quadratic`` mapping, ``Lattice`` initialization,and ``Euler`` for advancing in time the following code segment should appear in the ``Config`` file of the ``Simulations`` directory. .. container:: codeseg PARTICLETYPE passive INITMETHOD lattice MAPMETHOD quadratic ADVMETHOD Euler REQUIRES Particles/ParticlesMain REQUESTS Particles/ParticlesMain/passive/Euler REQUESTS Particles/ParticlesMapping/Quadratic REQUESTS Particles/ParticlesInitialization/Lattice - | ``PARTICLEPROP`` *name* *type* | This keyword indicates that the particles data structure will allocate space for a sub-variable “NAME_PART_PROP.” For example if the Config file contains .. container:: codeseg PARTICLEPROP dens then the code can directly access this property as .. container:: codeseg particles(DENS_PART_PROP,1:localNumParticles) = densInitial *type* may be REAL or INT, however INT is presently unused. See for more information and examples. - | ``PARTICLEMAP`` TO *partname* FROM *vartype* *varname* | This keyword maps the value of the particle property *partname* to the variable *varname*. *vartype* can take the values VARIABLE, MASS_SCALAR, SPECIES, FACEX, FACEY, FACEZ, or one of SCRATCH types (SCRATCHVAR/ SCRATCHCENTERVAR, SCRATCHFACEXVAR. SCRATCHFACEYVAR, SCRATCHFACEZVAR) These maps are used to generate ``Simulation_mapParticlesVar``, which takes the particle property *partname* and returns *varname* and *vartype*. For example, to have a particle property tracing density: .. container:: codeseg PARTICLEPROP dens REAL PARTICLEMAP TO dens FROM VARIABLE dens - | ``SPECIES`` *name* [TO *number of ions*] | An application that uses multiple species uses this keyword to define them. See for more information. The user may also specify an optional number of ions for each element, *name*. For example, ``SPECIES`` *o* TO *8* creates 9 spaces in ``unk`` for Oxygen, that is, a single space for Oxygen and 8 spaces for each of its ions. This is relevant to simulations using the ``ionize`` unit. (Omitting the optional ``TO`` specifier is equivalent to specifying ``TO`` 0). - | ``DATAFILES`` *wildcard* | Declares that all files matching the given wildcard in the unit directory should be copied over to the object directory. For example, .. container:: codeseg DATAFILES \*.dat will copy all the “.dat” files to the object directory. - | ``KERNEL`` *[subdir]* | Declares that all subdirectories must be recursively included. This usually marks the end of the high level architecture of a unit. Directories below it may be third party software or a highly optimized solver, and are therefore not required to conform to |flashx| architecture. Without a *subdir*, the current directory (*i.e.*, the one containing the ``Config`` file with the ``KERNEL`` keyword) is marked as a kernel directory, so code from all its subdirectories (with the exception of subdirectories whose name begins with a dot) is included. When a *subdir* is given, then that subdirectory must exist, and it is treated as a kernel directory in the same way. Note that currently the ``setup`` script can process only one ``KERNEL`` directive per ``Config`` file. - | ``LIBRARY`` *name* | Specifies a library requirement. Different |flashx| units require different libraries, and they must inform ``setup`` so it can link the libraries into the executable. Some valid library names are ``HDF5, MPI``. Support for external libraries can be added by modifying the site-specific ``Makefile.h`` files to include appropriate Makefile macros. It is possible to use internal libraries, as well as switch libraries at setup time. To use these features, see - | ``LINKIF`` *filename componentname* | Specifies that the file *filename* should be used only when the compenent ``componentname`` is included. This keyword allows a unit to have multiple implementations of any part of its functionality, even down to the kernel level, without the necessity of creating children for every alternative. This is especially useful in Simulation setups where users may want to use different implementations of specific functions based upon the units included. For instance, a user may wish to supply his/her own implementation of ``Grid_markRefineDerefine.F90``, instead of using the default one provided by |flashx|. However, this function is aware of the internal workings of ``Grid``, and has different implementations for different grid packages. The user could therefore specify different versions of his/her own file that are intended for use with the different grids. For example, adding .. container:: codeseg LINKIF Grid_markRefineDerefine.F90.ug Grid/GridMain/UG LINKIF Grid_markRefineDerefine.F90.pmesh Grid/GridMain/paramesh to the ``Config`` file ensures that if the application is built with ``UG``, the file ``Grid_markRefineDerefine.F90.ug`` will be linked in as ``Grid_markRefineDerefine.F90``, whereas if it is built with ``|paramesh|2`` or ``|paramesh|4.0`` or ``|paramesh|4dev``, then the file ``Grid_markRefineDerefine.F90.pmesh`` will be linked in as ``Grid_markRefineDerefine.F90``. Alternatively, the user may want to provide only one implementation specific to, say, ``PARAMESH``. In this case, adding .. container:: codeseg LINKIF Grid_markRefineDerefine.F90 Grid/GridMain/paramesh to the Config file ensures that the user-supplied file is included when using ``PARAMESH``\ (either version), while the default |flashx| file is included when using ``UG``. - | ``PPDEFINE`` *sym1 sym2 ...* | Instructs setup to add the PreProcessor symbols ``SYM1`` and ``SYM2`` to the generated ``Simulation.h``. Here ``SYM1`` is *sym1* converted to uppercase. These pre-process symbols can be used in the code to distinguish between which units have been used in an application. For example, a Fortran subroutine could include .. container:: codeseg #ifdef |flashx|_GRID_UG ug specific code #endif #ifdef |flashx|_GRID_PARAMESH3OR4 pm3+ specific code #endif By convention, many preprocessor symbols defined in Config files included in the |flashx| code distribution start with the prefix “|flashx|\_”. - | ``USESETUPVARS var1, var2, …`` | This tells ``setup`` that the specified “Setup Variables” are being used in this ``Config`` file. The variables initialize to an empty string if no values are specified for them. Note that commas are required if listing several variables. - | ``CHILDORDER`` *child1 child2 …* | When ``setup`` links several implementations of the same function, it ensures that implementations of children override that of the parent. Its method is to lexicographically sort all the names and allow implementations occurring later to override those occurring earlier. This means that if two siblings implement the same code, the names of the siblings determine which implementation wins. Although it is very rare for two siblings to implement the same function, it does occur. This keyword permits the ``Config`` file to override the lexicographic order by one preferred by the user. Lexicographic ordering will prevail as usual when deciding among implementations that are not explicitly listed. - | ``GUARDCELLS`` *num* | Allows an application to choose the stencil size for updating grid points. The stencil determines the number of guardcells needed. The PPM algorithm requires :math:`4` guardcells, hence that is the default value. If an application specifies a smaller value, it will probably not be able to use the default ``monotonic`` AMR Grid interpolation; see the ``-gridinterpolation`` ``setup`` flag for additional information. - | ``SETUPERROR error message`` | This causes ``setup`` to abort with the specified error message. This is usually used only inside a conditional IF/ENDIF block (see below). - | ``IF, ELSEIF, ELSE, ENDIF`` | A conditional block is of the following form: .. container:: codeseg IF cond ... ELSEIF cond ... ELSE ... ENDIF where the ``ELSEIF`` and ``ELSE`` blocks are optional. There is no limit on the number of ``ELSEIF`` blocks. “...” is any sequence of valid ``Config`` file syntax. The conditional blocks may be nested. “cond” is any boolean valued Python expression using the setup variables specified in the ``USESETUPVARS``. .. _`Sec:The Setup Tool`: The Setup Tool ~~~~~~~~~~~~~~~~~~~~~~~~~ The setup tool parses and interprets the Config files to assemble an application instance in the specified object directory. It has three primary actions to perform. #. Compile a list of source files to be included in the application. This compilation process ends with all the needed files assembled in the object directory. Some may be linked directly from their location in the source tree, while others may have been generated through some additional actions taken by the setup tool described in Figure ... #. Compile and initialize a list of runtime parameters needed by the simulation #. Generate Makefile.Unit for each unit that will be included in the main Makefile used for compiling the source code. Note that the Flash-X build system assumes GNU Make. The setup tool traverses the Flash-X source tree starting from the directory hosting the specific application definition. This starting directory is essentially the selected implementation of the “Simulation” unit. It accumulates units, subunits and other compoments from where source code is to be collected through recursive traversal of all encountered dependencies. While traversing the source tree the setup tool uses the following inheritance rules to arbitrate on versions of the source code functions, versions of key definitions and initial values of runtime parameters. * All files with valid source extensions (such as .F90, .c, .h) are added to the list and are linked into the object directory * If a file with identical name in already included in the list the existing link in the object directory is removed and replaced with a link to the newly encountered file * The source files in the Simulation unit are the last ones to be added to the list, and therefore can override any file from the source tree in the object directory * If a file has a .ini extension it indicates to the setup tool that it contains definitions for the keys. All the keys defined in the file are added to the list of available keys. * If a key already existed in the list, its defintion is replaced by the most recently encountered definition. * Similar to source files, a definition in the Simulation unit overrides any existing definition of a key. * An -mc appended to the name of a file indicates to the setup tool that the file is augmented with keys and needs to be translated * An augmented file foo.F90-mc may have a *Novariant* directive, in which case the keys are replaced by the corresponding definitions and the emitted code is placed in a file foo.F90 in the object directory. Note that foo.F90 never comes into existence anywhere in the source tree, so foo.F90 in the object directory is a physical file, not a link. * If the aumented file does not have *Novariant* directive then the setup tool generates its variants. * It looks for paths specified for the variants in the list of REQUIRES. * For every variant it temporarily addes the keys defined in the path to list of available keys, which are removed from the list as soon as the translated code is emitted. * The emitted code is placed in the object directory with the name foo_thisvariant.F90. * It is assumed that the function or subroutine names are defined to be variant dependent in these files. For details on how to do this correctly see ..... * In Makefile.Unit instances of foo.o are replaced with foo_variants.o for all available variants. * If any file has the *Reorder* directive it undergoes a code transformation where the order of indices in the arrays is changed as specified. The emitted code is placed in a file with the same name as the original file and the link to the source tree is removed. * Runtime parameters defined in the Config file are added to the list of runtime parameters and initialized with the specified value. * If a runtime parameter already exists in the list its value is replaced with the most recently encountered value * The Simulation unit is the last one to be processed, therefore initial values specified in its Config file override other values. The ``setup`` script determines site-dependent configuration information by looking for a directory ``sites/`` where ```` is the hostname of the machine on which |flashx| is running. [1]_ Failing this, it looks in ``sites/Prototypes/`` for a directory with the same name as the output of the ``uname`` command. The site and operating system type can be overridden with the ``-site`` and ``-ostype`` command-line options to the ``setup`` command. Only one of these options can be used at one time. The directory for each site and operating system type contains a makefile fragment ``Makefile.h`` that sets command names, compiler flags, library paths, and any replacement or additional source files needed to compile |flashx| for that specific machine and machine type. .. _`Sec:ListSetupArgs`: Setup Arguments --------------- The setup script accepts a large number of command line arguments which affect the simulation in various ways. These arguments are divided into three categories: #. *Setup Options* (example: ``-auto``) begin with a dash and are built into the setup script itself. Many of the most commonly used arguments are setup options. #. *Setup Variables* (example: ``species=air,h2o``) are defined by individual units. When writing a ``Config`` file for any unit, you can define a setup variable. explains how setup variables can be created and used. #. *Setup Shortcuts* (example: ``+ug``) begin with a plus symbol and are essentially macros which automatically include a set of setup variables and/or setup options. New setup shortcuts can be easily defined, see for more information. *-verbose=* instructs the level of verbosity in progress messages printed by the setup tool. Different levels (in order of increasing verbosity) are ERROR, IMPINFO, WARN, INFO, DEBUG}. The default is WARN. *-auto* indicates to the setup tool that it should select *DEFAULT* in every traversed path unless an alternative is explicitly specified either at commandline or in one of the already traversed Config files. In the process a text file containing a list of all traversed paths is created and is placed in the object directory. *-[123]d specifies the dimensionality of the simulation. The default is 2d. *-maxblocks=* is relevant only when using PARAMESH. It lets the AMR know how many blocks to allocate for the state variables. Note that if the number of blocks generated on a process exceeds maxblocks the execution will abort. *-nxb=/ -nyb= / -nzb= * specify the number of data points (also called cells) along each dimension of the every block in the setup *-debug /-opt /-test* are options used to select the level of optimization to used by the compilers. The defauls is -opt. The Makefile.h in the site directory defines the flags associated with each of these options, and the users can customize them as they want. *-objdir=* allows the user to specify the path where they wish the executable to be built. The default is *object* at the same level as the source directory. *-with-unit=* is used to specify to the setup tool that the source code at the specified path is to be included. The specification of the path on commandline can override any dependency in the traversal that had a *REQUESTS* associated with it. And it can add dependencies that were not included with -auto option. *-without-unit=* is used to tell the setup tool not to include the code in the specified path. This option can also override *REQUIRES* encountered during the traversal. *-geometry=* is used to specify one of the supported geometries , the default is cartesian. *-defines=*/ causes the specified pre-processor symbols to be defined when the code is being compiled. This is useful for code segments surrounded by preprocessor directives that are to be included only if a particular implementation of a unit is selected. *-nofbs* non-fixed-block size mode where nxb, nyb and nzb are not fixed at compile time. *-gridinterpolation=* is used to select a scheme for Grid interpolation. *-makefile=}* is used when a site can work with more than one compiler suite. In such situations the site can have several Makefile.h's named as Makefile.h. and the one specified with this option will be selected *-index-reorder* instructs setup order the indices in state variables as specified. This feature is needed because the base data structure in PARAMESH expects the variable index to be first while AMReX needs it to be last. This feature permits the same code base to work in both modes. *-parfile=* causes setup to copy the specified runtime-parameters file in the simulation directory to the object directory and name it flash.par. By default every setup distributed has a flash.par which is copied into the object directory if this option is not used. *-append-parfiles=[location1],[location2] ... takes a comma-separated list of names of parameter files and combines them into one flash.par file in the object directory. File names without an absolute path are taken to be relative to the simulation directory, in a way similat to that for the -parfile option. *-portable* this option causes setup copy instead of linking source files. *-site=* specifies the suddirectory of the "sites" directory from where to fetch Makefile.h *-with-library=[,args]* instructs setup to link in the specified library when building the final executable. The libraty in questoin can be internal or external. If external the user must make sure that the appropriate path to the library is provided in their site's Makefile.h. An internal library will be built by the setup if it hasn't already been done so by an earlier invocation. *-without-library=* is used to override the need for a library specified by one the Config files traversed by the setup tool. .. _`Sec:SetupShortcuts`: Using Shortcuts --------------- Apart from the various setup options the ``setup`` script also allows you to use shortcuts for frequently used combinations of options. For example, instead of typing in .. container:: codeseg ./setup -a Sod -with-unit=Grid/GridMain/UG you can just type .. container:: codeseg ./setup -a Sod +ug The ``+ug`` or any setup option starting with a ‘+’ is considered as a shortcut. By default, setup looks at ``bin/setup_shortcuts.txt`` for a list of declared shortcuts. You can also specify a ":" delimited list of files in the environment variable ``SETUP_SHORTCUTS`` and ``setup`` will read all the files specified (and ignore those which don’t exist) for shortcut declarations. See for an example file. .. container:: fcodeseg # comment line # each line is of the form # shortcut:arg1:arg2:...: # These shortcuts can refer to each other. default:–with-library=mpi:-unit=IO/IOMain:-gridinterpolation=monotonic # io choices noio:–without-unit=IO/IOMain: io:–with-unit=IO/IOMain: # Choice of Grid ug:-unit=Grid/GridMain/UG: pm2:-unit=Grid/GridMain/paramesh/|paramesh|2: pm40:-unit=Grid/GridMain/paramesh/paramesh4/|paramesh|4.0: pm4dev:-unit=Grid/GridMain/paramesh/paramesh4/|paramesh|4dev: # frequently used geometries cube64:-nxb=64:-nyb=64:-nzb=64: The shortcuts are replaced by their expansions in place, so options which come after the shortcut override (or conflict with) options implied by the shortcut. A shortcut can also refer to other shortcuts as long as there are no cyclic references. The “default" shortcut is special. ``setup`` always prepends ``+default`` to its command line thus making ``./setup -a Sod`` equivalent to ``./setup +default -a Sod``. Thus changing the default IO to “hdf5/parallel", is as simple as changing the definition of the “default" shortcut. Some of the more commonly used shortcuts are described below: .. container:: center .. container:: :name: Tab:setup_shortcuts .. table:: Shortcuts for often-used options ============ =================================================== Shortcut Description ============ =================================================== +cartesian use cartesian geometry +cylindrical use cylindrical geometry +noio omit IO +nolog omit logging +pm4dev use the PARAMESH4DEV grid +polar use polar geometry +spherical use spherical geometry +ug use the uniform grid in a fixed block size mode +nofbs use the uniform grid in a non-fixed block size mode ============ =================================================== .. _`Sec:setupvariables`: Setup Variables and Preprocessing ``Config`` Files -------------------------------------------------- ``setup`` allows you to assign values to “Setup Variables”. These variables can be string-valued, integer-valued, or boolean. A ``setup`` call like .. container:: codeseg ./setup -a Sod Foo=Bar Baz=True sets the variable “Foo" to string “Bar" and “Baz" to boolean True [2]_. ``setup`` can conditionally include and exclude parts of the ``Config`` file it reads based on the values of these variables. For example, the ``IO/IOMain/hdf5/Config`` file contains .. container:: shrink .. container:: fcodeseg DEFAULT serial USESETUPVARS parallelIO IF parallelIO DEFAULT parallel ENDIF The code sets IO to its default value of “serial” and then resets it to “parallel" if the setup variable “parallelIO" is True. The ``USESETUPVARS`` keyword in the ``Config`` file instructs setup that the specified variables must be defined; undefined variables will be set to the empty string. Through judicious use of setup variables, the user can ensure that specific implementations are included or the simulation is properly configured. For example, the setup line ``./setup -a Sod +ug`` expands to ``./setup -a Sod -unit=Grid/GridMain/ Grid=UG``. The relevant part of the ``Grid/GridMain/Config`` file is given below: .. container:: shrink .. container:: fcodeseg # Requires use of the Grid SetupVariable USESETUPVARS Grid DEFAULT paramesh IF Grid==’UG’ DEFAULT UG ENDIF IF Grid==’PM2’ DEFAULT paramesh/|paramesh|2 ENDIF The ``Grid/GridMain/Config`` file defaults to choosing ``PARAMESH``. But when the setup variable Grid is set to “UG" through the shortcut ``+ug``, the default implementation is set to “UG". The same technique is used to ensure that the right IO unit is automatically included. See ``bin/Readme.SetupVars`` for an exhaustive list of Setup Variables which are used in the various Config files. For example the setup variable ``nDim`` can be test to ensure that a simulation is configured with the appropriate dimensionality (see for example ``Simulation/SimulationMain/unitTest/Eos/Config``). .. _`Sec:SetupMakefile`: Creating a Site-specific ``Makefile`` ------------------------------------- If ``setup`` does not find your hostname in the ``sites/`` directory it picks a default ``Makefile`` based on the operating system. This ``Makefile`` is not always correct but can be used as a template to create a ``Makefile`` for your machine. To create a Makefile specific to your system follow these instructions. - Create the directory ``sites/``, where ```` is the hostname of your machine. - Start by copying ``os//Makefile.h`` to ``sites/`` - Use ``bin/suggestMakefile.sh`` to help identify the locations of various libraries on your system. The script scans your system and displays the locations of some libraries. You must note the location of ``MPI`` library as well. If your compiler is actually an mpi-wrapper (*e.g.*\ ``mpif90``), you must still define ``LIB_MPI`` in your site specific ``Makefile.h`` as the empty string. - Edit ``sites//Makefile.h`` to provide the locations of various libraries on your system. - Edit ``sites//Makefile.h`` to specify the FORTRAN and C compilers to be used. .. container:: flashtip The Makefile.h *must* include a compiler flag to promote Fortran ``Reals`` to ``Double Precision``. |flashx| performs all ``MPI`` communication of Fortran ``Reals`` using ``MPI_DOUBLE_PRECISION`` type, and assumes that Fortran ``Reals`` are interoperable with C ``doubles`` in the I/O unit. Files Created During the ``setup`` Process ------------------------------------------ The setup tool generates many files in the directory. They fall into three major categories: - Files not required to build the Flash-X executable, but which contain useful information, - Generated or C code, and - Makefiles required to compile the |flashx| executable. Informational files ~~~~~~~~~~~~~~~~~~~ These files are generated before compilation by ``setup``. Each of these files begins with the prefix ``setup_`` for easy identification. +---------------------+-----------------------------------------------+ | | contains the options with which ``setup`` was | | | called and the command line resulting after | | ``setup_call`` | shortcut expansion | +---------------------+-----------------------------------------------+ | | contains the list of libraries and their | | ``setup_libraries`` | arguments used to generate the executable | +---------------------+-----------------------------------------------+ | | contains the list of all units which were | | ``setup_units`` | included in the current setup | +---------------------+-----------------------------------------------+ | | contains a list of all pre-process symbols | | | passed to the compiler invocation directly | | ``setup_defines`` | | +---------------------+-----------------------------------------------+ | ``setup_flags`` | contains the exact compiler and linker flags | +---------------------+-----------------------------------------------+ | | contains the list of runtime parameters | | | defined in the ``Config`` files processed by | | ``setup_params`` | ``setup`` | +---------------------+-----------------------------------------------+ | | contains the list of variables, fluxes | | ``setup_vars`` | with their descriptions. | +---------------------+-----------------------------------------------+ | | | +---------------------+-----------------------------------------------+ Code generated by the ``setup`` call ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ These routines are generated by the setup call and provide simulation-specific code. +----------------------------------+----------------------------------+ | ``setup_buildstamp.F90`` | contains code for the subroutine | | | ``setup_buildstamp`` which | | | returns the setup and build time | | | as well as code for | | | ``setup_systemInfo`` which | | | returns the *uname* of the | | | system used to setup the problem | +----------------------------------+----------------------------------+ | | contains code which returns | | | build statistics including the | | | actual ``setup`` call as well as | | | the compiler flags used for the | | ``setup_call.F90`` | build | +----------------------------------+----------------------------------+ | | contains code to retrieve the | | | number and list of flashUnits | | ``setup_getFlashUnits.F90`` | used to compile code | +----------------------------------+----------------------------------+ | | contains code to retrieve the | | | version of |flashx| used for the | | | build | +----------------------------------+----------------------------------+ | | contains simulation specific | | | preprocessor macros, which | | ``Simulation.h`` | change based upon setup unlike | | | ``constants.h``. It is described | | | in | +----------------------------------+----------------------------------+ | | contains code to map an index | | | described in ``Simulation.h`` to | | | a string described in the | | | ``Config`` file. | +----------------------------------+----------------------------------+ | | contains code to map a string | | | described in the ``Config`` file | | ``Simulation/Si | to an integer index described in | | mulation_mapStrToInt``\ ``.F90`` | the ``Simulation.h`` file. | +----------------------------------+----------------------------------+ | | contains a mapping between | | | particle properties and grid | | | variables. Only generated when | | | particles are included in a | | | simulation. | +----------------------------------+----------------------------------+ | | contains code to make a data | | | structure with information about | | ``Particles/Part | the mapping and initialization | | icles_specifyMethods``\ ``.F90`` | method for each type of | | | particle. Only generated when | | | particles are included in a | | | simulation. | +----------------------------------+----------------------------------+ | | | +----------------------------------+----------------------------------+ .. _`Sec:unitMakefiles`: Makefiles generated by ``setup`` ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Apart from the master ``Makefile``, ``setup`` generates a makefile for each unit, which is “included” in the master ``Makefile``. This is true even if the unit is not included in the application. These unit makefiles are named ``Makefile.Unit`` and are a concatenation of all the Makefiles found in unit hierarchy processed by ``setup``. For example, if an application uses ``Grid/GridMain/paramesh/paramesh4/|paramesh|4.0``, the file ``Makefile.Grid`` will be a concatenation of the Makefiles found in - ``Grid``, - ``Grid/GridMain``, - ``Grid/GridMain/paramesh``, - ``Grid/GridMain/paramesh/paramesh4``, and - ``Grid/GridMain/paramesh/paramesh4/|paramesh|4.0`` As another example, if an application does not use ``PhysicalConstants``, then ``Makefile.PhysicalConstants`` is just the contents of ``PhysicalConstants/Makefile`` at the API level. Since the order of concatenation is arbitrary, the behavior of the Makefiles should not depend on the order in which they have been concatenated. The makefiles inside the units contain lines of the form: .. container:: codeseg Unit += file1.o file2.o ... where ``Unit`` is the name of the unit, which was ``Grid`` in the example above. Dependency on data modules files *need not be specified* since the setup process determines this requirement automatically. .. _`Sec:hybridSetup`: Setup a hybrid MPI+|openmp| |flashx| application --------------------------------------------- The |flashx| multithreading requires a MPI-2 installation built with thread support (building with an MPI-1 installation or an MPI-2 installation without thread support is possible but strongly discouraged). The |flashx| application requests the thread support level ``MPI_THREAD_SERIALIZED`` to ensure that the MPI library is thread-safe and that any |openmp| thread can call MPI functions safely. You should also make sure that your compiler provides a version of |openmp| which is compliant with at least the |openmp| 2.5 (200505) standard (older versions may also work but I have not checked). In order to make use of the multithreaded code you must setup your application with one of the setup variables ``threadBlockList``, ``threadWithinBlock`` or ``threadRayTrace`` equal to ``True``, e.g. .. container:: codeseg ./setup Sedov -auto threadBlockList=True ./setup Sedov -auto threadBlockList=True +mpi1 (compatible with MPI-1 - unsafe!) When you do this the setup script will insert ``USEOPENMP = 1`` instead of ``USEOPENMP = 0`` in the generated Makefile. If it is equal to :math:`1` the Makefile will prepend an |openmp| variable to the ``FFLAGS``, ``CFLAGS``, ``LFLAGS`` variables. .. container:: flashtip In general you should not define ``FLAGS``, ``CFLAGS`` and ``LFLAGS`` in your ``Makefile.h``. It is much better to define ``FFLAGS_OPT``, ``FFLAGS_TEST``, ``FFLAGS_DEBUG``, ``CFLAGS_OPT``, ``CFLAGS_TEST``, ``CFLAGS_DEBUG``, ``LFLAGS_OPT``, ``LFLAGS_TEST`` and ``LFLAGS_DEBUG`` in your ``Makefile.h``. The setup script will then initialize the ``FFLAGS``, ``CFLAGS`` and ``LFLAGS`` variables in the Makefile appropriately for an optimized, test or debug build. The |openmp| variables should be defined in your ``Makefile.h`` and contain a compiler flag to recognize |openmp| directives. In most cases it is sufficient to define a single variable named ``OPENMP``, but you may encounter special situations when you need to define ``OPENMP_FORTRAN``, ``OPENMP_C`` and ``OPENMP_LINK``. If you want to build |flashx| with the GNU Fortran compiler ``gfortran`` and the GNU C compiler ``gcc`` then your ``Makefile.h`` should contain .. container:: codeseg OPENMP = -fopenmp If you want to do something more complicated like build |flashx| with the Lahey Fortran compiler ``lf90`` and the GNU C compiler ``gcc`` then your ``Makefile.h`` should contain .. container:: codeseg OPENMP_FORTRAN = –openmp -Kpureomp OPENMP_C = -fopenmp OPENMP_LINK = –openmp -Kpureomp When you run the hybrid |flashx| application it will print the level of thread support provided by the MPI library and the number of |openmp| threads in each parallel region .. container:: codeseg : Called MPI_Init_thread - requested level 2, given level 2 [Driver_initParallel]: Number of |openmp| threads in each parallel region 4 Note that the |flashx| application will still run if the MPI library does not provide the requested level of thread support, but will print a warning message alerting you to an unsafe level of MPI thread support. There is no guarantee that the program will work! I strongly recommend that you stop using this |flashx| application - you should build a MPI-2 library with thread support and then rebuild |flashx|. We record extra version and runtime information in the |flashx| log file for a threaded application. You should not setup a |flashx| application with both ``threadBlockList`` and ``threadWithinBlock`` equal to ``True`` - nested |openmp| parallelism is not supported. For further information about |flashx| multithreaded applications please refer to Chapter `[Chp:Multithreaded|flashx|] <#Chp:Multithreaded|flashx|>`__. .. [1] if a machine has multiple hostnames, setup tries them all .. [2] All non-integral values not equal to True/False/Yes/No/On/Off are considered to be string values