Driver Unit

The Driver unit controls the initialization and evolution of Flash-X simulations. In addition, at the highest level, the Driver unit organizes the interaction between units. Initialization can be from scratch or from a stored checkpoint file. It also implements methods for advancing the solution, and calls the IO unit at the end of every timestep to produce checkpoint files, plot files, or other output.

Driver Routines

The most important routines in the Driver API are those that initialize, evolve, and finalize the Flash-X program. The file main.F90 contains the main Flash-X program (equivalent to main() in C). The default top-level program of Flash-X, Simulation/main.F90, calls Driver routines in this order:

program Flashx

implicit none

call Driver_initParallel()

call Driver_initAll()

call Driver_evolveAll( )

call Driver_finalizeAll( )

end program Flashx

Therefore the no-operation stubs for these routines in the Driver source directory must be overridden by an implementation function in a unit implementation directory under the Driver or Simulation directory trees, in order for a simulation to perform any meaningful actions. The most commonly used implementations for these files are located in the Driver/DriverMain unit implementation directory, with a few specialized ones in Driver/DriverMain/Unsplit.

Driver_initAll

The first of these routines is Driver/Driver_initParallel, which initializes the parallel environment for the simulation. New in Flash-X is an ability to replicate the mesh where more than one copy of the discretized mesh may exist with some overlapping and some non-overlapping variables. Because of this feature, the parallel environment differentiates between global and mesh communicators. All the necessary communicators, and the attendant meta-data is generated in this routine. Also because of this modification, runtime parameters such as iProcs, jProcs etc, which were under the control of the Grid unit in Flash-X, are now under the control of the Driver unit. Several new accessor interface allow other code units to query the driver unit for this information. The Driver/Driver_initAll, the next routine, in general calls the initialization routines in each of the units. If a unit is not included in a simulation, its stub (or empty) implementation is called. Having stub implementations is very useful in the Driver unit because it allows the user to avoid writing a new driver for each simulation. For a more detailed explanation of stub implementations please see . It is important to note that when individual units are being initialized, order is often very important and the order of initialization is different depending on whether the run is from scratch or being restarted from a checkpoint file.

Driver_evolveAll

The next routine is Driver/Driver_evolveAll which controls the timestepping of the simulation, as well as the normal termination of Flash-X based on time. Driver_evolveAll checks the parameters tmax, nend to determine that the run should end, having reached a particular point in time, a certain number of steps, or a particular cosmological redshift, respectively. Likewise the initial simulation time, step number and cosmological redshift for a simulation can be set using the runtime parameters tmin, nbegin.

The implementation in the Driver/DriverMain/Unsplit directory is the default. This implementation in general calls each of the physics routines only once per time step, and each call advances solution vectors by one timestep. At the end of one loop of timestep advancement, the condition for updating the adaptive mesh refinement pattern is tested and applied.

Runtime Parameters

The Driver unit supplies certain runtime parameters regardless of which type of driver is chosen. These are described in the online Driver/Runtime Parameters Documentation page.

Driver_finalizeAll

Finally, the the Driver unit calls Driver/Driver_finalizeAll which calls the finalize routines for each unit. Typically this involves deallocating memory and any other necessary cleanup.

Driver accessor functions

Driver unit also provides a number of accessor functions to get data stored in the Driver unit, for example Driver/Driver_getDt, Driver/Driver_getNStep, Driver/Driver_getElapsedWCTime, Driver/Driver_getSimTime.

The Driver unit API also defines two interfaces for halting the code, Driver/Driver_abort and Driver/Driver_abortC .c. The ’c ’ routine version is available for calls written in C, so that the user does not have to worry about any name mangling. Both of these routines print an error message and call MPI_Abort.

Time Step Limiting

The Driver unit is responsible for determining the time step \(\Delta t\) that is used to advance the solution from time \(t=t^{n-1}\) to \(t^n\). At startup, a tentative \(\Delta t\) is chosen based in runtime parameters, in particular Driver/dtinit. The routine Driver/Driver_verifyInitDt (usually invoked from Driver/Driver_initAll) is used to check and, if necessary, modify the initial \(\Delta t\). Subsequently, the routine Driver/Driver_computeDt (usually invoked from Driver/Driver_evolveAll at the end of an iteration of the main evolution loop) is used to recompute \(\Delta t\) for the next evolution step.

The implementation of Driver/Driver_computeDt can lower or increase the time step. It takes various runtime parameters into consideration, including Driver/dtmin, Driver/dtmax, and Driver/tstep_change_factor. For the most part, however, Driver/Driver_computeDt calls on UNIT_computeDt routines of various code units to query those units for their time step requirements, and usually chooses the smallest time step that is acceptable to all units queried.

The code units that participate in this negotiation return a \(\Delta t\), and usually some additional information about which location in the simulaton domain caused the reequired step to be as low as returned. A unit’s time step requirement can depend on the current state of the solution as well as on further runtime parameters. For example, the dt returned by physics/Hydro/Hydro_computeDt depends on the state of density, pressure, and velocities (and possibly additional variables) in the cells of the domain, as well as on the runtime parameter Hydro/cfl.

Generic time step limiting for positive definiteness

A generic method for limiting time steps, based on a requirement that certain (user-specified) variables should never become negative, has been added in Flash-X. To understand the basic idea, think of each variable that this limiter is applied to as a density of some quantity \(q\). (Variables of PER_MASS type are actually converted to their PER_VOLUME counterparts in the implementation of the method.)

The algorithm is based on examining the distribution of \(q\) in neighboring cells, and making for each cell a near-worst-case estimate of the rate of depletion \(\dot q\) based on projected fluxes of \(q\) out of the cell over its faces. This can be applied to any variable, it is not required that the variable represents any quantity that is actually advected as described. See for how to use. This feature may be particularly useful when applied to “eion” in multidimensional 3T simulations in order to avoid some kinds of “negative ion temperature” failures, as an alternative to lowering the Hydro runtime parameter Hydro/cfl.