1. www.inl.gov DTSTEP: Development of an Integer Time Step Algorithm George L Mesina INL David L Aumiller BMPC September 2010

2. Outline Background and issues Integer time step Solutions and implementation Testing Conversion

3. Background: PVM Coupling Time Step Methods Synchronous coupling requires all coupled codes to use the same dt. – All codes suggest dt to the Executive every time step – The Executive limits dt based on the halving/doubling scheme dt = DTMAX(Exec)/2 k where dt <= smallest suggested dt. Asynchronous coupling only requires codes to arrive at an Executive mandated time target. Can use own dt, DTMAX, DTMIN, etc. – Parallel Asynch, each code marches independently to target – Sequential Asynch, one code marches to target, sends data to other; other marches to target.

4. Background: PVM Coupling Thermal-Hydraulic Methods Explicit coupling exchanges data at each coupling time. – Simple method that is numerically unstable for many situations. Semi-implicit coupling exchanges extra data at each time step to add stability. Only certain combinations of time step methods and thermal-hydraulic methods are permissible. AllowableSemi ImplicitExplicit SynchRequiredAllowed AsynchParallelSequential N/AAllowed

5. Principal Issue: Time Step Inaccuracy Issue With 64-bit floating point variables on an Opteron chip – Accurate to approximately 14 digits – The final value of timehy = 1005.0000000745, not 1005.0 exactly. This is inaccurate in the 12 digit. Inaccuracy builds up with longer time intervals. Such inaccuracy caused PVM-coupled codes to deadlock (hang). – For example, one code would hit a time target and send the proper message while the other would step past it and eventually send a totally different message to the first code. Often these hangs were caused by special cases, e.g. velocity flip-flop. T=1000.0; dt=0.005 Do 10 j = 1, 1000 10 timehy = timehy + dt

6. Sources of PVM Communication Issues Asynchronous codes can have different DTMAX and DTMIN. – Can cause time disparity for floating point sums. – Original solution was to create epsilon-tolerance if-tests Synchronous RELAP5-3D – Time targets were supplied, but change of timecard was not. RELAP5-3D DTSTEP fundamental algorithm required end time to operate correctly. – Both Executive and RELAP5-3D DTSTEP routines calculated synchronous cumulative times and time-based actions. Produced time discrepancies that deadlocked the machine. If (abs(timehy – target) <= eps) TAKE ACTION If DTMIN <= eps, can trigger action before actual target. If eps too small, can step right over target.

7. Summary of DTSTEP Subroutines and Issues Floating point inaccuracy caused trouble hitting time-targets exactly – Especially problematic for PVM communications – General solution is integer time step There were other algorithmic and implementation errors – These will be listed later with individual solutions DTSTEP did too much for one subprogram It was very difficult to read and understand – Too many jumps – Logical variables with non-mnemonic names – Too many pre-compiler directives and conditional coding – Almost no internal documentation

8. Original RELAP DTSTEP Flow (Simplified) Initialization Normal &1 st time only Abrupt Changes Repeat with same dt, halve/double, terminate transient Gather Statistics New Advancement End timecard or transient, SS check Output (9 kinds) Screen, plots, minor, major, restart, GUI PVM Explicit Exchange Parallel or Seq, synch or asynch New Time Targets Optional PVM Semi-imp. rcv, New time targets User Interfaces Time step control dt-stretch, Courant/mass error, dt >= DTMAX/2 k Time step selection Optional PVM Synch Exch, 105 card handler, advance time, EXIT

9. Integer Time Step Based upon Clock-cycle RELAP5-3D and PVMEXEC use a timestep halving/doubling scheme that guarantees hitting integer multiples of the users DTMAX – Time targets from RELAP5-3D timecards are such multiples – plot, minor, major, restart edits – Special targets, timecard endtimes and PVM targets, need not be. No step smaller than users DTMIN allowed. Smallest allowable dt is: – dtsmall = DTMAX/2 n > DTMIN > DTMAX/2 n+1. All normal values of dt = 2 k dtsmall, k=0, 1,..., n. Dtsmall is effectively the clock-cycle of RELAP5. Always has been. To convert to integer time step, use d = Round(dt/dtsmall) = 2 k. – Only within DTSTEP, outside DTSTEP use dt = d * dtsmall. dtsmall = DTMAX/2 n d = Round(dt/dtsmall) = 2 k

10. Integer Time Steps vs. Floating Point Time Integer arithmetic is infinitely precise for add, subtract and multiply. – The earlier do-loop-10 example creates no summing error. – Error tolerances are eliminated. Time, T = * dtsmall, is more exact and slightly different from floating point calculated time. – Most of your RELAP5-3D results will be slightly different. Integer cumulative time becomes large and requires a 64-bit integer. – 2 63 = 2 3 2 60 = 8(1024) 6 > 8x(10 3 ) 6 = 8x10 18. – This is sufficient to represent 10 9 seconds with DTMIN = 10 -9. Integer Time,, starts at zero on each successive timecard. – T = * dtsmall + T(start of timecard) = floating point time. – Required because user can change DTMAX and DTMIN. T = * dtsmall + T(start of timecard )

11. Comparison of Integer and Real Time t (seconds) (cycles) 0.004 0.008 3278465568 1.00 819600 realinteger dtsmallDTMAX/2 n 1 Largest dtDTMAX2n2n Test|x-y| < epsix == iy TimeT bgn + k dt k T bgn + dtsmall* k idt k

12. Normal vs. Unusual Target Times Normal time target, T, is an integer multiple of DTMAX Unusual time target, T, is not integer multiple of DTMAX. – e.g. DTMAX= 0.33, Timecard end = 10.0, Assume no cuts. At T = 9.9, need dt = 0.1, but 0.1 /= DTMAX/2 k. Unusual times are handled by using a floating point dt, which exactly reaches the unusual time, and a resynch variable. – The resynch variable indicates need to resynch dt and d. e.g.Dtmax = 0.004, dtmin = 1.0e-7, dtsmall =1.22e-7, target=1.00021 t (seconds) 1.0001.0021.004 t 2 =1.0021 t3t3 (cycles) 1 2 3 = 1 +d sav t1t1

13. Some Other DTSTEP Errors Explosive Doubling – When PVM and RELAP5-3D timecards differed with R5 timecard end time exceeding PVM transient end-time, – The timestep could double on every timestep until the R5 timecard end time was exceeded. – The source was variable, NREPET. – The solution: PVM timecards override the RELAP5-3D timecards. Penultimate timecard stop – Some asynchronous processes stopped at end-time of second to last timecard. – Cause: an error in resetting variable CURCLM in R-DTSTEP. – This error has been fixed.

14. Some Other DTSTEP Errors (continued) Hang when subroutine HYDRO set variable, FAIL – R5 incorrectly sent a message that the timestep was succesful – Solution: Set PVM success message to repeat (as R5 repeats) Hang in explicit asynchronous sequential – When follower received go-ahead message from leader, but had to repeat its first advancement. It returned to receive go-ahead from leader message => hang – Solution: Logical variable prevents return to this receive message until time actually advances. Hang from repeat condition at minimum dt (expl asyn parallel) – In certain cases RELAP5 DTSTEP subroutine incorrectly set dt twice the size of EXEC dt. – Solution: Rewrote/simplified R-DTSTEP code correctly.

15. Some Other DTSTEP Errors (continued) Stretch logic error – On a 2-step approach to an unusual time, first step was half the distance, but second step went to multiple of DTMAX, not endtime – Source: one real target time was k*DTMAX, not endtime – Solution: (P- and R-DTSTEP) reset ALL time targets to the unusual endtime when within one DTMAX Error Messages P-DTSTEP and some PVM diagnostics were improved.

16. Solutions and Implementation Replace floating point time with integer based time. – Extra integer variables were created in both R-DTSTEP and P-DTSTEP – Replaced floating point if-tests by integer versions – Created internal subroutines for initializing and calculating integer time from real and vice-versa. – Create unifying mechanisms for quantities and processes shared by both DTSTEPS (next slide) Shared module

17. Some Unifying Implementation Features Isnormal – determines if time is normal or unusual – abs(1-dt/idt*dtsmallest) < 1.0e-10 – Interpretation: dt & idt*dtsmallest agree to 10 places => dt normal. Calctimehy – Calculate real time from certain integer time variables – Uses quad precision to guarantee correct conversion idtmod – f95 module – Contains internal subroutines and data common to both P- and R- DTSTEP subroutines Used by both – Ensures both perform same functions in same way – Reduces coding, increases maintainability

18. Implementation continued Simplified and enhanced both RELAP5 DTSTEP and PVMEXEC DTSTEP by moving portions of coding into internal subroutines. – Separated flowchart sections were pulled together. – Create internal subroutines for reused sections of code, e.g. output – Create internal subroutines for most precompiler-protected code – Replace jumps to a section of code with call of internal subroutine Enhance readability – Determine definitions of all variables and document – Give mnemonic names to logical variables – Implement Outline-style documentation for sections of code – Simplify code and reduce logic paths – Eliminate dead code and unused variables

19. Implementation continued Identified and fixed algorithmic and implementation errors Created new test cases to increase coverage – Especially for failure conditions and PVM coupling Verified that all test cases run correctly.

20. DTSTEP Test Matrix To fully test the relevant logic paths through DTSTEP, a matrix of over 2000 cases was constructed. BASIC – 17 tests cases that cover normal, special case, and failure case and some combinations. Group A(34) Basic @ normal dt Basic @ minimal dt Group B(102) Group A @ minor edit Group A @ expl exchange Group A @ transient end Group C(408) Group B w/ normal target Group B w/ unusual time Group B w/ normal then unusual target Group B w/ target every step Group D(2856) Group C R5 standalone Group C R5-R5 expl asynch parallel Group C R5-R5 semi-impl Group C R5-R5-R5 expl & semi-impl Group C R5-R5 expl synch Group C R5-R5 expl asynch sequ Group C R5 controled by Exec

21. Conversion to F95 The integer algorithm was developed in version 2.4. – These use the old container array (FA) and equivalences. All R-DTSTEP, P-DTSTEP, and related changes were ported to version 2.9.5 – Merge issues resolved (post-2.4 development and differences in integer time implementation) Conversion to F95 – Access F95 modules, replace FA variables with module ones Resolution of variable declarations – Issue with precompiler directive and 32-bit PVM Verified again that all test cases worked correctly.