A Brief Overview of Methods for Computing Derivatives Wenbin Yu Department of Mechanical & Aerospace Engineering Utah State University, Logan, UT

Finite Difference vs Complex Step  Forward finite difference  Advantages: easy to use, no need to access to the source codes, no need to understand the equation or the code  Disadvantages step-size dilemma (small enough to avoid truncation error, big enough to avoid subtractive cancellation error) Expensive: always n+1 times of analysis time for n perturbations  Complex step approximation  Better than finite difference if implemented correctly  Complex variable  Complex function

Finite Difference vs Complex Step  Complex step approximation (cont.)  If analytic, Cauchy-Reimann equation holds  We deal with real functions of real variables  Not explicitly subject to subtractive cancellation errors and the truncation errors can be made as small as possible

Dual Number Automatic Differentiation (DNAD)  Extend all real numbers by adding a second component  d is just a symbol, analogous to the imaginary unit, but all powers of d higher than one equal to zero  Example:

 Dual-number arithmetic  Complex step arithmetic Dual Number Automatic Differentiation (DNAD)

 Comparing DNAD and complex step  DNAD is more efficient as calculations are never more and mostly less (less for *, /, and most intrinsic functions)  DNAD is more accurate as it delivers the analytical derivatives up to machine precision while complex-step is accurate only for extremely small imaginary parts; cancelation and subtraction errors can occur for some functions  Complex-step only has implementation and compiling optimization advantage for codes in languages supporting complex algebra (Fortran), while DNAD as a concept can be used for codes written in any strongly typed languages with real numbers defined  Complex step is not applicable to codes having complex operations and it can only compute sensitivities with respect to one variable  Changes the calculation of the original analysis and program flow  Hard to debug as many complex operations are defined by not what you need Derivative of Exp(|x|) at x=-3 Dual Number Automatic Differentiation (cont.) IF(ABS(x)>0) THEN …… ELSE …….. ENDIF

Performance Comparison Efficiency Comparison x=1.0; y=2.0; z=3.0 ftot=0.0d0 DO i=1, f=x*y-x*sin(y)*log(z) ftot= (ftot- f)/exp(z) ENDDO write(*,*) ftot Accuracy Comparison Derivative of Exp(|z|) at z=-3 Complex step: DNAD: Exact: Both complex step and DNAD are implemented in F90/95 # of Design Variables Finite Difference1.64*21.64*41.64*101.64*161.64*17 Complex Step *43.94*93.94*153.94*16 DNAD Time (seconds) used by different methods

Implementation Using F90/95  A general-purpose F90/95 module for automatic differentiation of any Fortran codes including Fortran 77/90/95  Define a new data type DUAL_NUM TYPE,PUBLIC:: DUAL_NUM REAL(DBL_AD)::x_ad_ REAL(DBL_AD)::xp_ad_ END TYPE DUAL_NUM  Overload functions/operations needed in the analysis codes to this new data type: relational operators, arithmetic operators/functions Change to “xp_ad_(n)” with n as # of DVs for sensitivities wrt to multiple DVs

Implementation Using F90/95 (cont.)  Define EXP INTERFACE EXP MODULE PROCEDURE EXP_D END INTERFACE ELEMENTAL FUNCTION EXP_D(u) RESULT(res) TYPE (DUAL_NUM), INTENT(IN)::u REAL(DBL_AD)::tmp TYPE (DUAL_NUM)::res tmp=EXP(u%x_ad_) res%x_ad_ = tmp res%xp_ad_ =u%xp_ad_* tmp END FUNCTION EXP_D

How to Use DNAD  To AD a Fortran code use DNAD 1.Replace all the definitions of real numbers with dual numbers REAL(8) :: x TYPE(DUAL_NUM) :: x REAL(8), PARAMETER:: ONE=1.0D0 TYPE(DUAL_NUM),PARAMETER::ONE=DUAL_NUM(1.0D0,0.D0) 2.Insert “Use DNAD” right after Module/Function/Subroutine/ Program statements. 3.Change IO commands correspondingly if the code does not use free formatting read and write (can be automated by written some general-purpose utility subroutines) 4.Recompile the source along with DNAD.o 5.The whole process can be automated, and even manually it only takes just a few minutes for most real analysis codes, although step 3 is code dependent

How to Use DNAD (cont.)  To use the sensitivity capability  Insert 0 after all real inputs not affected by the design variable  Insert 1 after the real input if it directly represents the design variable  Insert the corresponding sensitivities calculated by other codes if the real inputs are affected indirectly by design variable, such as the sensitivity of nodal coordinates due to change of geometry  The sensitivities are reported in the outputs as the number following the function value  Designers only need to manipulate inputs/outputs of the code

DNAD Example PROGRAM CircleArea REAL(8),PARAMETER:: PI= D0 REAL(8):: radius, area READ(*,*) radius Area=PI*radius**2 WRITE(*,*) "AREA=", Area END PROGRAM CircleArea PROGRAM CircleArea USE DNAD TYPE (DUAL_NUM),PARAMETER:: PI=DUAL_NUM( D0,0.D0) TYPE (DUAL_NUM):: radius,area READ(*,*) radius Area=PI*radius**2 WRITE(*,*) "AREA=",Area END PROGRAM CircleArea Input: 5 AREA= Input: 5,1 AREA= ,

Value ( ) Sens. (E) ( ) Sens. (b) ( ) Sens. (h) ( ) EA EI EI GJ GJ (exact) Example (VABS-AD) VABS: 10,000+ lines of F 90/95 codes An isotropic rectangular section meshed with two 8-noded quads Element 1 Loss of accuracy due to coarse mesh remains the same, can be verified by sensitivity wrt E which is equal to GJ/E and GJ is linear of E h b Element 2

Example (VABS-AD) Changes to the inputs Sensitivity wrt E: Change 0.26E E+00 To 0.26E E Sensitivity wrt b: Sensitivity wrt h: For more complex geometry and mesh, such inputs should be prepared by a mesh generator: so-called geometry sensitivity. Note # of design variables is not the same as # of seeds for inputs

ValueSensitivityValueSensitivity Exact (linear) GEBT (linear) GEBT (nonlinear) GEBT (follower) Example (GEBT-AD) 12 Responses and sensitivities by different analyses GEBT: 5000 lines of Fortran 90/95 codes 20,000 lines of Fortran 77 codes Includes BLAS, MA28 sparse linear solver, LAPACK, ARPACK sparse eigensolver Sensitivity with respect to L

Analytic Method  If we know the equations, there are even more efficient methods  A linear system  Unknowns: q; design parameter x; objective function:  Derivative of the objective  Direct method  Adjoint method

Recommendation  Neither equations no source codes are available: finite difference (FD) method, step-size dilemma  AD: source codes available  Computational/algorithmic/automatic differentiation (AD) (apply chain rule to each operation in the program flow)  Forward (direct): SRT: ADIFOR, OpenAD, TAF, TAPENADE; OO : AUTO_DERIV, HSL_AD02, ADF,SCOOT  Reverse (adjoint): very difficult for a general-purpose implementation  complex-step: better than FD, less accurate/efficient than AD  Analytic methods: equations are known  Continuous sensitivity: differentiate then solve  Discrete sensitivity: approximate then differentiate  Direct differentiation (forward) or adjoint (reverse) formulation  Source codes can be exploited if the algorithms are also known

Recommendation (cont.)  Forward (direct) vs reverse (adjoint)  Forward mode is in principle more efficient if the number of objectives and constraints is larger than number of design variables (geometric sensitivities, many stress constraints)  Forward mode is easier and more straightforward for implementation, easier to exploit sparsity and etc.  AD vs analytic methods  AD: very little effort to differentiate a code (conditional compilation); can be done by analysts using AD tools developed by professional differentiators; not efficient  Analytic methods: efficient; needs to know the equations; exploiting of existing codes is possible but need to know the algorithms, more changes to the original codes

Recommendation (cont.)  Continuous vs discrete sensitivity  Continuous method obtains approximate solution for exact derivatives, while discrete method obtains exact derivatives of approximate solution  Continuous sensitivity is more accurate/efficient, particularly for problems with changing domain (topology/shape design)  Continuous sensitivity requires deep understanding of the problem (GDEs & BCs), significant effects are needed to derive sensitivity equations and BCs. Discrete sensitivity only requires nominal understanding of the equations and algorithms  Continuous sensitivity usually requires more changes to existing codes, while discrete sensitivity needs less changes and most of the change can be done automatically  Continuous sensitivity is the same as the discrete sensitivity if the same discretization, numerical integration, and linear design velocity fields are used for both methods

Sensitivity Analysis of MDO  Multiple analysis codes: preprocessors (CAD/mesh generator), aerodynamic codes, structure codes, performance analysis codes  Different people involved:  Analysts: developers of analysis codes  Differentiators: sensitivity enablers of analysis codes,  Designers: end users of multiple analysis codes for MDO  They could be all different, and collaboration may not be practical (e.g. sensitivity analysis of NASTRAN)  Possible two-way communications between analysis codes (e.g. iterative process); only sensitivities of the converged state are needed, a linear problem could be solved directly for one code or iteratively for multiple coupled codes

Sensitivity Analysis of MDO (cont.)  Suggestions 1.If source codes are not accessible, use finite difference 2.If source codes are available, but we don’t know much about the equations/algorithms (NASTRAN, CAD), use AD; If possible, iterative nonlinear solver should be avoided for efficiency 3.If we have some knowledge of the equations and algorithms, use discrete analytical method 4.If we have a deep knowledge of the equations/algorithms, use continuous analytical method 5.If # of objectives/constraints is larger than # of design variables, use forward mode, otherwise use reverse (adjoint) mode 6.Source codes differentiated by AD can be used as excellent tools to verify analytical sensitivity methods 7.The designer should only deal with inputs and outputs of the code and not have to access to the source and recompile the codes: similar to finite difference but with the capability to handle multiple design variables 8.If possible, collaborations should be facilitated between designers, differentiators, and analysts