1. © Semantic Designs, Inc. 1 5/26/2016 DMS ® Software Quality Enhancement via Automated Software Analysis, Modification and Generation Ira D. Baxter Semantic Designs, Inc. www.semdesigns.com September 2002

2. © Semantic Designs, Inc. 2 5/26/2016 Semantic Designs Corporate Goal To enable our customers to produce and maintain timely, robust and economical software by providing world-class Software Engineering tools using deep problem domain knowledge and high degrees of automation.

3. © Semantic Designs, Inc. 3 5/26/2016 Modern Software Engineering Large software system in multiple languages 80% Maintenance/Enhancements Little accurate design documentation Largely manual effort How to: –Understand software structure? –Reorganize structure to enable change? –Make sweeping changes? –Make reliable changes?

4. © Semantic Designs, Inc. 4 5/26/2016 DMS ® Software Reengineering Toolkit Customized, automated analysis, modification, porting or generation –Enables wide variety of source-based SE tasks to be automated –For sources for large scale software systems Scalable to millions of source lines, tens of thousands of files Parallel processing foundations to support scale –Handles many and mixed languages simultaneously C, C++, Java, Ada, Fortran, SQL, XML, assembler, … –Generalized compiler technology conveniently integrated Parsing, Analyzing, Transforming, Prettyprinting Enables practical customization for desired automation task Predefined support for standard computer languages Huge infrastructure cost amortized over many tasks/customers Semantic Designs Supporting Services and Tools –Consulting & Training to customer on DMS usage –Implementation of DMS customization –Selected SE tasks prepackaged: formatting, test coverage, …

5. © Semantic Designs, Inc. 5 5/26/2016 Changing the Economics of Software Present software difficult to build/maintain –Key difficulties: Manual process: slow, poor engineering scale Absence of expert knowledge: –General “how to” knowledge –Specific details about structure of artifact Tackle by integrating many research ideas well engineered –Generalized compilers Program Transformations (DARPA-funded technology 1970-1990!) –NIST/ATP funded practical implementation (1995) –Capture expert knowledge as rules Benefits –Faster (automation) –Better (knowledge) –Radically Cheaper (lowered engineering costs)

6. © Semantic Designs, Inc. 6 5/26/2016 Manual Software Methods Engineering Activity Sr. Programmers/ Engineers Software System Sources (MSLOC) example … a in b… if a in c then p=7 else p=2 ; Analyses of Software System a in b b in c a in c Revised/Generated Software System Sources … a in b… p=7; Implicit Background Knowledge Expensive! Unknown; lost between iterations! Slow! Hard to repeat! Editors, Compilers Low level of automation!

7. © Semantic Designs, Inc. 7 5/26/2016 Automation In Software Engineering Manual coding/analysis expensive –Typical: $100K/year/man/4KSLOC  $25/SLOC –Presently not much automated help; very expensive to build Automation possible –For problem domains with well-defined semantics Computer languages, specification languages, … –For well-defined tasks Analysis: error detection, test support, documentation extraction, reverse engineering, … Modification: structure improvement, error handler insertion, API change, code porting, … Code generation: from specs, diagnostics, test cases,…. –Using Program Transformation technology = Generalized compiler componentry Researched by community over past 25 years

8. © Semantic Designs, Inc. 8 5/26/2016 Stepwise Semiautomatic Conversion of Specs to Code fSfS f1f1 t1t1 t2t2 (x-1)y+2y (xy-1y)+2y xy-y+2y xy+y (x+1)y Transforms Spec Prog Transform Engine cici distributive law unity multiplier fkfk fGfG t k-1 tktk f k-1 remove parentheses factoring fSfS fGfG t k-2 like-term combination Rqmts... Transformation Systems

9. © Semantic Designs, Inc. 9 5/26/2016 DMS Concept Engineering Activity Background Knowledge for Software Analysis or Modification problem (KSLOC) Language Definitions Ada, SQL, Java … General Language Analyses ”var=expression” => “var modified” General Language Transforms ”if true then s else t endif -> s” DSL Language Analyses “x in y and y in z” => “x in z” DSL Application Context Knowledge “b in c” DSL Refinement Language Transforms “?x in ?y” => “f=false; for (i…) if ?x==?y[i] then f=true; from Semantic Designs or Consultants/Sr. Programmers/Engineers Software System Sources (MSLOC) example … a in b… if a in c then p=7 else p=2 ; Analyses of Software System a in b b in c a in c Revised/Generated Software Sources … a in b… p=7; Automated! Repeatable! Scalable! Captured! DMS

10. © Semantic Designs, Inc. 10 5/26/2016 DMS Impact on Quality and Process Quality –(Re)use of tested specification techniques avoids ad hoc descriptions –Reuse of abstract generative components (transforms) Not code reuse, but rather implementation knowledge reuse –Specifications and implementation steps inspectable by others –Reuse of tested synthesis/modification methods –Mechanical, reliable construction of product –Easier recovery from errors: correct mistake, re-execute task Process –Reliable components --> avoid rework after changes –Mechanically repeatable implementation steps –Focus on knowledge acquisition rather than repeated coding events

11. © Semantic Designs, Inc. 11 5/26/2016 Overview DMS ® Software Reengineering Toolkit –Defining notations (“domains”) for specs and legacy systems –Parsing and prettyprint –Transformation mechanics Applications for Software Quality Improvement –C++ preprocessor conditional removal –Software Test Coverage –Cross Reference and Dead Code –Refactoring Java –Automatic Code Generation (XML Parsers) –Fast HTML generation using XSLT –Clone Detection/Removal –Porting application software to new languages

12. © Semantic Designs, Inc. 12 5/26/2016 DMS Domain (Notation + Semantics) Parts External Form (what you can say: string or graphical) Internal Form (How DMS stores it) Parser (how to convert external form to internal form) PrettyPrinter (how to display the Internal Form) Semantics (what the Internal Form means) Optimizations (how to optimize in the domain) Refinements (how to transform IF to another IF) Analyzers (how to analyze in the domain) Attachments (procedures to enhance DMS efficiency)

13. © Semantic Designs, Inc. 13 5/26/2016 nested_class_declaration = nested_class_modifiers class_header class_body ; >: { V(H(nested_class_modifiers,class_header),class_body); } class_header = 'class' IDENTIFIER ; >: { H('class',IDENTIFIER); } class_header = 'class' IDENTIFIER 'implements' name_list ; >: { H('class',IDENTIFIER,'implements',name_list); } class_header = 'class' IDENTIFIER 'extends' name; >: { H('class',IDENTIFIER,'extends',name); } class_header = 'class' IDENTIFIER 'extends' name 'implements' name_list ; >: { H('class',IDENTIFIER,'extends',name,'implements',name_list); } class_body = '{' class_body_declarations '}' ; >: { V(H('{',STRING(" "),class_body_declarations),'}'); } nested_class_modifiers = nested_class_modifiers nested_class_modifier ; >: { H(CH(nested_class_modifiers[1]),nested_class_modifier); } DMS Domain for Java Parser + Pretty Printer … + 300 more rules…(COBOL is 3500!)

14. © Semantic Designs, Inc. 14 5/26/2016 Parsing to Abstract Syntax Trees A Program Representation analyzable by Computers Use DMS grammar domain to define language syntax DMS generates lexer/parser automatically Parser reads source file(s) –Captures comments –Carries out lexical conversions (e.g, FP text -> IEEE binary fp) –Builds Abstract Syntax Tree –Records Position of every node (file, line, col) Present capability for the following domains –Specification: Spectrum, BNF, Rose Models –Technology: XML, IDL, SQL –Implementation: C/C++, COBOL, Java, Ada, VB6, Fortran, Verilog

15. © Semantic Designs, Inc. 15 5/26/2016 A Simple Java Program /* Fib.java */ public class NumberTheory { int Fib(int x) { if (x < 1) return 1; // base case else return Fib(x-1)+Fib(x-2); } 001 002 003 004 005 006 007 008 009 010

16. © Semantic Designs, Inc. 16 5/26/2016 Abstract Syntax Tree (AST) for Fib Class … free of lexical properties (‘text shape’) of program... Class Header Class Body Method Declaration Type INT ID `x’ Type INT Parameter ID `Number Theory’ Method Modifiers ID `Fib` Parameters Empty Throwlist Block Stmt Sequence If Then Else < ID `x’ NUMBER 1 Return NUMBER 1 Return + Function Call Function Call ID `Fib’ - ID `Fib’ - ID `x’ NUMBER 1 ID `x’ NUMBER 2 Not shown: File/line/column annotation on each node /* Fib.java */ public class NumberTheory { int Fib(int x) { if (x < 1) return 1; // base case else return Fib(x-1)+Fib(x-2); }

17. © Semantic Designs, Inc. 17 5/26/2016 PrettyPrinting: “AntiParsing” Conversion of AST back to text file Handles indentation, comments, literal formats... Uses DMS Box language to compose PP fragments If Then < ID `x’ NUMBER 1 Return NUMBER 1 V(H(‘if’,’(‘,condition,’)’), I(then_stmt)); H(‘return’,expression,’;’); H(expression1,’<‘,expression2); Prettyprinted result: if (x<1) return 1;

18. © Semantic Designs, Inc. 18 5/26/2016 Winner: Obfuscated “C” Contest The Maintenance Programmer’s Nightmare #include double L,o,P,_=dt,T,Z,D=1,d, s[999],E,h= 8,I, J,K,w[999],M,m,O,n[999],j=33e-3,i= 1E3,r,t, u,v,W,S= 74.5,l=221,X=7.26, a,B,A=32.2,c, F,H; int N,q, C, y,p,U; Window z; char f[52] ; GC k; main(){ Display*e= XOpenDisplay( 0); z=RootWindow(e,0); for (XSetForeground(e,k=XCreateGC (e,z,0,0),BlackPixel(e,0)) ; scanf("%lf%lf%lf",y +n,w+y, y+s)+1; y ++); XSelectInput(e,z= XCreateSimpleWindow(e,z,0,0,400,400, 0,0,WhitePixel(e,0) ),KeyPressMask); for(XMapWindow(e,z); ; T=sin(O)){ struct timeval G={ 0,dt*1e6} ; K= cos(j); N=1e4; M+= H*_; Z=D*K; F+=_*P; r=E*K; W=cos( O); m=K*W; H=K*T; O+=D*_*F/ K+d/K*E*_; B= sin(j); a=B*T*D-E*W; XClearWindow(e,z); t=T*E+ D*B*W; j+=d*_*D-_*F*E; P=W*E*B-T*D; for (o+=(I=D*W+E *T*B,E*d/K *B+v+B/K*F*D)*_; p<y; ){ T=p[s]+i; E=c-p[w]; D=n[p]-L; K=D*m-B*T-H*E; if(p [n]+w[ p]+p[s ]== 0|K K)N=1e4; else{ q=W/K *4E2+2e2; C= 2E2+4e2/ K *D; N-1E4&& XDrawLine(e,z,k,N,U,q,C); N=q; U=C; } ++p; } L+=_* (X*t +P*M+m*l); T=X*X+ l*l+M *M; XDrawString(e,z,k,20,380,f,17); D=v/l*15; i+=(B *l-M*r -X*Z)*_; for(; XPending(e); u *=CS!=N){ XEvent z; XNextEvent(e,&z); ++*((N=XLookupKeysym (&z.xkey,0))-IT? N-LT? UP-N?& E:& J:& u: &h); --*( DN -N? N-DT ?N== RT?&u: & W:&h:&J ); } m=15*F/l; c+=(I=M/ l,l*H +I*M+a*X)*_; H =A*r+v*X-F*l+( E=.1+X*4.9/l,t =T*m/32-I*T/24 )/S; K=F*M+( h* 1e4/l-(T+ E*5*T*E)/3e2 )/S-X*d-B*A; a=2.63 /l*d; X+=( d*l-T/S *(.19*E +a *.64+J/1e3 )-M* v +A* Z)*_; l += K *_; W=d; sprintf(f, "%5d %3d" "%7d",p =l /1.7,(C=9E3+ O*57.3)%0550,(int)i); d+=T*(.45-14/l* X-a*130-J*.14)*_/125e2+F*_*v; P=(T*(47 *I-m* 52+E*94 *D-t*.38+u*.21*E) /1e2+W* 179*v)/2312; select(p=0,0,0,0,&G); v-=( W*F-T*(.63*m-I*.086+m*E*19-D*25-.11*u )/107e2)*_; D=cos(o); E=sin(o); } }

19. © Semantic Designs, Inc. 19 5/26/2016 Pretty Printing to un-obfuscate #include double L, o, P, _ = dt, T, Z, D = 1, d, s[999], E, h = 8, I, J, K, w[999], M, m, O, n[999], j = 3.3e-2, i = 1e3, r, t, u, v, W, S = 7.45e1, l = 221, X = 7.26, a, B, A = 3.22e1, c, F, H; int N, q, C, y, p, U; Window z; char f[52]; GC k; main() { Display * e = XOpenDisplay(0); z = RootWindow(e, 0); for (XSetForeground(e, k = XCreateGC(e, z, 0, 0), BlackPixel(e, 0)); scanf("%lf%lf%lf", y + n, w + y, y + s) + 1; y++); XSelectInput(e, z = XCreateSimpleWindow(e, z, 0, 0, 400, 400, 0, 0, WhitePixel(e, 0)), KeyPressMask); for (XMapWindow(e, z);; T = sin(O)) { struct timeval G = { 0, dt * 1e6 }; K = cos(j); N = 1e4; M += H * _; Z = D * K; F += _ * P; r = E * K; W = cos(O); m = K * W; H = K * T; O += D * _ * F / K + d / K * E * _; B = sin(j); a = B * T * D - E * W; XClearWindow(e, z); t = T * E + D * B * W; j += d * _ * D - _ * F * E; P = W * E * B - T * D; for (o += (I = D * W + E * T * B, E * d / K * B + v + B / K * F * D) * _; p < y;) { T = p[s] + i; E = c - p[w]; D = n[p] - L; K = D * m - B * T - H * E; if (p[n] + w[p] + p[s] == 0 | K K) N = 1e4; else { q = W / K * 4e2 + 2e2; C = 2e2 + 4e2 / K * D; N - 1e4 && XDrawLine(e, z, k, N, U, q, C); N = q; U = C; } ++p; } L += _ * (X * t + P * M + m * l); T = X * X + l * l + M * M; XDrawString(e, z, k, 20, 380, f, 17); D = v / l * 15; i += (B * l - M * r - X * Z) * _; for (; XPending(e); u *= CS != N) { XEvent z; XNextEvent(e, & z); ++ * ((N = XLookupKeysym(& z.xkey, 0)) - IT ? N - LT ? UP - N ? & E : & J : & u : & h); -- * (DN - N ? N - DT ? N == RT ? & u : & W : & h : & J); } m = 15 * F / l; c += (I = M / l, l * H + I * M + a * X) * _; H = A * r + v * X - F * l + (E = 1e-1 + X * 4.9 / l, t = T * m / 32 - I * T / 24) / S; K = F * M + (h * 1e4 / l - (T + E * 5 * T * E) / 3e2) / S - X * d - B * A; a = 2.63 / l * d; X += (d * l - T / S * (1.9e-1 * E + a * 6.4e-1 + J / 1e3) - M * v + A * Z) * _; l += K * _; W = d; sprintf(f, "%5d %3d" "%7d", p = l / 1.7, (C = 9e3 + O * 5.73e1) % 0550, (int) i); d += T * (4.5e-1 - 14 / l * X - a * 130 - J * 1.4e-1) * _ / 1.25e4 + F * _ * v; P = (T * (47 * I - m * 52 + E * 94 * D - t * 3.8e-1 + u * 2.1e-1 * E) / 1e2 + W * 179 * v) / 2312; select(p = 0, 0, 0, 0, & G); v -= (W * F - T * (6.3e-1 * m - I * 8.6e-2 + m * E * 19 - D * 25 - 1.1e-1 * u) / 1.07e4) * _; D = cos(o); E = sin(o); }

20. © Semantic Designs, Inc. 20 5/26/2016 PrettyPrint to Obfuscate Consistent identifier scrambling import javax.swing.JOptionPane; public class Program1 { public static void O0(String l1[]) { String l10,O11; double l100; double O101,l110,l111,O1000; String l1001 = " "; l1001 = JOptionPane.showInputDialog("P\145se c\150o\040ne of the \157ns:" + "\012" + "t\145rate \145f \164e"); l100 = Double.parseDouble(l1001); while (l100 != 6) { if (l100 == 1) { l10 = JOptionPane.showInputDialog("\105n\164\145\162\040t\150e\040v\141\154\165\145\040o\146\040r\141\144iu\163"); O101 = Double.parseDouble(l10); l111 = Math.PI * O101 * O101; JOptionPane.showMessageDialog(null,"\124a\040of\040\164\150e Ci\145\040 \040"+l111,"result",JOptionPane.INFORMATION_MESSAGE); } else if (l100 == 2) { l10 = JOptionPane.showInputDialog("Ent\145r\040\164\150e\040v\141l\165e \157\146\040\154en\147ht"); O11 = JOptionPane.showInputDialog("\105\156\164\145\162 \164h\145 value of\040width"); O1000 = Double.parseDouble(l10); l110 = Double.parseDouble(O11); l111 = (l110 * O1000) / 2; JOptionPane.showMessageDialog(null, “Area \157f\154e: " + l111, "re\163ult", JOptionPane.INFORMATION_MESSAGE); } else if (l100 == 3) { l10 = JOptionPane.showInputDialog("\105\156\164\145\162 \164h\145 value\040of lengt\150"); O1000 = Double.parseDouble(l10); l111 = O1000 * O1000; JOptionPane.showMessageDialog(null, "\124he area \163\161uare: " + l111, " re\163ul\164", JOptionPane.INFORMATION_MESSAGE); } else if (l100 == 4) { l10 = JOptionPane.showInputDialog("\105\156\164er t\150e\040v\141lu\145\040\157f len\147\164h"); O11 = JOptionPane.showInputDialog("E\156\164er the\040v\141l\165e \157\146 width"); O1000 = Double.parseDouble(l10); l110 = Double.parseDouble(O11); l111 = l110 * O1000; JOptionPane.showMessageDialog(null, "are\141 \164h\145 rectangle: " + l111, "\162\145\163\165\154t", JOptionPane.INFORMATION_MESSAGE); } else if (l100 == 5) { l10 = JOptionPane.showInputDialog("E\156ter t\150\145\040v\141l\165\145\040of lengt\150"); O1000 = Double.parseDouble(l10); l111 = 6 * O1000; JOptionPane.showMessageDialog(null, "T\150e area \157f\040t\150e\040\143\165\142e: " + l111, "r\145sult", JOptionPane.INFORMATION_MESSAGE); } l1001 = JOptionPane.showInputDialog("P\154ease\040\040\160r\157\147r\141m"); l100 = Double.parseDouble(l1001); } System.out.println("\120r\157\147\162a\155 \164e\162mi\156a\164ed\012"); System.exit(0); }

21. © Semantic Designs, Inc. 21 5/26/2016 Optimization transform for DMS Rewrite Rule Language Domain Syntax Domain Name We’ll see this idea again later. default base domain Java; rule merge-ifs(\condition1, \condition2, \then-statements) “if (\condition1) if (\condition2) { \then-statements } ” rewrites to “if (\condition1 && \condition2) { \then-statements } ”;

22. © Semantic Designs, Inc. 22 5/26/2016 DMS transforms work on ASTs, not text Not fooled by any lexical properties of text! If Then \condition1 If Then \condition2 \then statements If Then \condition1\condition2 \then statements && rewrites -to To modify programs: 1) define transforms 2) Parse program 3) Apply transforms a) match LHS pattern b) replace with RHS substitution 4) Prettyprint program Right hand sideLeft hand side

23. © Semantic Designs, Inc. 23 5/26/2016 Expertise == Number of Rules Mathematics –Novice (9th grade algebra): x+0  x –Amateur (HS Senior): sin^2(x)+cos^2(x)  1 –Journeyman: (Frosh Calculus) integrals –Craftsman: (B.S. Math) Linear Algebra, Group Theory –Expert: (Ph.D. Mathematics) Category Theory, Topology, … DMS –Toy: several rules –Useful: 50 rules (simplification/optimization) –Powerful: 250 rules (testing, code generation) –Indispensable: 2000 rules (massive program translation)

24. © Semantic Designs, Inc. 24 5/26/2016 DMS Toolkit = Generalized Compiler Underlying Hypergraph representation: trees, graphs, … Parsing/Prettyprinting –UNICODE lexer with binary conversions, lexical format/comment capture –GLR (context-free) parser with automatic tree builder –"Text Box" building language; reproduces comments! Analysis –Multipass attribute grammars –Generalized symbol table support: inheritance, overloading, … –Next: Generic Control/Data Flow Transformation –Complete procedural AST interface => procedural transforms (& analyzers) –Conditional Source to Source transforms w/ associative/commutative laws –Next: Goal-directed metaprogramming Predefined Domains –Spec, Technology, and Legacy languages Spectrum,.MDL XML, SQL, IDL C/C++, Java, COBOL, … assembler!

25. © Semantic Designs, Inc. 25 5/26/2016 Fundamental Issue: Scale Engineering hard but straightforward –Ugly details: C preprocessors, etc. Reasoning/Analysis costs –"Incremental" Knowledge capture => domains –Computers do Symbolic computation slowly => Parallel foundations –Future: Rule compilers Legacy Systems are huge –MSLOC + tens of thousands of files Careful design of hypergraph to conserve space –Arbitrary languages => robust parsing technology Need for domain agility: fast domain/dialect definition Use domain notations define knowledge –Applications use multiple languages reasoning and transforms must work with mix Other scale issues –Software Versions, Large Engineering Teams, Long term Transactions

26. © Semantic Designs, Inc. 26 5/26/2016 Overview DMS ® Software Reengineering Toolkit –Defining notations (“domains”) for specs and legacy systems –Parsing and prettyprint –Transformation mechanics Applications for Software Quality Improvement –C++ preprocessor conditional removal –Software Test Coverage –Cross Reference and Dead Code –Refactoring Java –Automatic Code Generation (XML Parsers) –Fast HTML generation using XSLT –Clone Detection/Removal –Porting application software to new languages

27. © Semantic Designs, Inc. 27 5/26/2016 Useless Conditional Elimination Problem: Too many configuration #IF s –Application on many platforms: WNT, SUN, VAX,... –#IF s still in code –Too many to remove by hand, confusing to manage –Does delivered system work with all combinations? Solution: Use DMS to remove designated #IF –Engineer names dead configuration variables (VAX) –DMS use transforms to remove #IFs

28. © Semantic Designs, Inc. 28 5/26/2016 C++ simplifying transforms rule simplify_and_false(e:expression): expression->expression = “\e & 0” -> “0”. rule simplify_and_true(e:expression): expression -> expression = “\e & 1” -> “\e”. rule simplify_or_true(e:expression): expression->expression = “\e ! 1” -> “1”. rule simplify_or_false(e:expression): expression -> expression = “\e ! 0” -> “\e”. rule pp_if_true_simplify(b:block):statement->statement = “#if 1 \b #endif” -> “{\b}”. rule pp_if_false_simplify(b:block):statement->statement = “#if 0 \b #endif” -> “;”. rule pp_if_then_else_false_simplify(b1:block,b2:block):statement->statement = “#if 0 \b1 #else \b2 #endif” -> “{\b2}”.

29. © Semantic Designs, Inc. 29 5/26/2016 C++ sample code… simplified Add Rule for dead configuration variables rule VAX: expression -> expression = “VAX” -> “0”. Before #IF VAX!UNIX syslog(logfile->file_descriptor,”display output”); #ENDIF... #IF VAX sysclose(logfile->file_descriptor); #ELSE fclose(logfile->file_descriptor); #ENDIF After #IF UNIX syslog(logfile->file_descriptor,”display output”); #ENDIF... fclose(logfile->file_descriptor);

30. © Semantic Designs, Inc. 30 5/26/2016 Software Test Coverage Analysis of code executed by test cases –Non-executed code likely to have flaws Key problem: tracking program control flow –Need way to identify possible program parts –Capture “executed” status of parts via tests –Display execution status of program parts Secondary problem: exercising all parts –Exercising individual part –Generation of tests from specifications

31. © Semantic Designs, Inc. 31 5/26/2016 Test Coverage by Marking visited Blocks bool fibcached[1000]; int fibvalue[1000]; int fib(int i) { int t; switch (i) { case 0: case 1: return 1; default: if fibcached(i) return fibvalue(i); else { t=fib(i-1); return t+fib(i-2); }; bool fibcached[1000]; int fibvalue[1000]; int fib(int i) { int t; visited[1]=true; switch (i) { case 0: visited[2]=true; case 1: visited[3]=true; return 1; default: visited[4]=true; if fibcached(i) { visited[5]=true; return fibvalue(i);} else { visited[6]=true; t=fib(i-1); return t+fib(i-2); }; Original “C” programMarking program

32. © Semantic Designs, Inc. 32 5/26/2016 default base domain C; rule mark_function_entry(result:type, name:identifier, decls:declaration_list, stmts:statement_sequence) = “\result \name { \decls \stmts };” rewrites to “\result \name { \decls { visited[\place\(\stmts\)]=true; \stmts };”. rule mark_if_then_else(condition:expression; tstmt:statement; estmt:statement) = “if (\condition)\tstmt else \estmt;” rewrites to “if (\condition) { visited[\place\(\tstmt\)]=true; \tstmt} else {visited[\place\(\estmt\)]=true; \estmt};”. rule mark_while_loop(condition:expression, stmt:statement) = “while (\condition) \stmt” rewrites to “while (\condition) { visited[\place\(\stmts\)=true; \stmt }”. rule mark_case_clause(e:expression, stmts:statements) = “case \e: \stmts” rewrites to “case \e: { visited[\place\(\stmts\)=true; \stmts }”. DMS transform(s) to mark program

33. © Semantic Designs, Inc. 33 5/26/2016 Test Coverage Tool Flow DMS: Add marking code Compile & Run tests Display Coverage Source Code Visit-adding Transforms Test Data Decorated Code visited Vector Source line information for visited[i] Note: incrementing visited rather then setting true changes this to profiler tool!

34. © Semantic Designs, Inc. 34 5/26/2016 Display Tool Application with 3500 Java classes

35. © Semantic Designs, Inc. 35 5/26/2016 Cross Reference and Dead Code Harness name declarations per domain –Enables semantically complex transforms –Collect by attribute evaluation with symbol table Combine with PrettyPrinting for full XREF –Customize PrettyPrinter to produce HTML –Attribute evaluation collects use points –Add links from identifiers to defs/uses –Add other documentation features (simulate JavaDoc) Implement Dead Code detection/removal –FAA DO187B : “no dead code in commercial aircraft” –Diagnose unreferenced definitions –Delete using simple transforms

36. © Semantic Designs, Inc. 36 5/26/2016 JavaDoc + Full System XRef

37. © Semantic Designs, Inc. 37 5/26/2016 Detecting Dead Code package edu.ksu.cis.test; public class SuperClass{ protected transient int yy =1; void outer() {System.out.println("I'm outer"); new AB();} } class AB { SuperClass s = new SuperClass(); //AB ab = new AB(); class SuperClass{ void inner() {System.out.println("I'm inner");} } void callmethod(){s.inner();} } package edu.ksu.cis.test; public class Test extends SuperClass{ void m(){ // called from main Test t = new Test(); Test t2 = Test.this; t2.yy = 3; } class A { void m2(){ Test t = new Test(); t.yy =2; } class B {} SuperClass.java 7 : d:/users/hzheng/JavaTestSuite/first/edu/ksu/cis/test/SuperClass.java outer -> MethodDeclaredInClassType @ Line 4 Column 10 inner -> MethodDeclaredInClassType @ Line 10 Column 14 s -> FieldDeclaredInClassType @ Line 7 Column 16 AB -> TopLevelClassDeclaration @ Line 6 Column 1 callmethod -> MethodDeclaredInClassType @ Line 12 Column 10 8 : d:/users/hzheng/JavaTestSuite/first/edu/ksu/cis/test/Test.java A -> TopLevelClassDeclaration @ Line 10 Column 1 m2 -> MethodDeclaredInClassType @ Line 11 Column 10 t -> LocalVariableDeclaredInBlock @ Line 4 Column 11 B -> TopLevelClassDeclaration @ Line 17 Column 1 t -> LocalVariableDeclaredInBlock @ Line 12 Column 10 Test.java Deactive Code detection

38. © Semantic Designs, Inc. 38 5/26/2016 (define IsDeactiveDeclaration (lambda Registry:MatchingCondition (~ (SymbolUsesMap:GetUses symbol_uses_map (NodeUsesMap:GetWrapperDeclaration node_uses_map arguments:1))) )lambda )define (define RemoveEntryFromSymbolTable (action Registry:RuleSideEffect (local [wrapper_declaration JavaSymbolTable:WrapperDeclaration] (;; `remove entry of arguments:1 from symboltable': (= wrapper_declaration (NodeUsesMap:GetWrapperDeclaration node_uses_map arguments:1)) (Console:Put (. `~l Remove symbol table entry for node : ')) (AST:PrintSourcePosition OutputStream:StandardOutput arguments:1) (JavaSymbolTable:RemoveOneEntryFromSymbolTable symbol_table wrapper_declaration arguments:1 removed_symbol_space) );; )local )action )define Removing Dead Code I private rule remove_deactive_method_declaration_7(mm:method_modifiers,id:IDENTIFIER, p:parameters,b:brackets,bb:block_body) :class_body_declaration->class_body_declaration = "\mm void \id \p \b { \bb }" -> ";" with side-effect remove_entry_from_symbol_table(id) if is_deactive_declaration(id). Transform Procedural Helpers Using Symbol Table

39. © Semantic Designs, Inc. 39 5/26/2016 Removing Dead Code II package edu.ksu.cis.test; public class SuperClass{ protected transient int yy =1; void outer() {System.out.println("I'm outer"); new AB();} } class AB { SuperClass s = new SuperClass(); //AB ab = new AB(); class SuperClass{ void inner() {System.out.println("I'm inner");} } void callmethod(){s.inner();} } package edu.ksu.cis.test; public class Test extends SuperClass{ void m(){ // called from main Test t = new Test(); Test t2 = Test.this; t2.yy = 3; } class A { void m2(){ Test t = new Test(); t.yy =2; } class B {} SuperClass.java Test.java BeforeAfter package edu.ksu.cis.test; public class SuperClass { protected transient int yy = 1; } package edu.ksu.cis.test; public class Test extends SuperClass { void m() { // called from main Test t2 = Test.this ; t2.yy = 3; } Remove symbol table entry for node : Line 17 Column 1 File d:/users/hzheng/JavaTestSuite/first/edu/ksu/cis/test/Test.java Remove symbol table entry for node : Line 12 Column 10 File d:/users/hzheng/JavaTestSuite/first/edu/ksu/cis/test/Test.java Remove symbol table entry for node : Line 11 Column 10 File d:/users/hzheng/JavaTestSuite/first/edu/ksu/cis/test/Test.java Remove symbol table entry for node : Line 10 Column 1 File d:/users/hzheng/JavaTestSuite/first/edu/ksu/cis/test/Test.java Remove symbol table entry for node : Line 4 Column 11 File d:/users/hzheng/JavaTestSuite/first/edu/ksu/cis/test/Test.java Remove symbol table entry for node : Line 12 Column 10 File d:/users/hzheng/JavaTestSuite/first/edu/ksu/cis/test/SuperClass.java Remove symbol table entry for node : Line 10 Column 14 File d:/users/hzheng/JavaTestSuite/first/edu/ksu/cis/test/SuperClass.java Remove symbol table entry for node : Line 7 Column 16 File d:/users/hzheng/JavaTestSuite/first/edu/ksu/cis/test/SuperClass.java Remove symbol table entry for node : Line 6 Column 1 File d:/users/hzheng/JavaTestSuite/first/edu/ksu/cis/test/SuperClass.java Remove symbol table entry for node : Line 4 Column 10 File d:/users/hzheng/JavaTestSuite/first/edu/ksu/cis/test/SuperClass.java

40. © Semantic Designs, Inc. 40 5/26/2016 Java O-O Refactoring · Many useful refactorings: move-method, generalize-method,... · Fowler view: teach programmers, implement manually  DMS view: mechanical transforms --> CAPTURE IN TOOL! –“Refactoring Object-oriented Frameworks” (Opdyke92) –C++ refactoring (Tokuda98/99) –http://st-www.cs.uiuc.edu/~brant/RefactoringBrowser/Brant98 (Rewrite Rule Editor) “… changing a software system in such a way that it does not alter the external behavior of the code, yet improves its internal structure” Refactoring (Martin Fowler,Addison-Wesley, 1999) Extract Method, p. 110 void printOwing(double amount) { printBanner(); // print details System.out.println(“Name:” = _name); System.out.println(“Amount” + amount); } void printOwing(double amount) { printBanner(); printDetails(amount); } void printDetails (double amount) { System.out.println(“Name:” = _name); System.out.println(“Amount” + amount); }

41. © Semantic Designs, Inc. 41 5/26/2016 “Extract Method” as a Mechanical Transform Mix of DMS RSL pattern matching and procedure default base domain Java. rule ExtractMethod(...) "\parent_class_modifiers \class_header { \some_declarations \method_modifiers \type \method_name \parameters \brackets \throw_list { \leader_statements \extractable_statements \follower_statements } \more_declarations }":class_body rewrites to "\parent_class_modifiers \class_header { \some_declarations \extracted_method(\extractable_statements) \method_modifiers \type \method_name \parameters \brackets \throw_list { \leader_statements \new_methodcall(\extractable_statements) \follower_statements } \more_declarations }" if extractable(\leader_statements,\extractable_statements) -- no bad side effects -- Properties of computed new method: -- method name not member parameter_names(\class_header) or declared(\leader_statements) -- method throw list = exceptions_escaping(\extractable_statements) -- declarations = declared(\leader_statements) & escapes(\executable_statements) -- argumentnames = (parameter_names(\class_header) union declared(\leader_statements)) intersect reads(\executable_statements) -- argumentbindings =... Computes new method Computes call to method Java pattern

42. © Semantic Designs, Inc. 42 5/26/2016 OO Promise inhibited by Scale Typical 21st Century Application: IBM’s “San Francisco” (ERP) –2 million lines Java –9000 Class definitions for ERP framework Scale refactoring by manual methods Very Difficult –Impractical to carry out thoroughly and reliably –Failure to refactor -> Framework evolution frozen -> Future ? Scalable re-engineering tools will become necessary Transformation systems can provide that foundation –Can support huge libraries of Refactoring operations (Fowler99) –Can support other useful transforms (optimizations, etc.)

43. © Semantic Designs, Inc. 43 5/26/2016 XML Parser Generation XML enables Electronic Data Interchange –Neutral form for moving information between systems Problem: Need System1 to XML to System2 translators –For arbitrary data –Must be fast to support high-volume EDI Typical solution: Use Standard XML -> DOM reader –Incomplete: Doesn’t solve “to XML” problem –Slow: Parse arbitrary XML, validate against DTD schema –Slow: Procedural interface interpreting DTD for data access –Clumsy: DOM calls clutters application code Idea: Generate DTD-specific XML parser/generators –Produce application-target-language code specific to DTD –Free validation; XML data in direct-access native data structures  Faster parsing/processing, Easier application coding

44. © Semantic Designs, Inc. 44 5/26/2016 Sample orderform Wiley Coyote Dinner, Inc. 1 Mesa Highway Southwest Park 98765 RocketSkates 2 29.95 BirdSeed 2000.01 <!DOCTYPE orderform [ ]> Simple XML DTD for OrderForm items is “list of item ”

45. © Semantic Designs, Inc. 45 5/26/2016 Java Code Generation Plan For each DTD element: –Produce an “element” class to hold data for that element PCDATA for leaves Class references for non-leaves –Produce an element-specific parsing procedure –Produce an element-specific unparsing produre Handle sequences as array of element class references Handle choices as class reference + “which alternative” integer (1..n) name Parse() { while … sequence[i]= field. Parse(); } class XML_Element_ name implements Union { public Union sequence[] = new Union[]; }

46. © Semantic Designs, Inc. 46 5/26/2016 Sample Code Generation Transforms pattern CLASS_SEQUENCE(name:NAME,sequence:cp_sequence):class_body_declarations@Java = "\JavaClassName\(\name\) Parse() throws XML.NotValidForDTD { if (!XML.QueryOpeningTag(\JavaXMLOpenTagString\(\name\))) return null; \JavaClassName\(\name\) result=new \JavaClassName\(\name\)(); { \SEQUENCE_FETCH\(\sequence\,\SEQUENCE_LENGTH\(\sequence\)\,sequence1\) } XML.RequireClosingTag(\JavaXMLCloseTagString\(\name\)); return result; } class \JavaClassName\(\name\) extends XML_IO implements Union { \XML_PARSER_DECLARATIONS_FOR_CP_SEQUENCE\(\sequence\,1\,\>XML\:NAME \%contentspec sequence1 \<\:NAME\) // produces nested subclasses public Union sequence[]= new Union[\SEQUENCE_LENGTH\(\sequence\)]; public Generate() { XML.WriteOpeningTag(\JavaXMLOpenTagString\(\name\)); for (i=1;i<=\SEQUENCE_LENGTH\(\sequence\);i++) sequence[i].Generate(); XML.WriteClosingTag(\JavaXMLOpenTagString\(\name\)); }} ". rule refine_ELEMENT_sequence_to_class(name:NAME,sequence:cp_sequence) :elementdecl -> class_body_declarations = "\%markupdecl " -> CLASS_SEQUENCE(name,sequence). Generates class Generates Parser method About 100 rules ~~ 2000 lines for all transforms

47. © Semantic Designs, Inc. 47 5/26/2016 Generated Java code for items class XML_Element_item extends XML_IO implements Union { public Union sequence[] = new Union[3]; } XML_Element_item Parse() throws XML.NotValidForDTD { if ( !XML.QueryOpeningTag("item")) return null; XML_Element_item result = new XML_Element_item(); { if ((sequence[1] = XML_Element_partnumber.Parse()) == null) throw XML.NotValidForDTD; if ((sequence[2] = XML_Element_quantity.Parse()) == null) throw XML.NotValidForDTD; if ((sequence[3] = XML_Element_unitprice.Parse()) == null) throw XML.NotValidForDTD; } XML.RequireClosingTag("item"); return result; } class XML_Element_items extends XML_IO implements Union { public Union sequence[] = new Union[1]; } } XML_Element_items Parse() throws XML.NotValidForDTD { if ( !XML.QueryOpeningTag("items")) return null; XML_Element_items result = new XML_Element_items(); { if ((sequence[1] = XML_Element_item.Parse()) == null) throw XML.NotValidForDTD; } XML.RequireClosingTag("items"); return result; }

48. © Semantic Designs, Inc. 48 5/26/2016 Generated Java code for item content: partnumber, quantity, unitprice class XML_Element_unitprice extends XML_IO implements Union { public String PCDATA; } XML_Element_unitprice Parse() throws XML.NotValidForDTD { if (XML.QueryOpeningTag("unitprice")) return null; XML_Element_unitprice result = new XML_Element_unitprice(); result.PCDATA = XML.AcceptNonEmptyPCDATA(); XML.RequireClosingTag("unitprice"); return result; } class XML_Element_partnumber extends XML_IO implements Union { public String PCDATA; } } XML_Element_partnumber Parse() throws XML.NotValidForDTD { if (XML.QueryOpeningTag("partnumber")) return null; XML_Element_partnumber result = new XML_Element_partnumber(); result.PCDATA = XML.AcceptNonEmptyPCDATA(); XML.RequireClosingTag("partnumber"); return result; } class XML_Element_quantity extends XML_IO implements Union { public String PCDATA; } } XML_Element_quantity Parse() throws XML.NotValidForDTD { if (XML.QueryOpeningTag("quantity")) return null; XML_Element_quantity result = new XML_Element_quantity(); result.PCDATA = XML.AcceptNonEmptyPCDATA(); XML.RequireClosingTag("quantity"); return result; }

49. © Semantic Designs, Inc. 49 5/26/2016 Orderform orderform =new Orderform.Parse(); // exception thrown if invalid w.r.t. DTD { Print(“Customer: ”); Print(orderform.name.firstname.PCDATA);Print(“ “); Print(orderform.name.lastname.PCDATA); } { Print(“Company: ”); Print(orderform.company.PCDATA); PrintNewline(); } { Print(“Address: ”); Print(orderform.address.street.PCDATA); PrintNewLine(); Print(orderform.address.city.PCDATA); PrintNewline(); if (orderform.addreess.region.which=1) Print(orderform.address.region.zipcode.PCDATA); else Print(orderform.address.region.country.PCDATA); PrintNewline(); } Print (“ITEMS” Product Quantity Cost Extension”); float Total=0; for (item=1;item<length(orderform.items.item);1) { PrintNumber(item); PrintTab(); Print(order.items.sequence[item].partnumber.PCDATA); PrintTab(); Print(order.items.sequence[item].quantity.PCDATA); PrintTab(); Print(order.items.sequence[item].price.PCDATA); PrintTab(); float extension=Value(order.items[item].quantity.PCDATA) *Value(order.items[item].price.PCDATA); PrintNumber(extension);PrintNewline(); Total+=extension; } Print “Invoice total”;PrintNumber(Total);PrintNewline(); Application Code to Print Orderform Note direct access to XML data

50. © Semantic Designs, Inc. 50 5/26/2016 Preliminary performance From http://www.pankaj-k.net/xpb4j/docs/Measurements-May30/ measurements-May30-2002.html#MeasurementProcess 1.Comparison among DOM Parsers: … for XML parsing rate … 30ms SD DTDtoJava orderform.class 8kb XML.class 24kb

51. © Semantic Designs, Inc. 51 5/26/2016 Partial Evaluation “PE”: Faster Computation using Context Example –We want to repeatedly compute X+Y+4 –Somehow we find out: X=7 all the time –We can compute 7+Y+4 –Better, we can compute Y+11 Partial Evaluation (implemented using transforms) –Find a program fragment with variables –Find a constant variable value (computation in constant context) –Substitute that value into program fragment –Simplify the result Spectacular examples: –JIT=PE(JavaInterpreter,JavaProgram) --> CompiledJava –PE(XSLInterpreter,XSLProgram) --> CompiledXSL

52. © Semantic Designs, Inc. 52 5/26/2016 Cost of Running an HTML server using XSLT interpreter Big E-commerce site –1 million visits/month –Each visit uses 100 “clicks” –Each click displays 1 XML screen-page as HTML MS IE 5.5: 1 screen-page in.075 sec (Time measured on 4 way SMP 200Mhz Ppro) –MSXML core component of IE5.5 –33300 * 100 *.075 sec --> 3 CPU days/elapsed day –Servers have lots of other work to do!

53. © Semantic Designs, Inc. 53 5/26/2016 Read XSL driving XML->HTML conversion Compile XSL to “C” code using transforms Why? Compiled code typ. 100x interpreted code (MSXML) –C structs instead of DOM nodes –C code instead of XSL paths –C code instead of JavaScript idioms Run compiled “C” code instead of IE5/MSXML –Estimated runtime: 306 seconds (vs. 3 CPU days for IE5) –Other XSL processors may be faster than IE5 Still probably “interpretive” --> slow Major savings in server CPU –fewer servers (lower cost to maintain) –fast response (better customer experience) Compile “XML -> HTML” for speed Backup Spreadsheet

54. © Semantic Designs, Inc. 54 5/26/2016 XSL Compiler using DMS Analyze Generate/ Transform Parse (read) Pretty Print XSL C XML Language Definition Task Definition C Transforms XML Patterns C Language Definition

55. © Semantic Designs, Inc. 55 5/26/2016 DMS Scale Application: Clone Removal Original System with code clones 1 M SLOC DeCloned System with automatic names 10-20% reduction // sort array A for (I=1,I<10,I++) for (j=i,j>1,J--) if (A[j]>A[j-1]) swap(A[I],A[J]); for (I=1, I<2*Q, I++) for (I1=i, I1>1, I1--) // exchange if less if ( K[I1] > K[I1-1] ) swap( K[I], K[I1] ); // sort my data for (z=1,z<1000,z++) for (j=i,j>1,J--) if (D[j]>D[j-1]) swap(D[z], D[J]); // sort array A Clone27(I,10,j,A); Clone27(I,2*Q,I1,K); // sort my data Clone27(z,1000,j,D); #define Clone27(a,b,c,d)\ for (a=1,a<b,a++)\ for (c=a,c>1,c--)\ if (d[c]>d[c-1])\ swap(d[a],d[c]); for (a=1,a<b,a++) for (c=a,c>1,c--) if (d[c]>d[c-1]) swap(d[a],d[c]); Skeleton of detected clones... code block 1...... code block 2...... code block 3...... code block 4...... code block 1...... code block 2...... code block 3...... code block 4...

56. © Semantic Designs, Inc. 56 5/26/2016 COBOL Clone Detection/Removal Find Clones by matching every program fragment (AST) to every other –Expensive! California Community Colleges application –Course Inventory Construction for each campus Parameters: –77,000 SLOC ANSI COBOL 85 --> 774,645 AST nodes 128 Second Parse time, 40 minutes for clone detection 500 Mb RAM, 6 CPUs –Number of exact clone pairs = 78, near miss pairs = 95 –Largest clone: 5 copies, 1017 lines each, *1* parameter! – Number of cloned lines = 30727 --> can remove 15363 SLOC reduction by removing clones = 19.7% Getting ready to try on 2M SLOC Java code…

57. © Semantic Designs, Inc. 57 5/26/2016 Sample COBOL Clones Similarity =.99178082191781 from 35179 to 35204 file = example.cbl 9700-OUTPUT-REPORT-TOTALS. MOVE CURRENT-COLLEGE-ID TO REPORT-CODE1 REPORT-CODE2 REPORT-CODE3. SET EDIT-ERROR-LITERAL-INDEX TO 1. SET DISTRICT-COUNT-ROW-INDEX TO 1. PERFORM 9710-OUTPUT-TOTALS1 UNTIL EDIT-ERROR-LITERAL-INDEX > 30. PERFORM 9720-OUTPUT-TOTALS2. PERFORM 9730-OUTPUT-TOTALS3. 9710-OUTPUT-TOTALS1. MOVE REPORT-SUM1 TO TOTALS-ID1. MOVE ELEMENT-NUMBER (EDIT-ERROR-LITERAL-INDEX) TO DED-NUMBER. MOVE DISTRICT-COUNT (DISTRICT-COUNT-ROW-INDEX 1) TO EXCEPT-COUNT. MOVE DISTRICT-COUNT (DISTRICT-COUNT-ROW-INDEX 2) TO UNKNOWN-COUNT. MOVE DISTRICT-COUNT (DISTRICT-COUNT-ROW-INDEX 3) TO REASON-COUNT. MOVE DISTRICT-COUNT (DISTRICT-COUNT-ROW-INDEX 4) TO GRP3-COUNT. WRITE REPORT-TOTALS-RECORD1. SET EDIT-ERROR-LITERAL-INDEX UP BY 1. SET DISTRICT-COUNT-ROW-INDEX UP BY 1. ------------------------------------ from 15368 to 15393 file = example.cbl 9700-OUTPUT-REPORT-TOTALS. MOVE HOLD-DISTRICT-ID TO REPORT-CODE1 REPORT-CODE2 REPORT-CODE3. SET EDIT-ERROR-LITERAL-INDEX TO 1. SET DISTRICT-COUNT-ROW-INDEX TO 1. PERFORM 9710-OUTPUT-TOTALS1 UNTIL EDIT-ERROR-LITERAL-INDEX > 17. PERFORM 9720-OUTPUT-TOTALS2. PERFORM 9730-OUTPUT-TOTALS3. 9710-OUTPUT-TOTALS1. MOVE REPORT-SUM1 TO TOTALS-ID1. MOVE ELEMENT-NUMBER (EDIT-ERROR-LITERAL-INDEX) TO DED-NUMBER. MOVE DISTRICT-COUNT (DISTRICT-COUNT-ROW-INDEX 1) TO EXCEPT-COUNT. MOVE DISTRICT-COUNT (DISTRICT-COUNT-ROW-INDEX 2) TO UNKNOWN-COUNT. MOVE DISTRICT-COUNT (DISTRICT-COUNT-ROW-INDEX 3) TO REASON-COUNT. MOVE DISTRICT-COUNT (DISTRICT-COUNT-ROW-INDEX 4) TO GRP3-COUNT. WRITE REPORT-TOTALS-RECORD1. SET EDIT-ERROR-LITERAL-INDEX UP BY 1. SET DISTRICT-COUNT-ROW-INDEX UP BY 1. #2 ==> 30 ==> 17 --------- #1 ==> CURRENT-COLLEGE-ID ==> HOLD-DISTRICT-ID #1 #2 Report by College-ID Report by District-ID

58. © Semantic Designs, Inc. 58 5/26/2016 The generated COPYLIB 9700-OUTPUT-REPORT-TOTALS. MOVE PARAMETER-1 TO REPORT-CODE1 REPORT-CODE2 REPORT-CODE3. SET EDIT-ERROR-LITERAL-INDEX TO 1. SET DISTRICT-COUNT-ROW-INDEX TO 1. PERFORM 9710-OUTPUT-TOTALS1 UNTIL EDIT-ERROR-LITERAL-INDEX > PARAMETER-2. PERFORM 9720-OUTPUT-TOTALS2. PERFORM 9730-OUTPUT-TOTALS3. 9710-OUTPUT-TOTALS1. MOVE REPORT-SUM1 TO TOTALS-ID1. MOVE ELEMENT-NUMBER ( EDIT-ERROR-LITERAL-INDEX ) TO DED-NUMBER. MOVE DISTRICT-COUNT ( DISTRICT-COUNT-ROW-INDEX 1 ) TO EXCEPT-COUNT. MOVE DISTRICT-COUNT ( DISTRICT-COUNT-ROW-INDEX 2 ) TO UNKNOWN-COUNT. MOVE DISTRICT-COUNT ( DISTRICT-COUNT-ROW-INDEX 3 ) TO REASON-COUNT. MOVE DISTRICT-COUNT ( DISTRICT-COUNT-ROW-INDEX 4 ) TO GRP3-COUNT. WRITE REPORT-TOTALS-RECORD1. SET EDIT-ERROR-LITERAL-INDEX UP BY 1. SET DISTRICT-COUNT-ROW-INDEX UP BY 1.

59. © Semantic Designs, Inc. 59 5/26/2016 Source file change MOVE DISTRICT-INT-CNT (11) TO INT-CNT-OUT-B. WRITE PRINT-RECORD-2 FROM INTEGRITY-ERROR-B AFTER ADVANCING 2 LINES. MOVE DISTRICT-INT-CNT (12) TO INT-CNT-OUT-C. WRITE PRINT-RECORD-2 FROM INTEGRITY-ERROR-C AFTER ADVANCING 2 LINES. 9700-OUTPUT-REPORT-TOTALS. MOVE HOLD-DISTRICT-ID TO REPORT-CODE1, REPORT-CODE2, REPORT-CODE3. SET EDIT-ERROR-LITERAL-INDEX TO 1. SET DISTRICT-COUNT-ROW-INDEX TO 1. PERFORM 9710-OUTPUT-TOTALS1 UNTIL EDIT-ERROR-LITERAL-INDEX > 17. PERFORM 9720-OUTPUT-TOTALS2. PERFORM 9730-OUTPUT-TOTALS3. 9710-OUTPUT-TOTALS1. MOVE REPORT-SUM1 TO TOTALS-ID1. MOVE ELEMENT-NUMBER (EDIT-ERROR-LITERAL-INDEX) TO DED-NUMBER. MOVE DISTRICT-COUNT (DISTRICT-COUNT-ROW-INDEX, 1) TO EXCEPT-COUNT. MOVE DISTRICT-COUNT (DISTRICT-COUNT-ROW-INDEX, 2) TO UNKNOWN-COUNT. MOVE DISTRICT-COUNT (DISTRICT-COUNT-ROW-INDEX, 3) TO REASON-COUNT. MOVE DISTRICT-COUNT (DISTRICT-COUNT-ROW-INDEX, 4) TO GRP3-COUNT. WRITE REPORT-TOTALS-RECORD1. SET EDIT-ERROR-LITERAL-INDEX UP BY 1. SET DISTRICT-COUNT-ROW-INDEX UP BY 1. 9720-OUTPUT-TOTALS2. MOVE REPORT-SUM2 TO TOTALS-ID2. MOVE DISTRICT-INT-CNT (1) TO INTEGRITY-ERROR-COUNT. MOVE '01' TO INTEGRITY-ERROR-CODE. WRITE REPORT-TOTALS-RECORD2. MOVE REPORT-SUM2 TO TOTALS-ID2. MOVE DISTRICT-INT-CNT (2) TO INTEGRITY-ERROR-COUNT. MOVE '02' TO INTEGRITY-ERROR-CODE. MOVE DISTRICT-INT-CNT (11) TO INT-CNT-OUT-B. WRITE PRINT-RECORD-2 FROM INTEGRITY-ERROR-B AFTER ADVANCING 2 LINES. MOVE DISTRICT-INT-CNT (12) TO INT-CNT-OUT-C. WRITE PRINT-RECORD-2 FROM INTEGRITY-ERROR-C AFTER ADVANCING 2 LINES. COPY CDR_clone10 REPLACING PARAMETER-1 BY == HOLD-DISTRICT-ID == PARAMETER-2 BY == 17 == 9720-OUTPUT-TOTALS2. MOVE REPORT-SUM2 TO TOTALS-ID2. MOVE DISTRICT-INT-CNT (1) TO INTEGRITY-ERROR-COUNT. MOVE '01' TO INTEGRITY-ERROR-CODE. WRITE REPORT-TOTALS-RECORD2. MOVE REPORT-SUM2 TO TOTALS-ID2. MOVE DISTRICT-INT-CNT (2) TO INTEGRITY-ERROR-COUNT. MOVE '02' TO INTEGRITY-ERROR-CODE.

60. © Semantic Designs, Inc. 60 5/26/2016 Clone Detection/Removal Statistics Application Process Control PARLANSE Compiler CCC Registration Roccade Accounting SLC Insurance SWING Toolkit Language C C COBOL85 IBM COBOL PROGRESS Java SLOC 427921 42413 77631 567798 356413 241460 Removable SLOC (est.) 54346 12.7% 4133 9.7% 15500(39967) 20.0 (51.5)% 130494 23% 58783 16.5% 26355 10.9% Removed SLOC N.A. (pre removal capability for C 6/98) 2568 6.0% 15034 19.9% N.A. (not yet performed)

61. © Semantic Designs, Inc. 61 5/26/2016 Typical Porting Scenarios JOVIAL73 on MIL1750  C on PowerPC –Military Avionics + Weapons management COBOL74 + IDMS  COBOL85 + SQL –UNISYS 1100 retirement; must move data, too! K&R C + custom RTOS  ANSI C + VXworks –Microprocessor modernization Clipper + green screen  Delphi + GUI –Legacy 3GL data processing language MODCOMP ASM  C –Defense Radar modernization; 12 computer languages! Verilog  VHDL –Reuse of Chip Design in new context

62. © Semantic Designs, Inc. 62 5/26/2016 How DMS handles Porting Accepts definitions of source, target and design languages –Syntax, Semantics, Optimization Transforms and Analysis rules Accepts specifications of (porting) transformations –Written in terms of the language syntax, conditioned by analyses –Source-language idioms often map directly to Target-language idioms –Transforms for Complex idioms/OS/Library calls abstracted to design languages, then refined to target languages Parses entire source system (thousands of files!) Apply Porting transformations, then Optimizing transforms PrettyPrints the results in compilable target language form Test Result using Application Regression Test Revise transforms and repeat till done

63. © Semantic Designs, Inc. 63 5/26/2016 default source domain Jovial; default target domain C; private rule refine_data_reference_dereference_NAME (n1:identifier@C,n2:identifier@C) :data_reference->expression = "\n1\:NAME @ \n2\:NAME" -> "\n2->\n1". private rule refine_for_loop_letter_2 (lc:identifier@C,f1:expression@C, f2:expression@C,s:statement@C) :statement->statement = "FOR \lc\:loop_control : \f1\:formula BY \f2\:formula; \s\:statement“ -> "{ int \lc = (\f1); for(;;\lc += (\f2)) { \s } }“ if is_letter_identifier(lc). A few DMS porting transforms Jovial to C Domain Name Source Domain Syntax Target Domain Syntax Pattern Variables

64. © Semantic Designs, Inc. 64 5/26/2016 Porting Transforms in Action Jovial to C JOVIAL Source: FOR i: j*3 BY 2 ; x@mydata = x@mydata+I; Translated C Result: { int i = j*3; for (;;i+=2) { mydata->x = mydata->x + i} Typically lots of small transforms for full translation ~1500 rules to translate full Jovial

65. © Semantic Designs, Inc. 65 5/26/2016 A More Complex Example Jovial to C START TABLE TFP'D'TWRDET (1:109,12:37); BEGIN % Main status boolean % ITEM TFP'G'TWRDET STATUS (V(YES),V(NO)); END TYPE TFP'D'TWRDET'TABLE TABLE (7:23) W 3; BEGIN ITEM TFP'ITM S 3 POS(0,3); "cube axis" END %begin proc% PROC PROC'A(c1) S; BEGIN ITEM match'count U 6; %an item% ITEM c1 C 5; "parameter value" ITEM c2 C 7; IF c1 c1; match'count = UBOUND(TFP'D'TWRDET,0) + UBOUND(TFP'D'TWRDET'TABLE,0); "result off by 1 so adjust" match'count = match'count+1; BEGIN match'count=match'count/2; PROC'A = match'count; % return answer % END "cleanup and exit"; END "end proc" TERM #include "jovial.h" static struct { /* Main status boolean */ enum { V(yes$OF$tfp_g_twrdet$OF$tfp_d_twrdet), V(no$OF$tfp_g_twrdet$OF$tfp_d_twrdet) } tfp_g_twrdet _size_as_word; } tfp_d_twrdet[109][26]; typedef union { W(3); struct { POS(0, 3) S(3) tfp_itm:4 _align_to_bit; /* cube axis */ }; } tfp_d_twrdet_table[17]; static S proc_a(C(5) c1); /* begin proc */ static S proc_a(C(5) c1) { __typeof__(proc_a(c1)) RESULT(proc_a); _main: { U(6) match_count; C(7) c2; if (CHARACTER_COMPARE(BYTE_CONVERT(C(7), c1, 7), c2) <= 0 && CHARACTER_COMPARE(c2, BYTE_CONVERT(C(7), c1, 7)) > 0) match_count = UBOUND(tfp_d_twrdet, 2, 0) + 16; /* result off by 1 so adjust */ match_count = (S(6))match_count + 1; { match_count = (S(6))match_count / 2; RESULT(proc_a) = (S(6))match_count; /* return answer */ } /* cleanup and exit */ ; } _return: return RESULT(proc_a); } /* end proc */ packed tables with bit offsets, typedefs, functions, string operations, comments Equivalent C (used with hand-coded macro library)

66. © Semantic Designs, Inc. 66 5/26/2016 Porting Process Run DMS Port Develop Porting Transforms Test Ported System Success Errors Analyze Existing Software transforms Ported Code Continuing Application Development Code 1 M SLOC DMS Engineer Repeated Port Cycles No Impact On Development!

67. © Semantic Designs, Inc. 67 5/26/2016 Porting by DMS is practical Enabled by generalized compiler technology –Requires: Specification of source, target and design languages Specification of inspectable transforms –Automates: Source file parsing and prettyprinting of results Application of sets of transforms –Scalable, fast, repeatable Transform thousands of files/millions of lines in one day Iterative development/testing of porting transforms/result Organizational benefits –Development team not disturbed by porting team –Application functionality preserved  happy users! –Far more cost effective than hand translation

68. © Semantic Designs, Inc. 68 5/26/2016 DMS: Conclusion Useful to automate analysis/modification of programs –Many possible custom reengineering possibilities –A key technology for software quality improvement Need generalized compiler-like infrastructure –Definable parsers, prettyprinters, transforms –Must scale to application systems with MSLOCs DMS provides these capabilities –on path to design capture/analysis/modification –growing infrastructure and standard language modules www.semdesigns.com

69. © Semantic Designs, Inc. 69 5/26/2016 Worst-Case Time/Space Analysis Define basis set of code fragments –Grammar + selected idioms –Calibrate space/time costs Annotate source code with limits –Maximum sizes, loop iterations Compute call Graph Compute costs over subroutines –“Tile” program with basis set –Estimate cost in terms in input values –Bottom up WRT to call graph

70. © Semantic Designs, Inc. 70 5/26/2016 Where can DMS be applied? Program modification –Application evolution Functionality change Performance change Technology change –Massive changes Porting: new language, target APIs,... Restructuring: Clone removal, Y2K fix,... Optimization: Dead code, parallelize,... –Customize reusable component in new context Program Analysis –Metrics SLOC, conditional, complexity –Organization style checking –Programming information extraction Clones, Slices, Call Graphs, Side effects –Domain information extraction Business rules Idiom recognition –Semantic Faults erroneous/dead/useless code Domain-specific program generation –Partial Differential Equation solvers –Factory control synthesis –Entity-Relationship compilers DB conversion generators –Protocol compilers –Automated Test Generation Legacy code reverse engineering –design recovery to domain abstractions aid code understanding enable application evolution –Incremental design capture –reusable component extraction –component extraction for domains –Legacy mergers Unify data schemas Modify applications Convert existing data –Business rule extraction Make explicit, easy to read/change