1. Things Structural Clones Tell that Simple Clones Don’t Hamid Abdul Basit 1

2. Software Clones Simple clones – the same or similar code fragments Structural clones – higher-level, larger similarities – Similarity of code and similarity of structure 2

3. Simple clones 3

4. 4 Structural Clones

5. CreateTask.BL ValidateTask() Task.DB AddTask() Task Table accesses Project.DB AddProject() Project Table executes CreateProject.BL ValidateProject() accesses executes visualizes User Interface Business Logic Database CreateTask CreateProject CreateTask.UI CreateTask() CreateProject.UI CreateProject() Collaborative structural clone Structural Clones 5

6. When are structural clones useful? showing a bigger picture of similarity situation - the forest from the trees Finding refactoring opportunities Architecture recovery, program understanding and maintenance – Structural clones often represent application domain or design concepts Re-engineering for reuse – The bigger the clones the better for reuse Some benefits for plagiarism detection 6

7. 7 A structure is a graph A D C B xy z w Entities {A,B,C,D} Relationships {w,x,y,z}

8. Entities Physically defined – Code fragments, files, web pages, directories Semantically defined – Methods, classes, packages Conceptually defined – Components, sub-systems 8

9. Relationships Physical co-location – Same file, same directory Runtime – Message passing – Hyperlink between web pages Design level – Inheritance – Association – Composition 9

10. e 8b e 6b x w z x z e 7b x y z e 8d y e 6d y e 7d y e 8c e 6c e 7c e 8a S8 e 6a S6 e 7a S7 e 4b x y z e 5b x y z e 4d y e 5d y e 4c e 5c e 4a S4 e 5a S5 Structural Clones 10

11. Higher level similarities are composed of lower level similarities Can be recovered by finding repeating configurations of lower level similarities 11 Observation

12. Detecting Structural Clones 12 Simple clones in files clone patterns 1 9876 5432 13 1214 10 1115 1,4,8,10,11,12 1,4,7,8,10,11,12 2,5,9,13,15

13. Detecting Structural Clones 13 File Analysis 1 8 1110 12 4 7135 9 15 2

14. Detecting Structural Clones 14 Directory Analysis 18111012 47135 9 15 2

15. Detecting Structural Clones 15 File Level Structural Clone Across Directories 110487 1513925

16. a1 b1 c1 a2 b2 c2 F1 F2 Simple Clone Structure (SCS) Across Files 16

17. F1 F2 File Clone Class (FCC) 17

18. File Clone Structure (FCS) Across / Within Directories 18 F1 X1Z2 F2 X2 Y1Y2 D1 D2 Z1

19. STRUCTURAL CLONES DETECTED BY CLONE MINER Simple Clone Structures (SCS) Across Methods Simple Clone Structures (SCS) Across Files Method Clone Classes (MCC) Method Clone Structures (MCS) Across Files File Clone Classes (FCC) File Clone Structures (FCS) Across Directories File Clone Structures (FCS) Within Directories 19

20. In earlier work We hypothesized the benefits of structural clones – Re-engineering for reuse, architecture recovery, … Defined structural clones Implemented Clone Miner – structural clone detector Did initial empirical evaluation 20

21. In work presented here How frequent are the different types of structural clones? Are structural clones more meaningful for program understanding and design recovery than simple clones? What is the value added by structural clone detection in identifying refactoring opportunities? 21

22. Case Study Systems 22 Apache Ivy http://ant.apache.org/ivy/index.html Apache Ant http://ant.apache.org/ Columba http://sourceforge.net/projects/columba/ Dnsjava http://www.dnsjava.org/ Javax-Swing http://www.oracle.com/technetwork/java/javase/downloa ds/index.html JFreeChart http://www.jfree.org/jfreechart/ ANTLR http://www.antlr.org/ DrJava http://www.drjava.org/ FreeCol http://www.freecol.org/ JEdit http://www.jedit.org/ JHotDraw http://www.jhotdraw.org/

23. Apache -IvtApache-AntColumbaDnsJavaJavax-SwingJFreechartAntlr Tokens240711711914569676137019960906505448243461 Files38611741549182620585258 Lines Of Code542352246421570003008428916019835161414 Methods415612849833420241525178423618 AntlrDrJavaFreeColJEditJHotdrawAVERAGETOTAL Tokens2434616157395187955361731649294731614964060 Files2586054885314846246476 Lines Of Code61414140571128854157079633641367961450519 Methods36187831589268734859723075373 Systems’ Overview 23

24. AVERAGETOTAL SCC127513231 SCC Instances412542924 Average Length of SCC Instance48 Methods Containing SCC153216198 % Methods Containing SCC22 Files Containing SCC3323476 % Files Containing SCC53 Directories Containing SCC51504 % Directories Containing SCC83 Simple Clone Classes (SCC) 24

25. % Methods Containing SCC 25

26. How frequent are the different types of structural clones? 26 1

27. Structural Clones are Frequent Simple clones tend to occur in groups – 56% of simple clones are within structural clones There are less structural clones than simple clones 27

28. Simple Clone Classes (SCC) and Simple Clone Structures (SCS) 28

29. AVERAGETOTAL SCS7277257 SCS Instances256725798 Average Instances In SCS3 Average Token Count (ATC) in SCS87 Average % Cover(APC) in SCS9 SCC Covered By SCS7037204 % SCC Covered By SCS54 Methods Containing SCS100910665 % Methods Containing SCS14 Files Containing SCS2983122 % Files Containing SCS48 Directories Containing SCS48467 % Directories Containing SCS80 Simple Clone Structures (SCS) Across Files 29

30. % SCC Covered By SCS Across Files 30

31. AVERAGETOTAL SCS3063229 SCS Instances115812362 Average Instances In SCS3 SCC Covered By SCS6947278 % SCC Covered By SCS55 Methods Containing SCS105411093 % Methods Containing SCS15 Files Containing SCS2122216 % Files Containing SCS36 Directories Containing SCS41413 % Directories Containing SCS72 Simple Clone Structures (SCS) Within Files 31

32. % Files Containing SCS Within Files 32

33. AVERAGETOTAL SCS106710996 SCS Instances378839398 Average Instances In SCS3 Average Token Count (ATC) in SCS55 Average %Cover(APC) in SCS53 SCC Covered By SCS108111168 % SCC Covered By SCS86 Methods Containing SCS330035666 % Methods Containing SCS21 Files Containing SCS3193344 % Files Containing SCS52 Directories Containing SCS50492 % Directories Containing SCS82 Simple Clone Structures (SCS) Across Methods 33

34. % SCC Covered By SCS Across Methods 34

35. Method Clone Classes (MCC) and Method Clone Structures (MCS) 35

36. AVERAGETOTAL MCC1651760 MCC Instances4755068 Average Instances In MCC2 SCC Covered By MCC2963139 % SCC Covered By MCC23 Methods Covered By MCC4755068 % Methods Covered By MCC7 Files Containing MCC1962066 % Files Containing MCC31 Directories Containing MCC39385 % Directories Containing MCC66 Method Clone Classes (MCC) 36

37. AVERAGETOTAL MCS22230 MCS Instances58612 Average Instances In MCS2 MCC Covered By MCS45484 % MCC Covered By MCS4 Methods Covered By MCS1621740 % Methods Covered By MCS2 Files Containing MCS36384 % Files Containing MCS6 Directories Containing MCS10106 % Directories Containing MCS23 Method Clone Structures (MCS) Across Files 37

38. File Clone Classes (FCC) and File Clone Structures (FCS) 38

39. AVERAGETOTAL FCC10112 FCC Instances29314 Average Instances In FCC3 SCC Covered By FCC69755 % SCC Covered By FCC5 Files Containing FCC28307 % Files Containing FCC4 Directories Containing FCC10110 % Directories Containing FCC23 File Clone Classes (FCC) 39

40. AVERAGETOTAL FCS440 FCS Instances1098 Average Instances In FCS2 FCC Covered By FCS438 % FCC Covered By FCS40 Files Containing FCS12116 % Files Containing FCS2 Directories Containing FCS764 % Directories Containing FCS16 File Clone Structures (FCS) Across Directories 40

41. AVERAGETOTAL FCS886 FCS Instances25267 Average Instances In FCS2 FCC Covered By FCS883 % FCC Covered By FCS66 Files Covered By FCS20218 % Files Covered By FCS3 Directories Containing FCS664 % Directories Containing FCS11 File Clone Structures (FCS) Within Directories 41

42. Are structural clones more meaningful for program understanding and design recovery than simple clones? 42 2

43. Improved Program Understanding and Design Recovery Analysis is more qualitative than quantitative – anecdotal evidences of interesting examples of various types of structural clones Larger program parts recovered as clones from a system are expected to be more meaningful than smaller ones High level structural clones like FCC and FCS appear to be a very useful tool for design recovery because of their size, highlighting the design level similarities between various parts of the system 43

44. FCC Examples from Apache-Ant TCPC FTP.java458652% FTPTaskMirrorImpl.java457663% TarFileSetTest.java45190% ZipFileSetTest.java45190% 44

45. TCPC ColonialAIPlayer.java513076% StandardAIPlayer.java508452% ReportCargoPanel.java68274% ReportNavalPanel.java68669% FCC examples from FreeCol 45

46. FCC examples from DrJava 46 File NamesTCPC BackSlashTest.java205084% SingleQuoteTest.java205088%

47. FCC examples from JFreeChart 47 File NamesTCPC CombinedDomainCategoryPlot.java100850% CombinedDomainXYPlot.java100552% CombinedDomainXYPlot.java118261% CombinedRangeCategoryPlot.java118571% CombinedRangeXYPlot.java118263%

48. FCC Example from Javax-Swing 48 File NamesTCPC MultiButtonUI.java64489 % MultiColorChooserUI.java64489 % MultiComboBoxUI.java79587 % MultiDesktopIconUI.java64489 % MultiDesktopPaneUI.java64489 % MultiFileChooserUI.java83375 % MultiInternalFrameUI.java64489 % MultiLabelUI.java64489 % MultiListUI.java70873 % MultiMenuBarUI.java64489 % MultiMenuItemUI.java64488 % MultiOptionPaneUI.java73787 % MultiPanelUI.java64489 % MultiPopupMenuUI.java70279 % MultiProgressBarUI.java64489 %

49. FCC Example from Javax-Swing 49 File NamesTCPC MultiScrollBarUI.java64489 % MultiScrollPaneUI.java64489 % MultiSeparatorUI.java64489 % MultiSliderUI.java64489 % MultiSpinnerUI.java64489 % MultiSplitPaneUI.java86280 % MultiTabbedPaneUI.java70874 % MultiTableHeaderUI.java64489 % MultiTableUI.java64489 % MultiTextUI.java79853 % MultiToolBarUI.java64489 % MultiToolTipUI.java64489 % MultiTreeUI.java85961 % MultiViewportUI.java64489 % MultiRootPaneUI.java64489 %

50. DirectoryTC PCFile Name renderer/category/78651 %GradientBarPainter.java renderer/xy/78651 %GradientXYBarPainter.java renderer/category/5377 %BarPainter.java renderer/xy/5376 %XYBarPainter.java FCS Example from JFreeChart 50

51. 51 Clones 77%, 53 tokens Clones 51%, 786 tokens Clones 77%, 53 tokens Clones 51%, 786 tokens GradientBarPainter.java BarPainter.java GradientXYBarPainter.java XYBarPainter.java Implements FCS Example from JFreeChart

52. FCS Within Directory in Columba 52 File NamesTCPC DownAction.java24765 % UpAction.java24763 % NextMessageAction.java19962 % PreviousMessageAction.java19962 % NextUnreadMessageAction.java22971% PreviousUnreadMessageAction.java22971 %

53. What is the value added by structural clone detection in identifying refactoring opportunities? 53 3

54. Better Refactoring Help Analysis of structural clones is helpful in locating places where high-level duplication is present that can be restructured or refactored. We can use the Form Template Method refactoring to unify similar methods that follow the same high-level algorithm but have implementation variations, with the Template Method design pattern Simple clones appear as candidates for the Extract Method refactoring, however, MCS Across Files could also indicate possible applications of Extract Super Class refactoring 54

55. Template method design pattern 55

56. traverse(){ backtrack=false; cur.i++; cur.next=n; checkSent(true); } init(); cur = findNext(); jumptToNext(); serialize(cur) send(cur); updatePos(-1); processGraph(){ setEdge(cur); cur.next=cur.i; checkSent(true); } init(); cur = findNext(); jumptToNext(); serialize(cur) send(cur); updatePos(1); A structural clone suitable for Form Template Method Refactoring (a toy example) 56

57. Target for Form Template Method refactoring from Javax-Swing protected void layoutVScrollbar(JScrollBar sb) { Dimension sbSize = sb.getSize(); Insets sbInsets = sb.getInsets(); int itemW = sbSize.width – (sbInsets.left+bInsets.right); int itemX = sbInsets.left; boolean squareButtons = DefaultLookup.getBoolean( scrollbar, this, "ScrollBar.squareButtons", false); int decrButtonH = squareButtons ? itemW : decrButton.getPreferredSize().height; int decrButtonY = sbInsets.top; int incrButtonH = squareButtons ? itemW : incrButton.getPreferredSize().height; int incrButtonY = sbSize.height - (sbInsets.bottom + incrButtonH); int sbInsetsH = sbInsets.top + sbInsets.bottom; int sbButtonsH = decrButtonH + incrButtonH; float trackH = sbSize.height – (sbInsetsH + sbButtonsH); float min = sb.getMinimum(); float extent = sb.getVisibleAmount(); float range = sb.getMaximum() - min; float value = getValue(sb); int thumbH = (range <= 0)? getMaximumThumbSize().height : (int)(trackH * (extent / range)); thumbH = Math.max(thumbH, getMinimumThumbSize().height); thumbH = Math.min(thumbH, getMaximumThumbSize().height); int thumbY = incrButtonY - thumbH; if (value < (sb.getMaximum() – sb.getVisibleAmount())) { float thumbRange = trackH - thumbH; thumbY = (int)(0.5f + (thumbRange * ((value – min) / (range - extent)))); thumbY += decrButtonY + decrButtonH; } int sbAvailButtonH = (sbSize.height - sbInsetsH); if (sbAvailButtonH < sbButtonsH) { incrButtonH = decrButtonH = sbAvailButtonH / 2; incrButtonY = sbSize.height - (sbInsets.bottom + incrButtonH); } decrButton.setBounds(itemX, decrButtonY, itemW, decrButtonH); incrButton.setBounds(itemX, incrButtonY, itemW, incrButtonH); int itrackY = decrButtonY + decrButtonH; int itrackH = incrButtonY - itrackY; trackRect.setBounds(itemX, itrackY, itemW, itrackH); if(thumbH >= (int)trackH){ setThumbBounds(0, 0, 0, 0); } else { if ((thumbY + thumbH) > incrButtonY) { thumbY = incrButtonY - thumbH; } if (thumbY < (decrButtonY + decrButtonH)) { thumbY = decrButtonY + decrButtonH + 1; } setThumbBounds(itemX, thumbY, itemW, thumbH); } protected void layoutHScrollbar(JScrollBar sb) { Dimension sbSize = sb.getSize(); Insets sbInsets = sb.getInsets(); int itemH = sbSize.height – (sbInsets.top + sbInsets.bottom); int itemY = sbInsets.top; boolean ltr = sb.getComponentOrientation().isLeftToRight(); boolean squareButtons = DefaultLookup.getBoolean( scrollbar, this, "ScrollBar.squareButtons", false); int leftButtonW = squareButtons ? itemH : decrButton.getPreferredSize().width; int rightButtonW = squareButtons ? itemH : incrButton.getPreferredSize().width; if (!ltr) { int temp = leftButtonW; leftButtonW = rightButtonW; rightButtonW = temp; } int leftButtonX = sbInsets.left; int rightButtonX = sbSize.width - (sbInsets.right + rightButtonW); int sbInsetsW = sbInsets.left + sbInsets.right; int sbButtonsW = leftButtonW + rightButtonW; float trackW = sbSize.width – (sbInsetsW + sbButtonsW); float min = sb.getMinimum(); float max = sb.getMaximum(); float extent = sb.getVisibleAmount(); float range = max - min; float value = getValue(sb); int thumbW = (range <= 0)? getMaximumThumbSize().width : (int)(trackW * (extent / range)); thumbW = Math.max(thumbW, getMinimumThumbSize().width); thumbW = Math.min(thumbW, getMaximumThumbSize().width); int thumbX = ltr ? rightButtonX - thumbW : leftButtonX + leftButtonW; if (value < (max - sb.getVisibleAmount())) { float thumbRange = trackW - thumbW; if( ltr ) { thumbX = (int)(0.5f + (thumbRange * ((value - min) / (range - extent)))); } else { thumbX = (int)(0.5f + (thumbRange * ( (max - extent - value) / (range - extent)))); } thumbX += leftButtonX + leftButtonW; } int sbAvailButtonW = (sbSize.width - sbInsetsW); if (sbAvailButtonW < sbButtonsW) { rightButtonW = leftButtonW = sbAvailButtonW / 2; rightButtonX = sbSize.width - (sbInsets.right + rightButtonW); } (ltr ? decrButton : incrButton).setBounds (leftButtonX, itemY, leftButtonW, itemH); (ltr ? incrButton : decrButton).setBounds(rightButtonX, itemY, rightButtonW, itemH); int itrackX = leftButtonX + leftButtonW; int itrackW = rightButtonX - itrackX; trackRect.setBounds(itrackX, itemY, itrackW, itemH); if (thumbW >= (int)trackW) { setThumbBounds(0, 0, 0, 0); } else { if (thumbX + thumbW > rightButtonX) { thumbX = rightButtonX - thumbW; } if (thumbX < leftButtonX + leftButtonW) { thumbX = leftButtonX + leftButtonW + 1; } setThumbBounds(thumbX, itemY, thumbW, itemH); } 57

58. Candidates for Form Template Method from JFreeChart ClassesMethods GroupedStackedBarRendererStackedBarRendererdrawItem() XYAreaRendererXYAreaRenderer2drawItem() LineAndShapeRendererScatterRenderergetLeagendItem() DataAxisPeriodAxisvalueToJava2D() ComparableObjectSeriesXYSeriesadd() CategoryPlotXYPlotreadObject() BarRendererLevelRendererdrawItem() DataAxisNumberAxisrefreshTicksHorizontal() XYBubbleRendererXYLineAndShapeRenderergetLeagendItem() MiddlePinNeedlePinNeedledrawNeedle() ComparableObjectSeriesXYSerieshashCode() AreaRendererBoxAndWhiskerRenderergetLegendItem() XYAreaRendererXYStepAreaRendererdrawItem() CompassPlotPiePlotsetSeriesNeedle() ColumnArrangementFlowArrangementarrangeNF() LogAxisNumberAxisselectVerticalAutoTickUnit() PaintMapStrokeMapequals() XYLineAndShapeRendererXYShapeRendererdrawSecondaryPass() MinuteSecondparseMinute() 58

59. CombinedDomainCategoryPlot.javaCombinedDomainXYPlot.java CombinedDomainCategoryPlot getGap setGap add remove getSubplots findSubplot zoomRangeAxes calculateAxisSpace draw setFixedRangeAxisSpaceForSubplots setOrientation getDataRange getLegendItems getCategories getCategoriesForAxis handleClick plotChanged equals clone CombinedDomainXYPlot getPlotType setOrientation getDataRange getGap setGap add remove getSubplots calculateAxisSpace draw getLegendItems zoomRangeAxes findSubplot setRenderer setFixedRangeAxisSpace setFixedRangeAxisSpaceForSubplots handleClick plotChanged equals clone Extract Superclass candidate from JFreeChart 59

60. Extract Super Class and Structural Clones 60 The two files contain a number of similarly named methods, but only through structural clone analysis we could find those methods that are also significantly similar in contents

61. Another Candidate for Extract Superclass from JFreeChart 61 FlowArrangementColumnArrangement FlowArrangement add arrange arrangeFN arrangeFR arrangeFF arrangeRR arrangeRF arrangeRN arrangeNN arrangeNF clear equals ColumnArrangement add arrange arrangeFF arrangeRR arrangeRF arrangeNN arrangeNF clear equals

62. More Candidates for Extract Superclass from JFreeChart 62 BarRendererLevelRenderer LogAxisNumberAxis AbstractCategoryItemRendererAbstractXYItemRenderer MilliSecondSecond PaintMapStrokeMap CombinedDomainXYPlotCombinedRangeXYPlot MinuteSecond CombinedRangeCategoryPlotCombinedDomainCategoryPlot CombinedRangeCategoryPlotCombinedRangeXYPlotCombinedDomainCategoryPlot DefaultIntervalXYDatasetDefaultXYDatasetDefaultXYZDataset

63. Conclusions Structural clones often represent important design concepts Structural clone detection becomes an aid in design recovery Structural clones show a bigger picture of code duplication and guides to the correct refactoring technique applicable in the given situation 63

64. Future Work Classification of structural clones Detection of other types of structural clones Better integration of structural clones with architecture recovery and re-modularization techniques Better visualization of structural clones Management of structural clones with meta- programming Re-engineering for reuse 64

65. 65 Thank you