1. Huiqing Li Claus Reinke Simon Thompson Computing Lab, University of Kent www.cs.kent.ac.uk/projects/refactor-fp/ Refactoring Functional Programs

2. Durham Feb 042 Writing a program -- format a list of Strings, one per line table :: [String] -> String table = concat. format format :: [String] -> String format [] = [] format [x] = [x] format (x:xs) = (x ++ "\t") : fomrat xs

3. Durham Feb 043 Writing a program -- format a list of Strings, one per line table :: [String] -> String table = concat. format format :: [String] -> String format [] = [] format [x] = [x] format (x:xs) = (x ++ "\t") : fomrat xs

4. Durham Feb 044 Writing a program -- format a list of Strings, one per line table :: [String] -> String table = concat. format format :: [String] -> String format [] = [] format [x] = [x] format (x:xs) = (x ++ "\t") : format xs

5. Durham Feb 045 Writing a program -- format a list of Strings, one per line table :: [String] -> String table = concat. format format :: [String] -> String format [] = [] format [x] = [x] format (x:xs) = (x ++ "\t") : format xs

6. Durham Feb 046 Writing a program -- format a list of Strings, one per line table :: [String] -> String table = concat. format format :: [String] -> [String] format [] = [] format [x] = [x] format (x:xs) = (x ++ "\t") : format xs

7. Durham Feb 047 Writing a program -- format a list of Strings, one per line table :: [String] -> String table = concat. format format :: [String] -> [String] format [] = [] format [x] = [x] format (x:xs) = (x ++ “\t") : format xs

8. Durham Feb 048 Writing a program -- format a list of Strings, one per line table :: [String] -> String table = concat. format format :: [String] -> [String] format [] = [] format [x] = [x] format (x:xs) = (x ++ "\n") : format xs

9. Durham Feb 049 Writing a program -- format a list of Strings, one per line table :: [String] -> String table = concat. format appNL :: [String] -> [String] appNL [] = [] appNL [x] = [x] appNL (x:xs) = (x ++ "\n") : appNL xs

10. Durham Feb 0410 Writing a program -- format a list of Strings, one per line table :: [String] -> String table = concat. format appNL :: [String] -> [String] appNL [] = [] appNL [x] = [x] appNL (x:xs) = (x ++ "\n") : appNL xs

11. Durham Feb 0411 Writing a program -- appNL a list of Strings, one per line table :: [String] -> String table = concat. appNL appNL :: [String] -> [String] appNL [] = [] appNL [x] = [x] appNL (x:xs) = (x ++ "\n") : appNL xs

12. Durham Feb 0412 Writing a program -- format a list of Strings, one per line table :: [String] -> String table = concat. appNL appNL :: [String] -> [String] appNL [] = [] appNL [x] = [x] appNL (x:xs) = (x ++ "\n") : appNL xs

13. Durham Feb 0413 Writing a program -- format a list of Strings, one per line table :: [String] -> String table = concat. appNL where appNL :: [String] -> [String] appNL [] = [] appNL [x] = [x] appNL (x:xs) = (x ++ "\n") : appNL xs

14. Durham Feb 0414 Refactoring Refactoring means changing the design of program … … without changing its behaviour. Refactoring comes in many forms micro refactoring as a part of program development, major refactoring as a preliminary to revision, as a part of debugging, … As programmers, we do it all the time.

15. Durham Feb 0415 Not just programming Paper or presentation moving sections about; amalgamate sections; move inline code to a figure; animation; … Proof introduce lemma; remove, amalgamate hypotheses, … Program the topic of the lecture

16. Durham Feb 0416 Overview of the talk Example refactorings … what do we learn? Refactoring functional programs Generalities Tooling: demo, rationale, design. What comes next? Conclusions

17. Durham Feb 0417 Refactoring Functional Programs 3-year EPSRC-funded project  Explore the prospects of refactoring functional programs  Catalogue useful refactorings  Look into the difference between OO and FP refactoring  A real life refactoring tool for Haskell programming  A formal way to specify refactorings … and a set of proofs that the implemented refactorings are correct. Currently mid-project: the second HaRe release is module-aware.

18. Durham Feb 0418 Refactoring functional programs Semantics: can articulate preconditions and … … verify transformations. Absence of side effects makes big changes predictable and verifiable … … unlike OO. Language support: expressive type system, abstraction mechanisms, HOFs, … Opens up other possibilities … proof …

19. Durham Feb 0419 Rename f x y = …  Name may be too specific, if the function is a candidate for reuse. findMaxVolume x y = …  Make the specific purpose of the function clearer. Needs scope information: just change this f and not all f s (e.g. local definitions or variables). Needs module information: change f wherever it is imported.

20. Durham Feb 0420 Lift / demote f x y = … h … where h = …  Hide a function which is clearly subsidiary to f ; clear up the namespace. f x y = … (h y) … h y = …  Makes h accessible to the other functions in the module and beyond. Needs free variable information: which of the parameters of f is used in the definition of h ? Need h not to be defined at the top level, …, DMR.

21. Durham Feb 0421 Lessons from the first examples Changes are not limited to a single point or even a single module: diffuse and bureaucratic … … unlike traditional program transformation. Many refactorings bidirectional … … as there is never a unique correct design.

22. Durham Feb 0422 How to apply refactoring? By hand, in a text editor Tedious Error-prone Depends on extensive testing With machine support Reliable Low cost: easy to make and un-make large changes. Exploratory … a full part of the programmer’s toolkit.

23. Durham Feb 0423 Machine support invaluable Current practice: editor + type checker (+ tests). Our project: automated support for a repertoire of refactorings … … integrated into the existing development process: Haskell IDEs such as vim and emacs.

24. Durham Feb 0424 Demonstration of HaRe, hosted in vim.

25. Durham Feb 0425 Proof of concept … To show proof of concept it is enough to: build a stand-alone tool, work with a subset of the language, ‘pretty print’ the refactored source code in a standard format.

26. Durham Feb 0426 … or a useful tool? To make a tool that will be used we must: integrate with existing program development tools: the program editors emacs and vim: only add to their capabilities; work with the complete Haskell 98 language; preserve the formatting and comments in the refactored source code; allow users to extend and script the system.

27. Durham Feb 0427 Refactorings implemented in HaRe Rename Delete Lift (top level / one level) Demote Introduce definition Remove definition Unfold Generalise Add and remove parameters All these refactorings are module aware.

28. Durham Feb 0428 Implementing HaRe: an example -- This is an example module Main where sumSquares x y = sq x + sq y where sq :: Int->Int sq x = x ^ pow pow = 2 :: Int main = sumSquares 10 20 Promote the definition of sq to top level

29. Durham Feb 0429 Implementing HaRe: an example -- This is an example module Main where sumSquares x y = sq x + sq y where sq :: Int->Int sq x = x ^ pow pow = 2 :: Int main = sumSquares 10 20 Identify the definition of sq to be promoted

30. Durham Feb 0430 Implementing HaRe: an example -- This is an example module Main where sumSquares x y = sq x + sq y where sq :: Int->Int sq x = x ^ pow pow = 2 :: Int main = sumSquares 10 20 Is sq defined at top level, here or in importing modules; is sq imported from elsewhere?

31. Durham Feb 0431 Implementing HaRe: an example -- This is an example module Main where sumSquares x y = sq x + sq y where sq :: Int->Int sq x = x ^ pow pow = 2 :: Int main = sumSquares 10 20 Does sq use anything defined locally to sumSquares ?

32. Durham Feb 0432 Implementing HaRe: an example -- This is an example module Main where sumSquares x y = sq pow x + sq pow y where sq :: Int->Int->Int sq pow x = x ^ pow pow = 2 :: Int main = sumSquares 10 20 If so, generalise to add these as parameters, and change type signature.

33. Durham Feb 0433 Implementing HaRe: an example -- This is an example module Main where sumSquares x y = sq pow x + sq pow y where pow = 2 :: Int sq :: Int->Int->Int sq pow x = x ^ pow main = sumSquares 10 20 Finally, move the definition to top level.

34. Durham Feb 0434 The Implementation of Hare Information gathering Pre-condition checking Program transformation Program rendering

35. Durham Feb 0435 Information needed Syntax: replace the function called sq, not the variable sq …… parse tree. Static semantics: replace this function sq, not all the sq functions …… scope information. Module information: what is the traffic between this module and its clients …… call graph. Type information: replace this identifier when it is used at this type …… type annotations.

36. Durham Feb 0436 Infrastructure To achieve this we chose to: build a tool that can interoperate with emacs, vim, … yet act separately. leverage existing libraries for processing Haskell 98, for tree transformation, yet … … modify them as little as possible. be as portable as possible, in the Haskell space.

37. Durham Feb 0437 The Haskell background Libraries parser:many type checker:few tree transformations:few Difficulties Haskell98 vs. Haskell extensions. Libraries: proof of concept vs. distributable. Source code regeneration. Real project

38. Durham Feb 0438 Programatica Project at OGI to build a Haskell system … … with integral support for verification at various levels: assertion, testing, proof etc. The Programatica project has built a Haskell front end in Haskell, supporting syntax, static, type and module analysis … … freely available under BSD licence.

39. Durham Feb 0439 The Implementation of Hare Information gathering Pre-condition checking Program transformation Program rendering

40. Durham Feb 0440 First steps … lifting and friends Use the Haddock parser … full Haskell given in 500 lines of data type definitions. Work by hand over the Haskell syntax: 27 cases for expressions … Code for finding free variables, for instance …

41. Durham Feb 0441 Finding free variables … 100 lines instance FreeVbls HsExp where freeVbls (HsVar v) = [v] freeVbls (HsApp f e) = freeVbls f ++ freeVbls e freeVbls (HsLambda ps e) = freeVbls e \\ concatMap paramNames ps freeVbls (HsCase exp cases) = freeVbls exp ++ concatMap freeVbls cases freeVbls (HsTuple _ es) = concatMap freeVbls es … etc.

42. Durham Feb 0442 This approach Boiler plate code … … 1000 lines for 100 lines of significant code. Error prone: significant code lost in the noise. Want to generate the boiler plate and the tree traversals … … DriFT: Winstanley, Wallace … Strafunski: Lämmel and Visser

43. Durham Feb 0443 Strafunski Strafunski allows a user to write general (read generic), type safe, tree traversing programs … … with ad hoc behaviour at particular points. Traverse through the tree accumulating free variables from component parts, except in the case of lambda abstraction, local scopes, … Strafunski allows us to work within Haskell … other options are under development.

44. Durham Feb 0444 Rename an identifier rename:: (Term t)=>PName->HsName->t->Maybe t rename oldName newName = applyTP worker where worker = full_tdTP (idTP ‘adhocTP‘ idSite) idSite :: PName -> Maybe PName idSite v@(PN name orig) | v == oldName = return (PN newName orig) idSite pn = return pn

45. Durham Feb 0445 The coding effort Transformations with Strafunski are straightforward … … the chore is implementing conditions that guarantee that the transformation is meaning- preserving. This is where the bulk of our code lies.

46. Durham Feb 0446 The Implementation of Hare Information gathering Pre-condition checking Program transformation Program rendering

47. Durham Feb 0447 Program rendering example -- This is an example module Main where sumSquares x y = sq x + sq y where sq :: Int->Int sq x = x ^ pow pow = 2 :: Int main = sumSquares 10 20 Promote the definition of sq to top level

48. Durham Feb 0448 Program rendering example module Main where sumSquares x y = sq pow x + sq pow y where pow = 2 :: Int sq :: Int->Int->Int sq pow x = x ^ pow main = sumSquares 10 20 Using a pretty printer: comments lost and layout quite different.

49. Durham Feb 0449 Program rendering example -- This is an example module Main where sumSquares x y = sq x + sq y where sq :: Int->Int sq x = x ^ pow pow = 2 :: Int main = sumSquares 10 20 Promote the definition of sq to top level

50. Durham Feb 0450 Program rendering example -- This is an example module Main where sumSquares x y = sq pow x + sq pow y where pow = 2 :: Int sq :: Int->Int->Int sq pow x = x ^ pow main = sumSquares 10 20 Layout and comments preserved.

51. Durham Feb 0451 Rendering: our approach White space and comments in the token stream. 2 views of the program: token stream and AST. Modification of the AST guides the modification of the token stream. After a refactoring, the program source is extracted from the token stream not the AST. Use heuristics to associate comments with semantic entities.

52. Durham Feb 0452 Production tool (version 0) Programatica parser and type checker Refactor using a Strafunski engine Render code from the token stream and syntax tree.

53. Durham Feb 0453 Production tool (version 1) Programatica parser and type checker Refactor using a Strafunski engine Render code from the token stream and syntax tree. Pass lexical information to update the syntax tree and so avoid reparsing

54. Durham Feb 0454 Module awareness: example Move a top-level definition f from module A to B. -- Is f defined at the top-level of B? -- Are the free variables in f accessible within module B? -- Will the move require recursive modules? -- Remove the definition of f from module A. -- Add the definition to module B. -- Modify the import/export in module A, B and the client modules of A and B if necessary. -- Change uses of A.f to B.f or f in all affected modules. -- Resolve ambiguity.

55. Durham Feb 0455 What have we learned? Emerging Haskell libraries make it a practical platform. Efficiency issues … type checking large systems. Limitations of IDE interactions in vim and emacs. Reflections on Haskell itself.

56. Durham Feb 0456 Reflecting on Haskell Cannot hide items in an export list (though you can on import). The formal semantics of pattern matching is problematic. ‘Ambiguity’ vs. name clash. ‘Tab’ is a nightmare! Correspondence principle fails …

57. Durham Feb 0457 Correspondence Operations on definitions and operations on expressions can be placed in correspondence (R.D.Tennent, 1980)

58. Durham Feb 0458 Correspondence Definitions where f x y = e f x | g1 = e1 | g2 = e2 Expressions let \x y -> e f x = if g1 then e1 else if g2 …

59. Durham Feb 0459 Where do we go next? Larger-scale examples: ADTs, monads, … An API for do-it-yourself refactorings, or … … a language for composing refactorings Detecting ‘bad smells’ Evolving the evidence: GC6.

60. Durham Feb 0460 What do users want? Find and remove duplicate code. Argument permutations. Data refactorings. More traditional program transformations. Monadification.

61. Durham Feb 0461 Monadification (cf Erwig) r = f e1 e2 dov1 <- e1 v2 <- e2 r <- f v1 v2 return r

62. Durham Feb 0462 Larger-scale examples More complex examples in the functional domain; often link with data types. Dawning realisation that can some refactorings are pretty powerful. Bidirectional … no right answer.

63. Durham Feb 0463 Algebraic or abstract type? data Tr a = Leaf a | Node a (Tr a) (Tr a) Tr Leaf Node flatten :: Tr a -> [a] flatten (Leaf x) = [x] flatten (Node s t) = flatten s ++ flatten t

64. Durham Feb 0464 Algebraic or abstract type? data Tr a = Leaf a | Node a (Tr a) (Tr a) isLeaf = … isNode = … … Tr isLeaf isNode leaf left right mkLeaf mkNode flatten :: Tr a -> [a] flatten t | isleaf t = [leaf t] | isNode t = flatten (left t) ++ flatten (right t)

65. Durham Feb 0465 Algebraic or abstract type?  Pattern matching syntax is more direct … … but can achieve a considerable amount with field names. Other reasons? Simplicity (due to other refactoring steps?).  Allows changes in the implementation type without affecting the client: e.g. might memoise Problematic with a primitive type as carrier. Allows an invariant to be preserved.

66. Durham Feb 0466 Outside or inside? data Tr a = Leaf a | Node a (Tr a) (Tr a) isLeaf = … … Tr isLeaf isNode leaf left right mkLeaf mkNode flatten :: Tr a -> [a] flatten t | isleaf t = [leaf t] | isNode t = flatten (left t) ++ flatten (right t)

67. Durham Feb 0467 Outside or inside? data Tr a = Leaf a | Node a (Tr a) (Tr a) isLeaf = … … flatten = … Tr isLeaf isNode leaf left right mkLeaf mkNode flatten

68. Durham Feb 0468 Outside or inside?  If inside and the type is reimplemented, need to reimplement everything in the signature, including flatten. The more outside the better, therefore.  If inside can modify the implementation to memoise values of flatten, or to give a better implementation using the concrete type. Layered types possible: put the utilities in a privileged zone.

69. Durham Feb 0469 API Refactorings Refactoring utilities Strafunski Haskell

70. Durham Feb 0470 DSL Refactorings Refactoring utilities Strafunski Haskell Combining forms

71. Durham Feb 0471 Detecting ‘bad smells’ Work by Chris Ryder

72. Durham Feb 0472 Evolving the evidence Dependable System Evolution is the software engineering grand challenge. Build systems with evidence of their dependability … … but this begs the question of how to evolve the evidence in line with the system. Refactoring proofs, test coverage data etc.

73. Durham Feb 0473 Teaching and learning design Exciting prospect of using a refactoring tool as an integral part of an elementary programming course. Learning a language: learn how you could modify the programs that you have written … … appreciate the design space, and … the features of the language.

74. Durham Feb 0474 Conclusions Refactoring + functional programming: good fit. Practical tool … not ‘yet another type tweak’. Leverage from available libraries … with work. We have begun to use the tool in building itself! Much more to do than we have time for. Martin Fowler’s ‘Rubicon’: ‘extract definition’ … in HaRe version 1 … fp productivity.

75. www.cs.kent.ac.uk/projects/refactor-fp/