1. 1 Muhammed Al-MulhemVisual Languages Visual Programming Languages ICS 519 Part 1 ICS Department KFUPM Sept. 15, 2002

2. 2 Muhammed Al-MulhemVisual Languages Learning programming is a difficult task. Writing and testing a program is a difficult and time consuming task. The majority of computer users are non programmers. Programming languages are designed for professional programmers and computer scientists. 1 Introduction

3. 3 Muhammed Al-MulhemVisual Languages There are historical reasons for this lack of emphasis on human convenience in programming languages (Shu, 1988). The computer scientists, for the last fifty years, have concentrated on machine efficiency. Computing costs were high and human costs were relatively low. In the early years of computing, it was justifiable that efficiency took precedence over ease of programming.

4. 4 Muhammed Al-MulhemVisual Languages With the decline in computing costs it is realized that it is the human resource that must be optimized. The challenge is to bring computer capabilities to people without special computer training. To meet the challenge a number of facilities have been introduced to ease the pain of programming and remembering commands. These facilities include icons, pointing devices, and menus.

5. 5 Muhammed Al-MulhemVisual Languages Programming languages have evolved over the last fifty years from first generation programming languages to fourth generation programming languages. One characteristic of most of these languages is that they use one-dimensional and textual formats to represent programs. These formats are suitable for sequential machines but not for the multi-dimensional and visual human mind (Chang, 1986).

6. 6 Muhammed Al-MulhemVisual Languages Visual programming is an attempt to simplify programming and to make better use of the human mind's capabilities George Raeder states "... human mind is strongly visually oriented and that people acquire information at a significantly higher rate by discovering graphical relationships in complex pictures than by reading text.... When we scan paragraph, headlines, and material in bold type to speed up text search, we are really acting in the pictorial mode to overcome this limitation" (Raeder, 1985).

7. 7 Muhammed Al-MulhemVisual Languages Visual programming represents a conceptually revolutionary departure from the traditional one- dimensional and textual programming languages. It is stimulated by the following premises: 1) Pictures are more powerful than words as a means of communication. They can convey more meaning in a more concise unit of expression. 2) Pictures aid understanding and remembering. 3) Pictures may provide an incentives for learning program.

8. 8 Muhammed Al-MulhemVisual Languages 4) Pictures do not have language barriers. When properly designed, they are understood by people regardless of what language they speak.

9. 9 Muhammed Al-MulhemVisual Languages Visual programming has gained momentum in recent years. The falling costs of graphics-related hardware and software has made it feasible to use pictures as a means of communicating with the computers. Visual programming is a very young but growing field. There is no common agreement on the definition of visual programming.

10. 10 Muhammed Al-MulhemVisual Languages 2 Definitions Visual programming languages is a new paradigm. There is no single formal definition for VPL. For this reason we will present a number of definitions that appear in the literature.

11. 11 Muhammed Al-MulhemVisual Languages 1) Chang, 1987 Visual programming languages, usually deals with objects that do not have an inherent visual representation. This includes traditional data types such as arrays, stacks, and queues and application data types such as forms, documents, and databases. Presenting these objects visually is helpful to the user. For the same reason, the languages themselves should be represented visually. In other words, both programming constructs and the rules to combine these constructs should be presented visually.

12. 12 Muhammed Al-MulhemVisual Languages 2) Shu, 1988 A visual programming language can be informally defined as a language which uses some visual representations ( in addition to or in place of words and numbers) to accomplish what would otherwise have to be written in a traditional one- dimensional programming language. Note that this definition imposes no restrictions on the type of data or information. –It is immaterial whether the object being operated on or being displayed to a user by a visual language is textual, numeric, pictorial, or even audio. –What is important is that the language itself must employ some meaningful visual expressions as a means of programming.

13. 13 Muhammed Al-MulhemVisual Languages 3) Tortora, 1990 A visual language models an icon system. In the icon system, a program consists of a spatial arrangement of pictorial symbols ( elementary icons ). Note that, the spatial arrangement is the two- dimensional counterpart of the sequential statements in the traditional programming languages. –In these languages, a program consists of a string in which terminals are concatenated. –Concatenation is the only construction rule, therefore it doesn’t appear in the language grammar.

14. 14 Muhammed Al-MulhemVisual Languages In the case of visual languages, three construction rules are used to spatially arrange icons: –Horizontal concatenation. –Vertical concatenation. –Spatial overlay. For this reason, there is a need to include both spatial operators and elementary icons in the vocabulary of terminal symbols of the visual language grammar. Elementary icons are partitioned into two categories: –Object icons which identify objects. –Process icons which express computation.

15. 15 Muhammed Al-MulhemVisual Languages 4) Al-Mulhem, 1993 Visual programming languages is a language which uses a combination of text and meaningful graphical symbols to write programs. These graphical/textual symbols represent the following: a) Data types such as numbers, characters, pictures, and sound. b) Data structures such as arrays, records, and pointers. c) Programming constructs such as iteration, selection, and conditional looping. d) Rules to combine the programming constructs.

16. 16 Muhammed Al-MulhemVisual Languages 5) Burnett (1999) Visual programming is programming in which more than one dimension is used to convey semantics. Examples of such additional dimensions are the use of multi-dimensional objects, the use of spatial relationships, or the use of the time dimension to specify “before-after” semantic relationships. Each potentially-significant multi-dimensional object or relationship is a token (just as in traditional textual programming languages each word is a token) and the collection of one or more such tokens is a visual expression.

17. 17 Muhammed Al-MulhemVisual Languages Examples of visual expressions used in visual programming include diagrams, free-hand sketches, icons, or demonstrations of actions performed by graphical objects. When a programming language’s (semantically- significant) syntax includes visual expressions, the programming language is a visual programming language (VPL).

18. 18 Muhammed Al-MulhemVisual Languages VPL Research Directions of research in this area are: 1.Visual approaches to traditional programming languages (flowchart). 2.Visual approaches that deviated from traditional programming (programming by demonstration).

19. 19 Muhammed Al-MulhemVisual Languages How good are they? Good for toy programs. Bad for realistically-sized programs.

20. 20 Muhammed Al-MulhemVisual Languages More approaches Incorporating visual techniques into programming environments: To support textual specification of GUI layout To support electronic forms of software engineering diagrams. To create/visualize relationships among data structures. To visually combine textually-programmed units to build new programs.

21. 21 Muhammed Al-MulhemVisual Languages Examples Visual Basic (for BASIC) VisualWork (for SmallTalk) CASE tools that support visual specificatio of relationships among program modules, automatic code generation, …etc.

22. 22 Muhammed Al-MulhemVisual Languages

23. 23 Muhammed Al-MulhemVisual Languages More approaches Domain-specific visual programming systems Examples: 1.LabVIEW – programming laboratory data acquestion. 2.AVS – programming Scientific Visualization 3.PhonePro – programming telephone and voice- mail behavior. 4.Cocoa – programming graphical simulations and games.

24. 24 Muhammed Al-MulhemVisual Languages What is next? VPLs for general-purpose programming. This can be reached by: 1.Improving the ways VP can be used. 2.Improving domain-specific programming

25. 25 Muhammed Al-MulhemVisual Languages Strategies used in VPLs Concrete: specify some aspect of semantics on a specific object or value. Direct: manipulate a specific object or value directly to specify semantics. Explicit: describe dataflow relationships by drawing directed edges among related variables. Immediate visual feedback: automatic display of effects of program edits.

26. 26 Muhammed Al-MulhemVisual Languages Classification 1.Pure visual programming 2.Hybrid (textual / visual) programming 3.Programming-by-example 4.Constraint-based programming 5.Form-based programming

27. 27 Muhammed Al-MulhemVisual Languages Pure visual programming Programmer manipulate icons to create a program. A program is debugged and executed in the same visual environment. A program is compiled directly from its visual representation. A program is never translated into intermediate text-based language. Examples: Prograph, VIPR, and PICT.

28. 28 Muhammed Al-MulhemVisual Languages Hybrid (textual / visual) programming Two forms: 1. Systems in which programs are created visually and then translated into an underlying high-level textual languages. Example: Rehearsal World – the user trains the system to solve a particular problem by manipulating graphical actors (icons) and then the system generates a Smalltalk program to implement the solution.

29. 29 Muhammed Al-MulhemVisual Languages Hybrid (continue) 2. Systems which involve the use of graphical elements in a textual language. Example: Work of Erwig et. al. – developing extensions to languages like C and C++ which allow programmers to mix their text code with diagrams. For instance, one can define a linked list data structure textually and then perform an operation like deletion of a node by drawing the steps in the process.

30. 30 Muhammed Al-MulhemVisual Languages Programming by example Allowing the user to create and manipulate graphical objects with which to teach the system how to perform a particular task. Examples: Rehearsal World and Pygmalion

31. 31 Muhammed Al-MulhemVisual Languages Constraint-oriented programming A programmer models physical objects as objects in the visual environment which are subject to constraints designed to mimic the behavior of natural laws, like gravity. Popular for simulation design. Examples: ThingLab, ARK

32. 32 Muhammed Al-MulhemVisual Languages Form-based programming Borrowed their visualizations and programming metaphors from a spreadsheets. Represent programming as altering a group of interconnected cells over time and often allow the programmer to visualize the execution of a program as a sequence of different cell states which progress through time. Example: Forms/3

33. 33 Muhammed Al-MulhemVisual Languages VPL Orphans Algorithm animation systems Provides interactive graphical displays of executing programs Example: BALSA user-interface design systems Provided with many modern compilers. Example: Visual Basic, Visual C++

34. 34 Muhammed Al-MulhemVisual Languages Theory Based on work by Chang Definitions Icon (generalized icon): An object with the dual representation of: –logical part (the meaning) –Physical part (the image) iconic system: structured set of related icons iconic sentence (visual sentence): spatial arrangement of icons from iconic system

35. 35 Muhammed Al-MulhemVisual Languages Theory (con.) visual language: set of iconic sentences constructed with given syntax and semantics syntactic analysis (spatial parsing): analysis of an iconic sentence to determine the underlying structure semantic analysis (spatial interpretation): analysis of an iconic sentence to determine the underlying meaning

36. 36 Muhammed Al-MulhemVisual Languages Theory (con.) A visual Language Models an Icon System. In the Icon System, a program (Visual Sentence) consists of a spatial arrangement of pictorial symbols (elementary icons). The spatial arrangement is the two-dimensional counterpart of the standard sequentialization in the construction of programs in the case of traditional programming languages.

37. 37 Muhammed Al-MulhemVisual Languages In traditional languages, a program is expressed by a string in which terminals are concatenated. As this operation (concatenation) is the only construction rule, it does not appear explicitly in the vocabulary of the language grammar. In the case of visual languages, three construction rules are used to spatially arrange icons: –Horizontal concatenation & –Vertical concatenation ^ –Spatial overlay+

38. 38 Muhammed Al-MulhemVisual Languages For this reason, there is a need to include the spatial operators and elementary icons in the vocabulary of terminals of the visual language grammar. The elementary icons are partitioned into: –object icons –process icons. Elementary object icons identify objects while process icons express computations. The meaning depends on the specific visual sentence in which it appears.

39. 39 Muhammed Al-MulhemVisual Languages Example Consider the following primitive icons from the Heidelberg icon set. where the square denotes a character, and its interpretation is unique in the system. the arrow can be used to express insertion or moving operations in different visual sentences.

40. 40 Muhammed Al-MulhemVisual Languages A Visual Language (VL) is specified by the triple »(ID, G o, B) where IDis the icon dictionary G o is a context-free grammar Bis a domain-specific knowledge base.

41. 41 Muhammed Al-MulhemVisual Languages The icon dictionary ID It is a set of triples of the form »i = (i l, i p, type(i)) Each triple i describes a primitive icon where –i l is the icon name (logical part or meaning) –i p is the icon sketch (physical part) –type(i) is the icon type (process or object)

42. 42 Muhammed Al-MulhemVisual Languages The grammar G o It is a context-free grammar (N, T, S, P). Where Nis the set of nonterminals T = T l U T sp T l = { i l | i = ( i l, i p, type(i))  ID} T sp = { &, ^, + } Sis the start symbol of the grammar Pis the set of productions of G o

43. 43 Muhammed Al-MulhemVisual Languages The grammar Go specifies how more complex icons can be constructed by spatially arranging elementary icons. Usually, nonterminals in Go & the start symbol represent composite object icons. Composite object icons can be derived in one or more steps starting from a given nonterminals, N, which is different from S. In other words, composite object icons are obtained by spatial arrangement of elementary object icons only and are assumed to be of type object.

44. 44 Muhammed Al-MulhemVisual Languages Visual Sentences contain at least one process icon. Visual Sentences have no type. Here, object icons denote both elementary object icons & composite object icons. It is easy to prove that all the icons involved in the icon system (elementary object, composite object, and visual sentences) are objects with dual representation of a logical and a physical part as follows:

45. 45 Muhammed Al-MulhemVisual Languages Elementary object --- by definition in ID. Composite object and visual sentences --- –The physical part is their image –The logical part (the meaning) can be derived from the meaning of the elementary objects occurring in them. The meaning of a visual sentence is expressed in terms of a conceptual tree, which represents an intermediate code for later execution.

46. 46 Muhammed Al-MulhemVisual Languages Conceptual Trees The meaning of a visual sentence is expressed in terms of a conceptual tree (CT), which represents an intermediate code for later execution. In CT notations a concept is denoted by [ ], and a relation is denoted by ( ).

47. 47 Muhammed Al-MulhemVisual Languages The concepts are expressed in the form [ OBJECT : object-name] [ EVENT : event-name] where object-name represents the logical part (meaning) of an object icon (elementary icon or composite object icon) event-name are event names, i.e., procedure names, which accomplish the task expressed by the visual sentence

48. 48 Muhammed Al-MulhemVisual Languages The following CTs are basic substructures occuring in all the CTs which represent the logical part of a visual sentence. [EVENT: event-name](rel1) [OBJECT: object-name1] (rel2) [OBJECT: object-name2] (1) [EVENT: event-name](rel) [OBJECT: object-name](2) [OBJECT: object-name] (rel) [OBJECT: object-name](3) The meanings of these CTs are as follows:

49. 49 Muhammed Al-MulhemVisual Languages 1) A process icon has two object icons as arguments Example: consider the following icon from the Heidelberg icon set

50. 50 Muhammed Al-MulhemVisual Languages It is composed of the object icons Selected-string String and the process icon Up-arrow

51. 51 Muhammed Al-MulhemVisual Languages Its meaning is expressed by the CT [EVENT: replace](object) [OBJECT: String] (place) [OBJECT: Selected-string]

52. 52 Muhammed Al-MulhemVisual Languages 2) A process icon requires only one argument. Example:

53. 53 Muhammed Al-MulhemVisual Languages It is obtained by composing in spatial overlay: the object icon Selected-string and the process icon cross

54. 54 Muhammed Al-MulhemVisual Languages Its meaning is described by the CT [EVENT: delete](object) [OBJECT: Selected-string]

55. 55 Muhammed Al-MulhemVisual Languages 3) Here we have two combined object icons. This is the case in which one of the two objects involved in the visual sentence represents a qualitative or quantitative specification of the other object. Example: The object "name” identifies the object "file"  File name

56. 56 Muhammed Al-MulhemVisual Languages The meaning is [OBJECT: file] (identifier) [OBJECT: name]

57. 57 Muhammed Al-MulhemVisual Languages Complex structures can be obtained by composing the basic substructures as shown below: 1 2 3 This visual sentence could mean: append file1 to file2 and copy the result into file3 

58. 58 Muhammed Al-MulhemVisual Languages The sentence could be decomposed into two subsentences: 1 2 and 2 3 

59. 59 Muhammed Al-MulhemVisual Languages These subsentences are described, respectively, by the following CTs [EVENT: append](object) [OBJECT: file1] (place) [OBJECT: file2] [EVENT: copy](source) [OBJECT: file2] (destination) [OBJECT: file3]

60. 60 Muhammed Al-MulhemVisual Languages The whole visual sentence is described by [EVENT: copy](destination) [OBJECT: file3] (source) [EVENT: append] (object) [OBJECT: file1] (object) [OBJECT: file2]

61. 61 Muhammed Al-MulhemVisual Languages The Knowledge Base B The Knowledge Base B contains domain-specific information necessary for constructing the meaning of a given visual sentence. It contains information regarding »Event names »Conceptual relations »Names of the resulting objects »References to the resulting objects.

62. 62 Muhammed Al-MulhemVisual Languages It is structured in the following seven tables: 1) EVENT-NAME [proc-name, obj-name1, obj-name2, op1, op2] This table returns the name of the event (procedure) to be associated to the process icon proc-name when the objects obj-name1, obj-name2 are spatially arranged (via op1, op2) to proc-name. 2)LEFT-REL [proc-name, obj-name1, obj-name2, op1, op2] 3)RIGHT-REL [proc-name, obj-name1, obj-name2, op1, op2] These two tables hold the conceptual relations existing between EVENT-NAME [Proc-name, obj-name, obj-name2, op1, op2] and obj-name1 or obj-name2 respectively.

63. 63 Muhammed Al-MulhemVisual Languages 4)RESULT-NAME [Proc-name, obj-name1, obj-name2, op1, op2] This table returns the name of the object resulting from the execution of the event. EVENT-NAME [Proc-name, obj-name1, obj-name2, op1, op2]. With arguments obj-name1 and obj-name2. 5)RESULT-REF [Proc-name, obj-name1, obj-name2, op1, op2]. This table returns a reference to the object representing the result.

64. 64 Muhammed Al-MulhemVisual Languages The following two tables take into account cases in which object icons are composed (composite object icons). 6) RELATION [obj-name1, obj-name2, op]. This returns the conceptual relation existing between obj- name1 and obj-name2 when they are spatially combined by means of the operator op. 7)RESULT-OBJECT [obj-name1, obj-name2, op]. This returns the name of the object resulting by the spatial combination (via op) of obj-name1 and obj-name2. In the previous tables, obj-namei refer to the logical part of the icons.

65. 65 Muhammed Al-MulhemVisual Languages Besides domain-specific relations the tables can also contain three special relations: :This relation means that the combination obj-name1 op obj-name2 gives rise to a new object, different from obj-name1 and obj-name2. :This relation means that the combination obj-name1 op obj-name2 generates an object that has the same logical part of the obj-name1 or obj-name2. In other words, it points out that no additional information is derived from such a combination.

66. 66 Muhammed Al-MulhemVisual Languages, : These two relations mean that the combination obj-name1 op1 proc-name op2 obj-name2 generates an object that has the same reference of the icon whose name is obj-name1 or obj- name2 respectively.

67. 67 Muhammed Al-MulhemVisual Languages  file2 file3  Obj-name1 Op1=& Proc-name Op2=& Obj-name2

68. 68 Muhammed Al-MulhemVisual Languages Table 1: returns the name of the event “copy” Table 2, 3: LEFT-REL [EVENT: copy](source) [OBJECT: file2] (destination) [OBJECT: file3] RIGHT-REL

69. 69 Muhammed Al-MulhemVisual Languages Table 4:Return file 3 Table 5:Return a reference (pointer) file3 Table 6: [OBJECT: file] (identifier) [OBJECT: file3] returns this Table 7:Return the name of the resulting object in 6

70. 70 Muhammed Al-MulhemVisual Languages The Attribute Grammar An attribute grammar consists of a semantics rules for synthesizing the meaning of a visual sentence AG consists of an underlying context-free grammar where i) each nonterminal in the context-free grammar has associated two attribute sets, namely, synthesized and inherited attributes.

71. 71 Muhammed Al-MulhemVisual Languages ii) each production has associated a set of semantics rules, used to determine the values of the attributes of the nonterminals depending on the values of the attributes of the remaining nonterminals appearing in the same production. The initial nonterminal has only synthesized attributes, and one of them is designated to hold the meaning of the derivation tree.

72. 72 Muhammed Al-MulhemVisual Languages The production rules in Go are of the following form: a)X  tt  T b)XYwhere type (X) = type (Y) c)XM op L | L op M | M1 op M2 d)XM1 op1 L op2 M2 where type (L) = PROCESS type (M) = type (M1) = type (M2) = OBJECT. op, op1, op2 = nonterminals for spatial operators ( &, *, +)

73. 73 Muhammed Al-MulhemVisual Languages The type for a given nonterminal X is determined as follows: Let X t be a derivation in Go and t  T* Then type (X) = spatial operator if t  Tsp type (X) = type (t) otherwise. The set of inherited attributes is empty for each nonterminal in Go.

74. 74 Muhammed Al-MulhemVisual Languages The set of synthesized attributes associated to each nonterminal X is defined as follows. For a given non-terminal X 1.if type (X) = OBJECT : NAME (X) is the name of the object icon represented by X. CG(X) is the CT associated to the subtree rooted in X. REF(X) is the reference number. The reference number is an index associated with each elementary object icon occurring in a visual sentence. It allows one to distinguish between different occurrences of the same elementary object icon in a visual sentence.

75. 75 Muhammed Al-MulhemVisual Languages 2. if type (X) = PROCESS: NAME(X) is the name of the process icon represented by X. 3.if type (X) :"spatial operator" : OP(X) is the spatial operator derived from X.

76. 76 Muhammed Al-MulhemVisual Languages The semantic rules associated with the four production rules are: a) X t a1) If t = icon-id where ( icon-id, icon-sk, type( i ) )  ID and type ( i ) = OBJECT (i.e. t is a terminal for an elementary icon). Then NAME(X) icon-id REF (X) ref-set CG (X) [OBJECT : NAME(X), REF(X)]

77. 77 Muhammed Al-MulhemVisual Languages a2) If t = icon-id where ( icon-id, icon-sk, type( i ) )  ID and type ( i ) = PROCESS Then NAME(X) t a3) If t  {&, L, +} Then OP(X)t

78. 78 Muhammed Al-MulhemVisual Languages b) X Y b1)If type (Y) = OBJECT Then NAME(X)NAME (Y) REF (X)REF (Y) CG (X)CG (Y) b2)If type (Y) = PROCESS Then NAME(X)NAME (Y) b3)If Y op andop  {&, L, +} Then OP(X)OP(Y)

79. 79 Muhammed Al-MulhemVisual Languages c) X M op L c1) If type (M) = OBJECT and type (L) = PROCESS Then NAME(X)RESULT-NAME [NAME(L), NAME (M), ‘null-obj’, op(op), ‘null-op’] REF(X)SET-REF (L, M, ‘null-obj’, op, ‘null-op’) CG (X)SELECTIVE-MERGE (L, M, ‘null-obj’, op, ‘null-op’)

80. 80 Muhammed Al-MulhemVisual Languages c2) If type (M) = PROCESS and type (L) = OBJECT Then NAME (X)RESULT-NAME [NAME(M), ‘null-obj’, NAME(L), ‘null-op’, OP(op)] REF(X)SET-REF (M, ‘null-obj’, L, ‘null-op’, op) CG (X)SELECTIVE-MERGE (M, ‘null-obj’, L, ‘null-op’, op)

81. 81 Muhammed Al-MulhemVisual Languages c3) If type (M) = type (L) = OBJECT Then NAME (X) RESULT-OBJECT [NAME (M), NAME(L), OP(op)] REF(X)REF (M) U REF (L) CG (X)LINK (X, M, L, op)

82. 82 Muhammed Al-MulhemVisual Languages d) X M1 op1 L op2 M2 If type (L) = PROCESS and type (M1) = type (M2) = OBJECT Then NAME (X)RESULT-NAME [NAME (L), NAME(M1), NAME (M2), OP(op1), OP (op2)] REF(X)SET-REF (L, M1, M2, op1, op2) CG (X)SELECTIVE-MERGE (L, M1, M2, op1, op2)

83. 83 Muhammed Al-MulhemVisual Languages Semantic rules c and d make use of the following procedures: Procedure LINK (X, M1, M2, op) relRELATION [NAME(M1), NAME (M2), OP(op)] case rel of : if NAME (X)  IDthen - add the new composite object NAME(X) to ID - return [OBJECT: NAME (X), REF(X)] endif : return CG(M1) else : return APPEND (CG(M1), CG(M2), rel) end case end LINK

84. 84 Muhammed Al-MulhemVisual Languages Procedure SELECTIVE-MERGE (L, M1, M2, op1, op2) C[EVENT:EVENT-NAME [NAME(L), NAME(M1), NAME(M2), OP(op1), OP(op2)] if OP(op1) = ‘null-op’ then G1 C else rel1 LEFT-REL [NAME(L), NAME(M1), NAME (M2), OP(op1), OP(op2)] G1 APPEND (C, CG(M1), rel1) endif if OP(op2) = ‘null-op’ then G2 C else rel2RIGHT-REL [NAME(L), NAME(M1), NAME (M2), OP(op1), OP(op2)] G2 APPEND (C, CG(M2), rel2) endif return HEADER-MERGE (G1, G2) end SELECTIVE-MERGE

85. 85 Muhammed Al-MulhemVisual Languages Procedure SET-REF (L, M1, M2, op1, op2) ref RESULT-REF [NAME(L), NAME(M1), NAME (M2), OP(op1), OP(op2)] case ref of :return REF(M1) :return REF(M2) :return NEXT-REF( ) end case end SET-REF

86. 86 Muhammed Al-MulhemVisual Languages The Icon Interpreter: The system diagram of the icon interpreter is as follows: G0 B ID Pattern Analysis Syntax Analysis Construction of the Logical part Visual Sentencce S Pattern String of S Parse Tree of S Meaning of S CG

87. 87 Muhammed Al-MulhemVisual Languages The module ”Pattern Analysis" transforms the visual sentence into pattern string. Example The visual sentence is represented by the pattern string (char & char & char) + cross

88. 88 Muhammed Al-MulhemVisual Languages The icon interpreter is logically divided into two phases: 1.The syntax analysis is accomplished by a parsing algorithm for context-free grammars. 2.Constructing the meaning of the given visual sentence by evaluating the attributes of the nonterminal on the root of the parse tree using the ATTRIBUTE_EVALUATOR procedure The attribute evaluation is accomplished by means of a postorder visit of the parse tree t of the visual sentence S

89. 89 Muhammed Al-MulhemVisual Languages Procedure ATTRIBUTE-EVALUATOR (X). if X is a nonterminal in Go then /* let p be the production applied to the node X in t, and let 1,..., np be the indices of nonterminals in the right-hand side of p*/ for j = 1 to np do call ATTRIBUTE-EVALUATOR (jth nonterminal son of X in t) end for apply the semantic rules associated to p so as to evaluate the attributes of X endif end ATTRIBUTE-EVALUATOR

90. 90 Muhammed Al-MulhemVisual Languages Example This example shows an iconic command language for simple operating system. The set of elementary icons ID and the grammar G 0, are shown in the following slides.

91. 91 Muhammed Al-MulhemVisual Languages

92. 92 Muhammed Al-MulhemVisual Languages

93. 93 Muhammed Al-MulhemVisual Languages The knowledge base B is given next. Each entry in the tables, which constitute the knowledge base, contains five items, which correspond, from top to bottom, to the tables: –EVENT_NAME –LEFT_REL –RIGHT_REL –RESULT_NAME –RESULT_REF

94. 94 Muhammed Al-MulhemVisual Languages

95. 95 Muhammed Al-MulhemVisual Languages Consider the following visual sentence: Whose corresponding pattern string is [FILE,1] & PLUS & [FILE,2] & ARROW&[FILE,3] The parse tree for this sentence is shown next. 

96. 96 Muhammed Al-MulhemVisual Languages

97. 97 Muhammed Al-MulhemVisual Languages The construction of the conceptual tree is given in the next table. At level 1, the resulting conceptual tree is given by: [EVENT: copy](destination) [OBJECT: file3] (source) [EVENT: append] (object) [OBJECT: file1] (object) [OBJECT: file2]

98. 98 Muhammed Al-MulhemVisual Languages Examples of VPLs Prograph (1989) – Dataflow programming DataLab (1990) – Programming by example Form/3 (1994) – Form-based programming Clarity (2001) – Functional programming

99. 99 Muhammed Al-MulhemVisual Languages Prograph The Prograph language is an iconic language. Program is drawings. Prograph interpreter and compiler execute those drawings. Prograph is object-oriented Prograph supports dataflow specification of program execution

100. 100 Muhammed Al-MulhemVisual Languages Simple program A simple Prograph code which sorts, possibly in parallel, three database indices and updates the database.

101. 101 Muhammed Al-MulhemVisual Languages Paragraph’s icons Prograph uses a set of icons: classes are represented by hexagons, data elements by triangles, pieces of code by rectangles with a small picture of data flow code inside, and inheritance by lines or downward pointing arrows.

102. 102 Muhammed Al-MulhemVisual Languages

103. 103 Muhammed Al-MulhemVisual Languages More Icons The representations can then be combined and used in a variety of language elements. For example, initialization methods - which are associated with an individual class - are depicted as hexagonally shaped icons with a small picture of data flow code inside.

104. 104 Muhammed Al-MulhemVisual Languages Icon (C0nt.) Triangles represent instance variables in a class’ Data Window to show that they’re data. Hexagons drawn with edges and interiors similar to the instance variable triangles represent class variables. This associates them with the class as a whole while also associating them with data. Next Figure shows three more complex representations: initialization methods, instance variables, and class variables.

105. 105 Muhammed Al-MulhemVisual Languages Icon (Cont.)

106. 106 Muhammed Al-MulhemVisual Languages Object-Oriented The Prograph language is object-oriented. Class-based, Single inheritance (a subclass can only inherit from one parent), Dynamic typing With garbage collection mechanism based on reference counting.

107. 107 Muhammed Al-MulhemVisual Languages Cont. The programmer can design and implement new classes, He can visually specifying: –the new class’ inheritance, –additional instance or class variables, –modified default values for inherited instance or class variables, –and new or overridden methods.

108. 108 Muhammed Al-MulhemVisual Languages Cont. Polymorphism allows each object to respond to a method call with its own method appropriate to its class. Binding of a polymorphic operation to a method in a particular class happens at run-time, and depends upon the class of the object. The syntax for this ‘message send’ is one of the more unusual aspects of Prograph’s syntax. The concept of ‘message sending’ per se is not used in Prograph.

109. 109 Muhammed Al-MulhemVisual Languages Annotation Rather, the idea of an annotation on the name of an operation is used. There are four cases: 1.No annotation means that the operation is not polymorphic. 2.‘/’ denotes that the operation is polymorphic and is to be resolved by method lookup starting with the class of the object flowing into the operation on its first input

110. 110 Muhammed Al-MulhemVisual Languages Annotation (Cont.) 3.‘//’ denotes that the operation is polymorphic and is to be resolved by method lookup starting with the class in which this method is defined. 4.‘//’ plus super annotation means that the operation is polymorphic and resolves by method lookup starting with the superclass of the class in which this method is defined. In Prograph there is no SELF construct (or a ‘this’ construct, to use the C++ terminology).

111. 111 Muhammed Al-MulhemVisual Languages Prograph has an almost completely iconic syntax, and this iconic syntax is the only representation of Prograph code. Next Figure shows a small code fragment typical of the Prograph language. There is no textual equivalent of a piece of a Prograph program - the Prograph interpreter and compiler translate directly from this graphical syntax into code.

112. 112 Muhammed Al-MulhemVisual Languages Example

113. 113 Muhammed Al-MulhemVisual Languages Icons Next Figure shows most of the lexicon of the language (icons). The inputs and outputs to an operation are specified by small circles at the top and bottom, respectively, of the icons. The number of inputs and outputs - the operation’s arity - is enforced only at run-time. In addition, there are a variety of annotations that can be applied to the inputs and outputs, or to the operation as a whole to implement looping or control flow.

114. 114 Muhammed Al-MulhemVisual Languages

115. 115 Muhammed Al-MulhemVisual Languages Spaghetti Code Visual languages like Prograph sometimes suffer from the spaghetti code problem: there are so many lines running all over that the code for any meaningful piece of work actually ends up looking like spaghetti. Prograph deals with this issue by enabling the programmer to ‘iconify’ any portion of any piece of code at any time during the development process.

116. 116 Muhammed Al-MulhemVisual Languages Local This iconified code is called a ‘local’. Effectively, locals are nested pieces of code. There is no limit to the level of nesting possible with locals, and there is no execution penalty to their use. In addition, locals can be named and this naming, if done well, can provide a useful documentation for the code.

117. 117 Muhammed Al-MulhemVisual Languages Example Next Figure is a code to build a dialog box containing a scrolling list of the installed fonts without the use of Prograph locals to factor the code.

118. 118 Muhammed Al-MulhemVisual Languages

119. 119 Muhammed Al-MulhemVisual Languages Same Example Next Figure is the same code with the use of Prograph locals to factor the code.

120. 120 Muhammed Al-MulhemVisual Languages

121. 121 Muhammed Al-MulhemVisual Languages Local (Cont.) Note that long names, often containing punctuation or other special characters can be used for the names of locals. The proper use of locals can dramatically improve the comprehensibility of a piece of code. The one negative aspect of their use is the so- called “rat hole” phenomena - every piece of code in its own rat hole. This can sometimes make finding code difficult.

122. 122 Muhammed Al-MulhemVisual Languages Syntax Three of the most interesting aspects of the Prograph syntax are: –list annotation, –injects, and –control annotation.

123. 123 Muhammed Al-MulhemVisual Languages List Annotation Any elementary operation can be made to loop by list annotating any of its inputs. When this is done, the operation is invoked multiple times - one time for each element of the list that is passed on this input. Thus, list annotation on an input causes the compiler to construct a loop for the programmer. Since this annotation can be made on a local as well as a primitive operation, this looping construct is quite powerful.

124. 124 Muhammed Al-MulhemVisual Languages List Annotation (Cont.) In addition, list annotation can be done on a output. On an output terminal, list annotation causes the system to put together all the outputs at this terminal for every iteration of the operation and to pass as the ‘final’ output of the terminal this collection of results as an output list.

125. 125 Muhammed Al-MulhemVisual Languages List Annotation (Cont.) List annotation, then, enables the programmer to easily make an operation into a loop that either breaks down a list into its component elements and runs that operation on the elements, or to builds up a list from the multiple executions of an operation. An individual operation can have any number of its inputs or outputs with list annotations.

126. 126 Muhammed Al-MulhemVisual Languages List Annotation (Cont.) Next Figure shows several examples of list annotation, an efficient mechanism for iterating over a list. The operation will automatically be called repeatedly for each element of a list, or its outputs will be packaged together into a list.

127. 127 Muhammed Al-MulhemVisual Languages

128. 128 Muhammed Al-MulhemVisual Languages

129. 129 Muhammed Al-MulhemVisual Languages

130. 130 Muhammed Al-MulhemVisual Languages Inject Inject lets you pass a name at run-time for an input which expects a function. This is similar to Smalltalk’s perform, procedure variables in Pascal or function pointers in C. Suppose, for example, that you want to implement a function FooMe that takes two arguments: a list of objects, and a reference to a method to be applied to each of those objects.

131. 131 Muhammed Al-MulhemVisual Languages Inject (Cont.) In Object Pascal pseudo-code, this function might look something like this: Function FooMe(theList: TList; Procedure DoThis(object: TObject)); { Iterates through the list of objects applying the DoThis procedure } BEGIN theList.Each(DoThis); END;

132. 132 Muhammed Al-MulhemVisual Languages Next Figure shows a Prograph implementation of FooMe. Note that just the name - as a string - of the method to be applied to each object is passed to FooMe. This string is turned into a method invocation by the inject.

133. 133 Muhammed Al-MulhemVisual Languages

134. 134 Muhammed Al-MulhemVisual Languages Inject (Cont.) The Prograph representation of inject - a nameless operation with one terminal descending into the operation’s icon - is a particularly well-designed graphic representation for this programming concept. When properly used, inject can result in extremely powerful and compact Prograph implementations of complex functions. However, when used improperly, it can result in code that is very difficult to read or debug, like the computed GOTO of FORTRAN.

135. 135 Muhammed Al-MulhemVisual Languages Control Flow Control flow is a combination of annotations on operations and a case structure. A Prograph method can consist of a number of mutually exclusive cases. These cases are somewhat similar to the cases in a Pascal CASE statement or a C switch expression. The main difference is that unlike Pascal or C, the code to determine which case will be executed is inside the case, not outside it.

136. 136 Muhammed Al-MulhemVisual Languages Example Suppose that we want to implement a small function that has one integer input, and if that input is between 1 and 10, the value of the function is computed one way; if it is between 11 and 100, another way, and if it is anything else, yet a third way. In Pascal-like pseudo-code, this function would look something like this:

137. 137 Muhammed Al-MulhemVisual Languages Function Foo( i: Integer): Integer; Begin Case i of 1..10: Foo := Calculation_A(i); 11..100: Foo := Calculation_B(i); OtherwiseFoo := Calculation_C(i); End; {Case} End; {Foo}

138. 138 Muhammed Al-MulhemVisual Languages Control Flow (Cont.) The implementation of Foo in Prograph would also involve three cases, as shown in the next Figure. The control annotations are the small boxes on the right of the match primitives at the top of the first two cases. (Note that the window titles show the total number of cases in a method, and which case this window is, as in “2:3” the second of three cases.)

139. 139 Muhammed Al-MulhemVisual Languages

140. 140 Muhammed Al-MulhemVisual Languages Control Flow (Cont.) In this simple example, the same annotation - a check mark in an otherwise empty rectangle - is used. The semantics of this annotation are: “If this operation succeeds, then go to the next case.” The check mark is the iconic representation of success, and the empty rectangle represents the ‘go to the next case’ notion. If the check mark were replaced by an ‘x’, then the semantics would have been: “If this operation fails, then go to the next case”.

141. 141 Muhammed Al-MulhemVisual Languages Control Flow (Cont.) In addition to the otherwise empty rectangle, there are four other possibilities, and the five different semantics of control annotations are shown together in the next Figure. It is a run-time error and a compile-time warning to have a next-case control annotation in a location where it cannot execute, for example, in the last case of a method.

142. 142 Muhammed Al-MulhemVisual Languages

143. 143 Muhammed Al-MulhemVisual Languages

144. 144 Muhammed Al-MulhemVisual Languages

145. 145 Muhammed Al-MulhemVisual Languages

146. 146 Muhammed Al-MulhemVisual Languages

147. 147 Muhammed Al-MulhemVisual Languages

148. 148 Muhammed Al-MulhemVisual Languages

149. 149 Muhammed Al-MulhemVisual Languages

150. 150 Muhammed Al-MulhemVisual Languages

151. 151 Muhammed Al-MulhemVisual Languages Classification of Visual Programming Looking at the recent work reported in the literature, we see visual programming progressing in number of directions. 1) Visual Environments for different applications. 2) Visual Languages. 3) Theory of Visual Languages.

152. 152 Muhammed Al-MulhemVisual Languages 1) Visualization of Data Typically, the information or data is stored internally in traditional databases, but expressed in graphical form and presented to the user in a special framework. Users can traverse the graphical surface or zoom into it to obtain greater detail with a joystick or a pointing device.

153. 153 Muhammed Al-MulhemVisual Languages Visualization of Data (Cont.) The user can use the mouse to answer many questions without the need of a keyboard. These systems are devoted primarily to using “direct manipulation” as a means of information retrieval, using a graphical view of a database for visualization of the information retrieved.

154. 154 Muhammed Al-MulhemVisual Languages Example A visual environment can be built for the retrieval of information from a database of ships. Each ship will be represented by an icon. Icon shape, color and size represent type, status, and size of the ship.

155. 155 Muhammed Al-MulhemVisual Languages 2) Visualization of Programs The goal of this is to provide programmers and users with an understanding of : –What the program can do –How they work –Why they work –What are the effects

156. 156 Muhammed Al-MulhemVisual Languages This is beneficial for beginners learning to program. professionals dealing with various aspects of programming tasks: –Designing, –Debugging, –Testing, and –Modification.

157. 157 Muhammed Al-MulhemVisual Languages BALSA BALSA ( Brown Algorithm Simulator and Animator ) developed by Sedgewick and his colleages in Brown University. [ Sedgwick, 1983 ] It enables users : –to control the speed of an animated algorithm –to decide which view to look at. –to specify the input data to be processed.

158. 158 Muhammed Al-MulhemVisual Languages BALSA include the following animated algorithms: Mathematical algorithms such as –Euclid’s GCD algorithm. –Random numbers. –Curve fitting. Sorting algorithms such as –Insertion sort. –Quick sort. –Merge sort. –External sort.

159. 159 Muhammed Al-MulhemVisual Languages Searching algorithms such as –Sequential search. –Balanced trees. –Hashing. –Radix searching. Selecting an icon from the iconic table with a mouse causes a 10 - 15 minute dynamic simulation of the selected topic to be run, with pauses at key images, after which the user can interact with the algorithms and images.

160. 160 Muhammed Al-MulhemVisual Languages

161. 161 Muhammed Al-MulhemVisual Languages 3) Visualization of Software Design It is very important throughout the software life cycle to understand: –specifications –design –system structure –dependencies among data and components –etc. This observation, together with the success of using graphical techniques has led to the development of a visual environment for software life cycle.

162. 162 Muhammed Al-MulhemVisual Languages The PEGASYS The main purpose of PegaSys is to provide a visual environment for the development and explanation of large program designs (as opposed to detailed algorithms and data structures in a program) PegaSys supports the specification and analysis of data and control dependencies among the components of large programs.

163. 163 Muhammed Al-MulhemVisual Languages A program design is described in PegaSys by a hierarchy of interrelated pictures. Icons denote predefined or user-defined concepts about dependencies in programs.

164. 164 Muhammed Al-MulhemVisual Languages The predefined primitives denote: –objects (such as subprograms, modules, processes, and data structures), –data dependencies (involving the declaration, manipulation, and sharing of data objects), and –control dependencies (dealing with asynchronous and synchronous activities). A PegaSys user can define new concepts in terms of the primitives.

165. 165 Muhammed Al-MulhemVisual Languages Example A network of 4 hosts can be represented in PegaSys as follows: –Each of the four hosts in the network is indicated by an ellipse, –the communication line indicated by a dashed rectangle. –a packet of data indicated by a label on arcs.

166. 166 Muhammed Al-MulhemVisual Languages Dependencies among hosts, packets, and the line are described by the write relation (denoted by the letter W on arrows) and the read relation (denoted by R) The system says that: the broadcast network consists of four hosts that communicate by means of a line. More precisely, processes named Host1,... Host4 write values of type pkt into a module called Line and read values of the same type from the Line module.

167. 167 Muhammed Al-MulhemVisual Languages

168. 168 Muhammed Al-MulhemVisual Languages Visualization of Parallel Systems A number of display facilities have been associated with: »debuggers »performance evaluation tools »program visualization systems

169. 169 Muhammed Al-MulhemVisual Languages Debugger displays Debugger displays attempt to provide some view of program state such as: the pattern of access to shared memory the interprocessor communication of a particular distributed memory architecture the order in which subroutines are called

170. 170 Muhammed Al-MulhemVisual Languages Displays for Performance Visualization Tools Performance visualization tools generally provide a graphical representation of standard metrics such as: Processor utilization Communication load

171. 171 Muhammed Al-MulhemVisual Languages Displays for Program Visualization Systems Program visualization systems provide highly application specific views of: the program's data structures the operations which update these data structures or some more abstract representation of the computation and its progress.

172. 172 Muhammed Al-MulhemVisual Languages Program Graphs Many systems display graphs in which: the vertices represent program entities and the arcs represent call relations or temporal orderings, similar to the graph shown in Fig. 1.

173. 173 Muhammed Al-MulhemVisual Languages Fig. 1. An animated call graph MAIN SELECT PRUNE EXPAND UPDATE PROCESS COUNT BROADCAST

174. 174 Muhammed Al-MulhemVisual Languages Concurrency of the Computation A number of systems use a graph to represent the concurrency of the computation. 1) The concurrency map The concurrency map displays process histories as event streams on a time grid.

175. 175 Muhammed Al-MulhemVisual Languages Each column of the display represents the sequential stream of events for a single process. Each row of the grid represents an interval of time during which the events in different columns may occur concurrently. Every event in one row must occur before any event in the next row.

176. 176 Muhammed Al-MulhemVisual Languages Communication Graphs A number of systems present displays that represent communication among processors through the use of boxes, arrows, and lines. –Boxes represent processors, –Lines represent connections. and –Arrows represent communication events, as shown in Fig. 2.

177. 177 Muhammed Al-MulhemVisual Languages Fig. 2: Communication Graph P6 P7 P3 P2 P4 P5 P0 P1

178. 178 Muhammed Al-MulhemVisual Languages VPL Here we will show some Visual Programming Languages.

179. 179 Muhammed Al-MulhemVisual Languages BLOX The basic elements in BLOX system are icons similar in shape to jigsaw puzzle pieces. Programs are composed by 1) Juxtaposing icons in appropriate ways, so that neighbors interlock. 2) Encapsulating substructure

180. 180 Muhammed Al-MulhemVisual Languages Pascal-BLOX WHILE The “lock and key” metaphor enforces proper syntax.

181. 181 Muhammed Al-MulhemVisual Languages Programs in Pascal - BLOX It uses top-down program design. The code segments could be encapsulated in the appropriate tiles. This include codes corresponding to: –the alternative branches –loop bodies –Boolean expressions associated with IF and WHILE statements.

182. 182 Muhammed Al-MulhemVisual Languages Example Begin S1; If not I1 Then Begin S2; S3; End Else While L1 Do Begin If I2 Then S4 Else S5; S6; End; While L2 Do S7; S8; End.

183. 183 Muhammed Al-MulhemVisual Languages Pascal - BLOX Program

184. 184 Muhammed Al-MulhemVisual Languages Advantages Top-down specification of programs Extension of a familiar metaphors from children's toys. Automatic elimination of the most source of possible syntax errors in user programs, by the use of the “Lock and Key” metaphor. The representation can be applied to diverse and varied activities in a uniform manner.

185. 185 Muhammed Al-MulhemVisual Languages Our VPLs Our efforts to design and implement visual programming languages include: 1) DataLab (with T. Lewis, and H. Kim, 1989) 2) VISO: A Visual Languages for Concurrent Programming (with S. Ali, 1994)

186. 186 Muhammed Al-MulhemVisual Languages VISO VISO is a visual programming language for concurrent programming. It has a graphical syntax based on the language OCCAM. It uses a modular approach of visual programming

187. 187 Muhammed Al-MulhemVisual Languages

188. 188 Muhammed Al-MulhemVisual Languages

189. 189 Muhammed Al-MulhemVisual Languages DataLab DataLab is a general-purpose system for the specification and synthesis of abstract data types (ADTs). DataLab uses a combination of graphical and textual forms as a specification model for ADTs. An imperative semantics is used to automatically generate an encapsulated executable code (Pascal modules) from such a specification.

190. 190 Muhammed Al-MulhemVisual Languages Contribution DataLab extends earlier work by 1) allowing data structures to be created and displayed in their classical notation, 2) generating encapsulated imperative code (Pascal), and 3) incorporating composition of simple structures into more complex structures.

191. 191 Muhammed Al-MulhemVisual Languages Example - Linked List The first step is to define the interface part unitLINKED_LIST; interface type LL_list=^LL_node; LL_node=record data:string; next:LL_list; end; procedure LL_insert(varlist : LL_list;item : string); implementation end.

192. 192 Muhammed Al-MulhemVisual Languages Next, the procedure "LL_insert" is specified by two Conditional transformations 0 node^.data := item; list list^ 1 2:list node^ 4 3 9:next 8 10:list 11 1 node 5 6:node 7

193. 193 Muhammed Al-MulhemVisual Languages 0 2:list3 1 8:next 7 list^.data := item; list list^ 4 5 6:list 9 2

194. 194 Muhammed Al-MulhemVisual Languages What is next for VPLs? More effective use of color and sound. 3-D instead of 2-D graphics. Animated graphics. Languages for concurrent and parallel programming.