1 Intro to Computer Graphics CSCI E 72 Jeff Parker, 2013 Based on lectures by Ed Angel © Ed Angel "Hardware: The parts of a computer system that can be kicked." - Jeff Pesis

2

3 Intro to Computer Graphics CSCI E 72 Jeff Parker, 2013 Based on lectures by Ed Angel © Ed Angel "Hardware: The parts of a computer system that can be kicked." - Jeff Pesis

4 Overview What is this course about? What should you know now? What will you be expected to do? What will you know when you finish? isites.harvard.edu/icb/icb.do?keyword=k97151 Don't need to cover all topics in this lecture tonight

5 What Computer Graphics isn't Let’s talk about what graphics isn’t. Image processing: taking an image and looking for features Red-eye removal: look for red dots, and change them Color correction Finding parts on assembly line Face detection and recognition Machine Vision: application of Image Processing to Robotics

6 What Computer Graphics isn't Graphics User Interfaces: writing user interfaces Menus and widgets Take Software Design Xerox 8010 compound document en.wikipedia.org

7 What Computer Graphics isn't Visualization: Can be used to Tracking a storm Show Gulf Stream currents Surface Temperature NASA NOAA

8 What Computer Graphics is All aspects of creating an image with a computer Modelling the objects in a scene Modelling physics of the objects & interactions Capturing object’s geometry Lines, Points, … Picking a point of view Pick a FOV (Zoom) Rendering the image Irmin Roberts' Dolly Zoom, or "Vertigo effect" Albert Hitchcock's Vertigo Copyright Pixar

9 What skills are important? Mathematics Modelling the objects Following transformations from objects to points on screen Programming Writing programs to create objects Making the calls to the graphics system Creativity Create interesting stories. Draw memorable characters. Some of you will know more of one subject than the other. We will need to help each other out. Wolfram Math World Pixar

Why Study Computer Graphics? Widely used in two major sectors of the US economy Video Games Motion Pictures Used in many other areas, such as Visualization An interesting application of ideas Some very clever algorithms, cutting edge hardware Can help you understand why it takes so long to refresh an image It is fun!

11 Expectations Let's talk about expectations and background Requisites Academic Honesty Engagement Communication

12 Requisites (Preconditions) Gary Larson, The Far Side

13 Requisites (Preconditions) It is often useful in programming to describe preconditions and postconditions What is assumed when you call this routine What will be true when it returns What are our assumptions about you? (what is equivalent to CSCI E 22?) You can program in a modern high level computer language You are comfortable with mathematics You can enter, run, and debug your programs You know JavaScript or can learn it in the next 2 weeks You are comfortable with Abstraction, Callbacks, and Pointers You have the time and the will to work hard for a semester to achieve a worthy goal You will turn in your own work You can turn off your cell phone and pager

14 Student Work Frequent small programming projects covering new material Pen-and-paper problems addressing graphical algorithms We start with series of collision examples – first is 2 circles Graduate students write a final project that uses many of the ideas we cover

15 Academic Honesty A computer program written to satisfy a course requirement is, like a paper, expected to be the original work of the student submitting it. Copying a program from another student or from any other source is a form of academic dishonesty, as is deriving a program substantially from the work of another. At times in this course, we will provide the main idea or even text of a program that is to be completed as an exercise. We will also provide useful idioms in the form of short code fragments. You should have no more reservation about using this help than you would have about using standard formulas in a Physics course. If you are in doubt about any instance of reuse, be sure to credit your source in a comment at the head of your program. You should become familiar with the full set of rules and regulations about the rules at Harvard

16 Academic Misconduct All homework assignments, projects, lab reports, papers, and examinations submitted to a course are expected to be the student’s own work. Students should always take great care to distinguish their own ideas and knowledge from information derived from sources. The term "sources" includes not only primary and secondary material published in print or on-line, but also information and opinions gained directly from other people. The responsibility for learning the proper forms of citation lies with the individual student. Quotations must be placed properly within quotation marks and must be cited fully. In addition, all paraphrased material must be acknowledged completely. Whenever ideas or facts are derived from a student’s reading and research or from a student’s own writings, the sources must be indicated (see also Submission of the Same Work to More than One Course) A computer program written to satisfy a course requirement is, like a paper, expected to be the original work of the student submitting it. Copying a program from another student or any other source is a form of academic dishonesty; so is deriving a program substantially from the work of another.. The amount of collaboration with others that is permitted in the completion of assignments can vary, depending upon the policy set by the head of the course. Students must assume that collaboration in the completion of assignments is prohibited unless explicitly permitted by the instructor. Students must acknowledge any collaboration and its extent in all submitted work.

17 Postconditions: Course Goals You will learn the important ideas of Computer Graphics You will learn a commonly-used API, OpenGL You will have seen some of the most interesting applications of mathematics to Computer Science Sampling Projective Geometry – into the 4 th dimension… You can create some interesting programs You will be exposed to more interesting ideas than we have time to pursue in class

18 Postconditions: Course Goals // You will be able to read a program like this by the end of the semester... public void init(GLDrawable drawable) { float[] mat_specular = {1.0f, 1.0f, 1.0f, 1.0f}; float[] mat_diffuse = {1.0f, 1.0f, 1.0f, 1.0f}; float[] mat_ambient = {1.0f, 1.0f, 1.0f, 1.0f}; float mat_shininess = 100.0f; float[] light_ambient = {0.0f, 0.0f, 0.0f, 1.0f}; float[] light_diffuse = {1.0f, 1.0f, 1.0f, 1.0f}; float[] light_specular = {1.0f, 1.0f, 1.0f, 1.0f}; GL gl = drawable.getGL(); /* set up ambient, diffuse, and specular components for light 0 */ gl.glLightfv(GL.GL_LIGHT0, GL.GL_AMBIENT, light_ambient); gl.glLightfv(GL.GL_LIGHT0, GL.GL_DIFFUSE, light_diffuse); gl.glLightfv(GL.GL_LIGHT0, GL.GL_SPECULAR, light_specular); gl.glShadeModel(GL.GL_SMOOTH); /* enable smooth shading */ gl.glEnable(GL.GL_LIGHTING); /* enable lighting */ gl.glEnable(GL.GL_LIGHT0); /* enable light 0 */ gl.glEnable(GL.GL_DEPTH_TEST); /* enable z buffer */ gl.glClearColor(1.0f, 1.0f, 1.0f, 1.0f); gl.glColor3f(0.0f, 0.0f, 0.0f); }

19 The Instructor Jeff Parker Ph.D. in Mathematics (1980) Teaching since Williams, Amherst, Boston College Harvard Extension Present Middlebury College Merrimack Currently Research Advisor at Harvard Extension Helping students write Thesis Proposals and Theses Industry Experience Phoenix - Postscript Interpreter Prime Computer - Client Server group Sun Microsystems - System Administration Framework Agile Networks (later Lucent) - Layer-3 Switch Nexabit Networks (later Lucent)- Big Fast Router Axiowave Networks – Big Fast Router

20 Communication Be sure to check out the iSite – Check Recent Site Updates Post questions and comments to BBoard To reach me, send . If I think your question is of general interest, I may remove your name and forward your question Note user is Guest – not logged in

21 Communication You can subscribe Click on "my account"

22 Our Textbook A classic text, now headed to 7 th edition (3 rd – 6 th are shown) Publisher has sent us prepublication copies I have placed several versions on reserve Course website has link to sample programs

23 OpenGL We will be writing our programs in JavaScript, using the Application Programing Interface (API) Open GL OpenGL was developed by Silicon Graphics, and runs on Windows, Linux, Mac Compile programs in C or C++ and link with libraries We will be using WebGL Runs on Chrome, FireFox, and Safari Nothing to compile: we use JavaScript interpreter

24 Book Outline: Part 1 Introduction: Chapter 1 What is Computer Graphics? Applications Areas History Image formation Basic Architecture

25 Part II – The OpenGL API Development of the OpenGL API OpenGL Architecture OpenGL as a state machine Functions Types Formats Simple program

26 Book Outline: Part 2 Basic OpenGL: Chapter 2 Architecture GLUT Simple programs in two and three dimensions Interaction GLUT GLU GL GLX, AGL or WGL X, Win32, Mac O/S software and/or hardware application program OpenGL Motif widget or similar

27 Book Outline: Part 3 Three-Dimensional Graphics: Chapters 3-5 Geometry Transformations Homogeneous Coordinates Viewing Shading

28 Book Outline: Part 4 Implementation: Chapter 6 Approaches (object vs image space) Implementing the pipeline Clipping Line drawing Polygon Fill Display issues (color)

29 Book Outline: Part 5 Discrete Methods: Chapter 7 Buffers Bitmaps and Pixel Maps Texture Mapping Compositing and Transparency Mark J. Kilgard

30 Book Outline: Part 6 Modeling – Chapters 8-9 Hierarchical Models Algorithmic Models

31 Book Outline: Part 7 Curves and Surfaces: Chapter 10 Splines and NURBS

32 Book Outline: Part 8 Ray Tracing – Chapter 11 Ray Tracing and Ray Casting Wikipedia

33 Outline: Part 8 Gilles Tran

34 Assignment 1 Goals for the first project: –Configure your browser to display WebGL –Setup a local development environment –Learn how a simple program is put together –Debug the project submission process Project: take a sample program, run it locally, and modify it. Strive for something that is visually interesting.

35 Assignment 1 Drop date is September 10 They strongly suggest that we give an assignment in the first week to give you an idea of what is ahead We will return to describe Assignment 1 Gary Larson, The Far Side

36 SGI and GL There had been a number of companies making graphics hardware Tektronix, Calcomp, … Several of these had private APIs for their systems There had also been a number of efforts to produce an open API IFIPS GKS PHIGS & X Windows – PEX It is always difficult to get an API right the first few times Silicon Graphics Inc. (SGI) revolutionized the graphics workstation by implementing a rich graphics pipeline in hardware (1982) To access the system, application programmers used a library called GL Relatively simple to program 3D interactive applications

37 OpenGL The success of GL lead to OpenGL (1992), a platform- independent API Easy to use Close enough to hardware to get excellent performance Focused on rendering Omitted windowing & input to avoid system dependencies Compare to DirectX

38 OpenGL Evolution Controlled by an Architectural Review Board (ARB) Members include SGI, Microsoft, Nvidia, HP, 3DLabs, IBM,……. Relatively stable (present version 4.4, July 2013) Evolution reflects new hardware capabilities Allows for platform specific features through extensions OpenGL ES (Embedded System) WebGL uses compliant browser to generate OpenGL ES Chrome, Firefox, Opera, Safari

39 OpenGL Libraries OpenGL core library OpenGL32 on Windows GL on most unix/linux systems (libGL.a) glVertex3f(x,y,z) OpenGL Utility Library (GLU) Provides some higher level functions based on the core. gluBuild2DMipmaps GLUT Toolkit (next slide) Provides functionality common to all window systems glutDisplayFunc( display ) Links with window system GLX for X window systems WGL for Windows AGL for Macintosh

40 GLUT OpenGL Utility Toolkit (GLUT) Provides functionality common to all window systems Open a window Get input from mouse and keyboard Menus Event-driven /* Register callbacks */ glutDisplayFunc( display ); glutReshapeFunc( reshape ); glutKeyboardFunc( keyboard ); glutMouseFunc( mouse ); Code is portable but GLUT lacks the functionality of a good toolkit for a specific platform – e.g. No slide bars

41 Software Organization GLUT GLU GL GLX, AGL or WGL X, Win32, Mac O/S software and/or hardware application program OpenGL Motif widget or similar

42 OpenGL Architecture Immediate Mode Display List Polynomial Evaluator Per Vertex Operations & Primitive Assembly Rasterization Per Fragment Operations Texture Memor y CPU Pixel Operations Frame Buffer geometry pipeline

43 OpenGL Functions Primitives Points Line Segments Polygons Attributes Transformations Viewing Modeling Control (GLUT) Input (GLUT) Query

44 OpenGL State OpenGL is a state machine OpenGL functions are of two types Primitive generating Can cause output if primitive is visible How vertices are processed and appearance of primitive are controlled by the state State changing Transformation functions Attribute functions

45 Lack of Object Orientation OpenGL is not object oriented so that there are multiple functions for a given logical function glVertex3f glVertex2i glVertex3dv Underlying storage mode is the same

46 OpenGL function format glVertex3f(x,y,z) belongs to GL library function name type - x,y,z are floats glVertex3fv(p) vector – p is a pointer to an array dimensions

47 WebGL program gasket2 Example 1

48 WebGL program gasket2.html attribute vec4 vPosition; void main() { gl_Position = vPosition; }

49 Vertex Shader attribute vec4 vPosition; void main() { gl_Position = vPosition; }

50 Fragment Shader precision mediump float; varying vec4 vColor; void main() { gl_FragColor = vec4( 1.0, 0.0, 0.0, 1.0 ); }

51 Fragment Shader precision mediump float; varying vec4 vColor; void main() { gl_FragColor = vec4( 1.0, 0.0, 0.0, 1.0 ); }

52 Other files <canvas id="gl-canvas" width="512"" height="512" Oops … your browser doesn't support HTML5 canvas element

53 gasket2.js var canvas; var gl; var NumTimesToSubdivide = 5; var points = []; window.onload = function init() { canvas = document.getElementById( "gl-canvas" ); gl = WebGLUtils.setupWebGL( canvas ); if ( !gl ) { alert( "WebGL isn't available" );

54 gasket2.js var points = []; window.onload = function init() { canvas = document.getElementById( "gl-canvas" ); gl = WebGLUtils.setupWebGL( canvas ); if ( !gl ) { alert( "WebGL isn't available" ); } var vertices = [ vec2( -1, -1 ), vec2( 0, 1 ), vec2( 1, -1 ) ];

55 gasket2.js var vertices = [ vec2( -1, -1 ), vec2( 0, 1 ), vec2( 1, -1 ) ]; divideTriangle( vertices[0], vertices[1], vertices[2], NumTimesToSubdivide); gl.viewport( 0, 0, canvas.width, canvas.height ); gl.clearColor( 1.0, 1.0, 1.0, 1.0 );

56 gasket2.js var program = initShaders( gl, "vertex-shader", "fragment-shader" ); gl.useProgram( program ); var bufferId = gl.createBuffer(); gl.bindBuffer( gl.ARRAY_BUFFER, bufferId ); gl.bufferData( gl.ARRAY_BUFFER, flatten(points), gl.STATIC_DRAW ); var vPos = gl.getAttribLocation( program, "vPosition" ); gl.vertexAttribPointer( vPos, 2, gl.FLOAT, false, 0, 0 ); gl.enableVertexAttribArray( vPos ); render(); }

57 gasket2.js var bufferId = gl.createBuffer(); gl.bindBuffer( gl.ARRAY_BUFFER, bufferId ); gl.bufferData( gl.ARRAY_BUFFER, flatten(points), gl.STATIC_DRAW ); Flatten?

58 gasket2.js var bufferId = gl.createBuffer(); gl.bindBuffer( gl.ARRAY_BUFFER, bufferId ); gl.bufferData( gl.ARRAY_BUFFER, flatten(points), gl.STATIC_DRAW ); Flatten? We have been adding vectors to our vector [ [x1, y1], [x2, y2], [x3, y3],... ] But GPU wants a 1-dimensional vector [ x1, y1, x2, y2, x3, y3,... ] We need to Flatten the vector

59 gasket2.js var bufferId = gl.createBuffer(); gl.bindBuffer( gl.ARRAY_BUFFER, bufferId ); gl.bufferData( gl.ARRAY_BUFFER, flatten(points), gl.STATIC_DRAW ); Static Draw - hint to the GL implementation as to how a buffer object's data store will be accessed. STATIC - The data store contents will be modified once and used many times. DRAW - The data store contents are modified by the application, and used as the source for GL drawing and image specification commands..

60 gasket2.js function triangle( a, b, c ) { points.push( a, b, c ); } function divideTriangle( a, b, c, count ) { if ( count === 0 ) { triangle( a, b, c ); } else { // bisect the sides var ab = mix( a, b, 0.5 ); var ac = mix( a, c, 0.5 ); var bc = mix( b, c, 0.5 );

61 gasket2.js function divideTriangle( a, b, c, count ) { if ( count === 0 ) { triangle( a, b, c ); } else { //bisect the sides var ab = mix( a, b, 0.5 ); var ac = mix( a, c, 0.5 ); var bc = mix( b, c, 0.5 ); --count; // three new triangles divideTriangle( a, ab, ac, count ); divideTriangle( c, ac, bc, count ); divideTriangle( b, bc, ab, count ); }

62 gasket2.js function render() { gl.clear( gl.COLOR_BUFFER_BIT ); gl.drawArrays( gl.TRIANGLES, 0, points.length ); window.requestAnimFrame( render, canvas ); }

63 Assignment 1 Configure your browser to display WebGL images Answer some questions about the API Get the program running on your web server. Make small changes (foreground color – background color) Answer some questions about certain changes to the program Come up with your own modification to the program

64 A Simple Program in pseudocode main()# Draw a sequence of points { initialize_system(); p = initial_point(); for (some number of points) { q = generate_a_point_based_on(p); display_point(q); p = q; } cleanup(); } Example 2

65 Alternative main()# Draw a sequence of points { initialize_system(); p = initial_point(); for (some number of points) { q = generate_a_point_based_on(p); store_point(q); p = q; } display_all_points(); cleanup(); }

66 Alternative #2 main()# Draw a sequence of points { initialize_system(); p = initial_point(); for (some number of points) { q = generate_a_point_based_on(p); store_point(q); p = q; } send_all_points_to_GPU(); display_points_on_GPU(); cleanup(); }

67 gasket1.html % diff gasket1.html gasket2.html 30,31c31, >

68 gasket1.html <canvas id="gl-canvas" width="512"" height="512" Oops... your browser doesn't support the HTML5 canvas element

69 gasket1.js var gl; var points; const NumPoints = 5000; window.onload = function init() { canvas = document.getElementById( "gl-canvas" ); gl = WebGLUtils.setupWebGL( canvas ); if ( !gl ) { alert( "WebGL isn't available" ); }

70 gasket1.js // // Initialize our data for the Sierpinski Gasket // Initialize corners of our gasket with 3 points. var vertices = [ vec2( -1, -1 ), vec2( 0, 1 ), vec2( 1, -1 ) ]; var u = add( vertices[0], vertices[1] ); var v = add( vertices[0], vertices[2] ); var p = scale( 0.5, add( u, v ) );

71 gasket1.js // Add randomly chosen point into array of points points = [ p ]; for ( var i = 0; points.length < NumPoints; ++i ) { var j = Math.floor(Math.random() * 3); p = add( points[i], vertices[j] ); p = scale( 0.5, p ); points.push( p ); }

72 gasket1.js // // Configure WebGL // gl.viewport( 0, 0, canvas.width, canvas.height ); gl.clearColor( 1.0, 1.0, 1.0, 1.0 ); // Load shaders and initialize attribute buffers var program = initShaders( gl, "vertex-shader", "fragment-shader" ); gl.useProgram( program ); // Load the data into the GPU var bufferId = gl.createBuffer();

73 gasket1.js // Load the data into the GPU var bufferId = gl.createBuffer(); gl.bindBuffer( gl.ARRAY_BUFFER, bufferId ); gl.bufferData( gl.ARRAY_BUFFER, flatten(points), gl.STATIC_DRAW ); var vPos = gl.getAttribLocation( program, "vPosition" ); gl.vertexAttribPointer( vPos, 2, gl.FLOAT, false, 0, 0 ); gl.enableVertexAttribArray( vPos ); render(); }

74 gasket1.js render(); } function render() { gl.clear( gl.COLOR_BUFFER_BIT ); gl.drawArrays( gl.POINTS, 0, points.length ); }

75 gasket1.js % diff gasket1.js gasket2.js Many other differences... 75c88,89 < function render() { --- > function render() > { 77c91 < gl.drawArrays( gl.POINTS, 0, points.length ); --- > gl.drawArrays( gl.TRIANGLES, 0, points.length );

76 Chaos Game Chaos Game is due to Michael Barnsley Given a starting dot and a goal triangle (green) Goal is to move dot into green triangle in fewest number of moves Johanna Voolich and Robert L. Devaney

77 Chaos Game

78 Picking the points points = [ p ]; for ( var i = 0; points.length < NumPoints; ++i ) { var j = Math.floor(Math.random() * 3); p = add( points[i], vertices[j] ); p = scale( 0.5, p ); points.push( p ); }

79 Add some comments points = [ p ]; // Compute and store N-1 new points for ( var i = 0; points.length < NumPoints; ++i ) { // Pick one of three corners at random var j = Math.floor(Math.random() * 3); // Compute point halfway between vertex and prior point p = add( points[i], vertices[j] ); p = scale( 0.5, p ); // Add it to the list points.push( p ); } Starting point Second point Second Random Vertex

80 Why does this work? In my version the starting point is in the center of the middle triangle Each time we move halfway towards a corner, we move into a blank triangle We will never land in a shaded triangle So why does the resulting set look the the Sierpinski Gasket?

81 Why does this work? Imagine our random vertex is on the north, so we map all points north At each step, our point is in the center of a blank triangle But at every step, the new triangle is half as big, so point is closer to gasket In the limit, point lies within epsilon of the Sierpinski gasket If the choice of next vertex is not random, we do not get the full set A BC

82 What is left? We have not described The colors we are using to draw, the background How we create a window to hold the image Where the image appears within the window The size of the image How much of the (infinite) image do we display How long will we show the image How we get the points over to the Hardware How we define the "Shaders" the operate on points

83 Emergency In case of an emergency, call my wife at (617)

84 Emergency In case of an emergency, call my wife at (617) If I die, my will is in left hand drawer of my desk

85 Emergency In case of an emergency, call my wife at (617) If I die, my will is in left hand drawer of my desk If the sink backs up, call the plumber at (617)

86 Event Handler If the sink backs up, call the plumber at (617) Tonight we saw one instance window.onload = function init()

87 Event Handler If the sink backs up, call the plumber at (617) Often we wish to prepare for an eventuality We might like to define a routine to do the work However, we might not be around to call the routine ourselves We leave instructions: If this happens, take that action In programming, we can define an Event Handler for some events Also called a Callback Then we register a handler for the event glutMouseFunc ( mouse );/* Registration */ Mouse events will be handled by the function mouse() The Framework we are using defines two things Which events are handled How to register for each type of event: the name and signature

88 Running Local WebGL files You want to run write and run your own WebGL programs I've copied the files to a local directory called AngelExamples I have preserved the directory structure Note that Angel includes.zip files for each directory Now I try to run a program by clicking the icon You may have problems, as the file refer to other files in other directories Fix is to setup your own web server See the Tips page for a full account: here is what I needed to do…

89 Local files % cd AngelExamples % find.../chap2./chap2/gasket1.html./chap2/gasket1.js./chap2/gasket2.html./chap2/gasket2.js..../Common./Common/Angel.js./Common/InitShaders.js./Common/MV.js./Common/webgl-utils.js

90 Run your own HTTP Server % cd AngelExamples % python -m SimpleHTTPServer Serving HTTP on port c

91 Open your browser and load the following URL The HTTP Server must be running in the root directory AngelExample

92 Open your browser and load the following URL The HTTP Server must be running in the root directory

93 Open your browser and load the following URL

94 Collision Problem I am working on my Men In Tights Video Game In this scene, two players are fighting with swords The swords are represented as line segments We represent line segments by a pair of endpoints When the two swords cross, play the audio file "clang.au" Write an algorithm that decides if two line segments overlap. This is too hard, so our first problem is: Do two circles intersect? bbc.co.uk

95 Review Analytic Geometry Points in 2-D space can be represented as pairs (x, y) Points in 3-D space can be represented as triple (x, y, z) Point in 4-D space can be … Lines in the plane can be represented as y = mx + b But we will see better ways How do we represent a line in 3-D space? Circles in the plane can be represented as r 2 = (x – a) 2 + (y – b) 2 But we will see better ways Points and Vectors Difference of two points is a vector We can add vectors – cannot add points We can multiply vectors – dot product and cross product Vectors have Length

96 Problem We want to know if two hoops touch each other. What do we need to know? Any special cases? circumferences

97 Problem We want to know if two hoops touch each other. What do we need to know? Centres and radius of each circle Special Cases?

Credits Non-credited images are from Ed Angel's book or are my creation

99 Assignment 1 Configure your browser to display WebGL images Answer some questions about the API Get the program running on your web server. Make small changes (foreground color – background color) Answer some questions about certain changes to the program Come up with your own modification to the program

What did we do tonight? Introduced some ideas we will see. Stepped through a program that uses shaders. Talked about initial assignment. I hope I have given you a taste of what you will be able to do