1. Introduction to Programming 3D Applications CE0056-1 Lecture 15 Input and interaction in 3D Environments

2. Topics Interaction Input devices Event-driven input Display lists Menus Picking Buffering Virtual Trackball

3. 1963 Sketchpad The first system to explore pen-based user-interfaces by Ivan Southerland (of Evans and …) broached field of HCI not graphics but important OpenGL doesn’t support interaction directly limits portability not specifically graphics we’ll use GLUT for device interactions

4. Input devices Physical vs. logical Physical mouse, keyboard, etc. perspective of how they interact with system Logical function perspective of what they send application higher-level

5. Input devices Logical examples Output: when we use printf in C, the actual string may be output to a monitor or printer Input: we get the position of a user’s cursor whether we use a mouse or a data tablet Our application doesn’t care at high level Two categories, pointer and keyboard pointers may be mice, pens, trackballs, etc. pointers do not return character codes can return relative or absolute positions

6. Input devices Absolute-positioning (w.r.t. some fixed position) pen returns position w.r.t. tablet (settable) data glove always returns 3D position coords Relative-positioning (w.r.t. current position) mouse pos. always begins where cursor is Trigger & measure trigger: physical input on device (button, key) measure: what device returns to application (button, state, x position, y position)

7. Input modes Request & sample modes request_locator (device_id, &measure); after trigger, wait until input received ex. get x,y pos. after right mouse button depressed sample_locator (device_id, &measure); assumes trigger, get input from a queue ex. get current x,y position of mouse limits user control of program which device to sample from? modal, user must specify context (not natural)

8. Input modes Event mode - the most flexible input mode triggers generate events (events store data) measures are sent to 1) an event queue program checks queue events are examined and acted upon 2) a special-purpose function program associates function with trigger up front function is called with the event trigger generates the function is called a callback

9. Event-driven input Callback for display glutDisplayFunc(display) Callback for mouse glutMouseFunc(mouse) void mouse (int btn, int state, int x, int y) { if (btn == GLUT_LEFT_BUTTON && state == GLUT_DOWN) drawSquare(x,y); if (btn == GLUT_MIDDLE_BUTTON && state == GLUT_DOWN) exit (); }

10. Event-driven input drawSquare() function use new mouse position to draw new square drawSquare(int x, int y) { y=w*h-y; glColor3ub(r, g, b); glBegin(GL_POLYGON) glVertex(x+size, y+size); glVertex(x-size, y+size); glVertex(x-size, y-size); glVertex(x+size, y-size); glEnd(); glFlush(); } What are we doing here? 500 (90, 100) (90, 400) 500 Account for different origins

11. Event-driven input Callback for keyboard glutKeyboardFunc(keyboard); Callback for display glutDisplayFunc(display); call in indirectly with glutPostRedisplay(); after changing parameters, attributes, etc. if nothing has changed, system will avoid drawing Callback for idle glutIdleFunc(idle) perform tasks when nothing else is happening

12. Event-driven input Callback for window resize event glutReshapeFunc(reshape) What do we do in reshape(width, height)? What if new window is bigger or smaller? Do we grow, shrink, or resize the image? What if aspect ratio has changed? Do we similarly resize the image, distort it, or clip it out? Use the new width and height to adjust clip rectangle and viewport sizes gluOrtho2D (0.0, width, 0.0, height); glViewport (0, 0, width, height);

13. Clients/server Model In general Servers perform tasks for clients In OpenGL assumes the OpenGL program is the client assumes workstations are graphics servers that provide display and input services to the OpenGL program for example, the client can display its output on any networked server

14. Display lists History refreshing the screen required special processing via a display processor that output a display list that was stored display memory the display list was rendered fast enough to avoid flicker, leaving the host computer free there is no longer any distinction between host and display processor (unless we count special graphics processing hardware)

15. Display lists In immediate-mode, we compute, rasterize, and display (cube.c) we re-compute to redisplay In retained-mode, we store the results of our computation in a display-list we send an instruction to redisplay disadvantages require memory on server require overhead to create

16. Display list example #define Box 1 glNewList(Box, GL_COMPILE) glBegin(GL_POLYGON) glColor3f(1.0, 0.0, 0.0); glVertex2f(-1.0, -1.0); glVertex2f(1.0, -1.0); glVertex2f(1.0, 1.0); glVertex2f(-1.0, 1.0); glEnd(GL_POLYGON); glEndList();

17. Display list example glMatrixMode(GL_PROJECTION); for (i=1; i<5; i++) { glLoadIdentity(); gluOrtho2D(-2.0*i, 2.0*i, -2.0*i, 2.0*i); glCallList(Box); } Clip rectangle results (left, right, bottom, top) i=1: (-2.0, 2.0, -2.0, 2.0) i=2: (-4.0, 4.0, -4.0, 4.0) i=3: (-6.0, 6.0, -6.0, 6.0) What if we want to change the color each time through? What happens as clip rectangle grows? What if it was huge? What happens as clip rectangle grows? What if it was huge?

18. Display lists Create multiple lists glGenLists(number) generates multiple ids glCallLists displays multiple lists

19. Menus Glut provides pop-up windows define entries link menu to mouse button define callback function for each menu entry

20. Menu example Menu creation is order-dependent Menu creation is order-dependent /*Create a menu named sub_menu with callback named size_menu, with two entries*/ /*Create a menu named sub_menu with callback named size_menu, with two entries*/ sub_menu = glutCreatMenu(size_menu); glutAddMenuEntry (“incr. square size”, 2); glutAddMenuEntry(“decr. square size”, 3); /*Create a menu with callback named top_menu, with two entries*/ /*Create a menu with callback named top_menu, with two entries*/glutCreateMenu(top_menu); glutAddMenuEntry(“quit”, 1); glutAddSubMenu(“resize”, sub_menu); /*Make right mouse button trigger menu event*/ /*Make right mouse button trigger menu event*/glutAttachMenu(GLUT_RIGHT_BUTTON); Menu creation is order-dependent Menu creation is order-dependent /*Create a menu named sub_menu with callback named size_menu, with two entries*/ /*Create a menu named sub_menu with callback named size_menu, with two entries*/ sub_menu = glutCreatMenu(size_menu); glutAddMenuEntry (“incr. square size”, 2); glutAddMenuEntry(“decr. square size”, 3); /*Create a menu with callback named top_menu, with two entries*/ /*Create a menu with callback named top_menu, with two entries*/glutCreateMenu(top_menu); glutAddMenuEntry(“quit”, 1); glutAddSubMenu(“resize”, sub_menu); /*Make right mouse button trigger menu event*/ /*Make right mouse button trigger menu event*/glutAttachMenu(GLUT_RIGHT_BUTTON); quit resizeincr. square size decr. square size incr. square size decr. square size

21. Menu example You write the callback functions You write the callback functions /*top_menu callback checks for quit only*/ /*top_menu callback checks for quit only*/ void top_menu (int id) { if (id == 1) exit(); } /*sub_menu processes resize options */ /*sub_menu processes resize options */ void size_menu (int id) { if (id == 2) size = 2.0*size; else if (id == 3 && size > 1) size = size /2.0; else if (id == 3) printf(“At minimum square size”); glutPostRedisplay(); glutPostRedisplay();} You write the callback functions You write the callback functions /*top_menu callback checks for quit only*/ /*top_menu callback checks for quit only*/ void top_menu (int id) { if (id == 1) exit(); } /*sub_menu processes resize options */ /*sub_menu processes resize options */ void size_menu (int id) { if (id == 2) size = 2.0*size; else if (id == 3 && size > 1) size = size /2.0; else if (id == 3) printf(“At minimum square size”); glutPostRedisplay(); glutPostRedisplay();}

22. Picking User selection of object on screen In immediate mode, we compute, rasterize and display. How does this present a problem for identifying objects on screen? we pick a pixel how to work backward from pixel to object? Several solutions render each object to its own clip rectangle and viewport for every position in frame buffer we add an obj id to a lookup table that is examined at each pick for every object we compute a bounding rectangle or box (or extents)*

23. Picking What if objects overlap? return list of objects, not just one What if cursor is near but not on object? bounding box will accommodate some also, you can leave slack in accuracy when you check position against object extents What if pick is in a virtual 3D environment? we will cast a ray into the scene and check each object for an intersection, returning list

24. Buffering Single buffering objects are always rendered into the same framebuffer, which is always being displayed delay of clearing of and re-drawing into frame-buffer will cause flicker if re-draw takes longer than refresh or refresh and animation not synced Double buffering keep two buffers (front and back) always display front, always render into back* swap front and back when rendering complete

25. Virtual Trackball Following slides developed by Ed Angel Professor of Computer Science, Electrical and Computer Engineering, and Media Arts University of New Mexico

26. Physical Trackball The trackball is an “upside down” mouse If there is little friction between the ball and the rollers, we can give the ball a push and it will keep rolling yielding continuous changes Two possible modes of operation Continuous pushing or tracking hand motion Spinning

27. A Trackball from a Mouse Problem: we want to get the two behavior modes from a mouse We would also like the mouse to emulate a frictionless (ideal) trackball Solve in two steps Map trackball position to mouse position Use GLUT to obtain the proper modes

28. Trackball Frame origin at center of ball

29. Projection of Trackball Position We can relate position on trackball to position on a normalized mouse pad by projecting orthogonally onto pad

30. Reversing Projection Because both the pad and the upper hemisphere of the ball are two-dimensional surfaces, we can reverse the projection A point (x,z) on the mouse pad corresponds to the point (x,y,z) on the upper hemisphere where y = if r  |x|  0, r  |z|  0

31. Computing Rotations Suppose that we have two points that were obtained from the mouse. We can project them up to the hemisphere to points p 1 and p 2 These points determine a great circle on the sphere We can rotate from p 1 to p 2 by finding the proper axis of rotation and the angle between the points

32. Using the cross product The axis of rotation is given by the normal to the plane determined by the origin, p 1, and p 2 n = p 1  p 2

33. Obtaining the angle The angle between p 1 and p 2 is given by If we move the mouse slowly or sample its position frequently, then  will be small and we can use the approximation | sin  | = sin 

34. Implementing with GLUT We will use the idle, motion, and mouse callbacks to implement the virtual trackball Define actions in terms of three booleans trackingMouse : if true update trackball position redrawContinue : if true, idle function posts a redisplay trackballMove : if true, update rotation matrix

35. Example In this example, we use the virtual trackball to rotate the color cube we modeled earlier The code for the colorcube function is omitted because it is unchanged from the earlier examples

36. Initialization #define bool int /* if system does not support bool type */ #define false 0 #define true 1 #define M_PI 3.14159 /* if not in math.h */ int winWidth, winHeight; float angle = 0.0, axis[3], trans[3]; bool trackingMouse = false; bool redrawContinue = false; bool trackballMove = false; float lastPos[3] = {0.0, 0.0, 0.0}; int curx, cury; int startX, startY;

37. The Projection Step void trackball_ptov(int x, int y, int width, int height, float v[3]){ float d, a; /* project x,y onto a hemisphere centered within width, height, note z is up here*/ v[0] = (2.0*x - width) / width; v[1] = (height - 2.0F*y) / height; d = sqrt(v[0]*v[0] + v[1]*v[1]); v[2] = cos((M_PI/2.0) * ((d < 1.0) ? d : 1.0)); a = 1.0 / sqrt(v[0]*v[0] + v[1]*v[1] + v[2]*v[2]); v[0] *= a; v[1] *= a; v[2] *= a; }

38. glutMotionFunc (1) void mouseMotion(int x, int y) { float curPos[3], dx, dy, dz; /* compute position on hemisphere */ trackball_ptov(x, y, winWidth, winHeight, curPos); if(trackingMouse) { /* compute the change in position on the hemisphere */ dx = curPos[0] - lastPos[0]; dy = curPos[1] - lastPos[1]; dz = curPos[2] - lastPos[2];

39. glutMotionFunc (2) if (dx || dy || dz) { /* compute theta and cross product */ angle = 90.0 * sqrt(dx*dx + dy*dy + dz*dz); axis[0] = lastPos[1]*curPos[2] – lastPos[2]*curPos[1]; axis[1] = lastPos[2]*curPos[0] – lastPos[0]*curPos[2]; axis[2] = lastPos[0]*curPos[1] – lastPos[1]*curPos[0]; /* update position */ lastPos[0] = curPos[0]; lastPos[1] = curPos[1]; lastPos[2] = curPos[2]; } glutPostRedisplay(); }

40. Idle and Display Callbacks void spinCube() { if (redrawContinue) glutPostRedisplay(); } void display() { glClear(GL_COLOR_BUFFER_BIT|GL_DEPTH_BUFF ER_BIT); if (trackballMove) { glRotatef(angle, axis[0], axis[1], axis[2]); } colorcube(); glutSwapBuffers(); }

41. Mouse Callback void mouseButton(int button, int state, int x, int y) { if(button==GLUT_RIGHT_BUTTON) exit(0); /* holding down left button allows user to rotate cube */ if(button==GLUT_LEFT_BUTTON) switch(state) { case GLUT_DOWN: y=winHeight-y; startMotion( x,y); break; case GLUT_UP: stopMotion( x,y); break; }

42. Start Function void startMotion(int x, int y) { trackingMouse = true; redrawContinue = false; startX = x; startY = y; curx = x; cury = y; trackball_ptov(x, y, winWidth, winHeight, lastPos); trackballMove=true; }

43. Stop Function void stopMotion(int x, int y) { trackingMouse = false; /* check if position has changed */ if (startX != x || startY != y) redrawContinue = true; else { angle = 0.0; redrawContinue = false; trackballMove = false; }

44. Quaternions Because the rotations are on the surface of a sphere, quaternions provide an interesting and more efficient way to implement the trackball Quaternions will be covered later