Presentation is loading. Please wait.

Presentation is loading. Please wait.

Making robust computer vision in games Presented by: Diarmid Campbell.

Similar presentations


Presentation on theme: "Making robust computer vision in games Presented by: Diarmid Campbell."— Presentation transcript:

1 Making robust computer vision in games Presented by: Diarmid Campbell

2 Introduction Who I am: Diarmid Campbell What I do: Run the Vision R&D group Where we do it : Sonys London development studio What we do: Research computer vision for camera based games This talk: Making robust computer vision in games

3 Contents What we do and why The development process Testing and videos Computer Vision Concepts A robust head tracker Marker based Augmented Reality The problems we faced A demo of EyePet

4 Camera based games Camera mountedon the TV You see yourself on the TV Game is overlaid on top of you

5 Past games on PS2

6 Computer Vision is hard Computer vision makes you want to kill yourself -Dr Nick Lord 2009

7 Why is it hard? Humans mange it effortlessly Image is a 2D array of numbers Take 5 images and plot them as a height map

8 Pick the odd one out

9 Odd one out

10 Pick the odd one out Odd one out

11 Pick the odd one out Odd one out

12 Factors affecting the pixels Background objects in scene Orientation/position of objects Lighting/Shadows Occlusion

13 George is in the pixels Not interested in those George was hidden in the pixels Here is an image, what is it of? The general computer vision problem is hard If we constrain the problem, it is much easier (but still hard)

14 Robust Inputs We can use computer vision as an input mechanism Motion detection in EyeToy games Robustness is how consistently an input mechanism does what the player is expecting An input mechanism must be robust

15 Importance of robustness If your fire button only worked 9 times out of 10, you would chuck your controller out.

16 Importance of robustness There are ways around it

17 Importance of robustness Imagine your gun is a champagne bottle

18 Importance of robustness Each button click shakes it Eventually the top blows off The lack of robustness is hidden

19 Importance of robustness Perhaps you need to now fight tortoises instead of warriors

20 Importance of robustness The mechanic is now robust But it is laggy and unresponsive Cannot rely on split-second timing

21 Importance of robustness Illustrates a general point If the game copes well with non- robust inputs It will also cope well with someone not playing it well It creates a skill ceiling Manifests itself as lack of game- play depth

22 Importance of robustness If you want a deep skill base game mechanics Robust input is essential

23 The Development Process Computer Vision ResearcherGame designer Tell me what the game mechanic is and Ill make you a state of the art solution Give me something that works and Ill see what we can make thats fun

24 The chicken and the egg You cannot do one before the other Both development timelines happen in parallel We are still figuring it out Here are some guidelines

25 Research timeline Convinced we can create the technology Something up and running Time Vision tech beta before game reaches alpha

26 Required infrastructure Prototyping environment Matlab Octave Be able to capture videos Runtime algorithms Open CV VXL

27 Videos and testing

28 Computer vision is hard because many variables affect the images The lighting The players clothes The wallpaper Spectators 3D cameras have their own pros and cons

29 Representative videos Videos allow us to capture these variables and test Videos MUST be representative Works in 99% of cases Useless if that 1% appear in 50% of living rooms Make videos early in development Demo: head tracker capturing

30 Head detection videos We run it through different algorithms Cell SDK face detector Show failure modes When it fails we can find the frame it failed in and debug

31 Regression testing Automated testing Run through load of videos Compare with expected results Expected results could be is head visible?

32 When videos arent enough SCEA R&D labs invented the forthcoming PlayStation®Move controller Uses a camera and other sensors to track the controller Videos were good early on But cannot change a video: Lighting Backgrounds Camera settings

33 Solution Video1 Video2

34 Reasons to buy a robot arm (as if you really need persuading) Can test the same motion under many different conditions Can try special hard cases

35 Computer Vision Concepts

36 Videos tell us when it fails How do we fix it? This is the field of computer vision I cannot go into details of techniques Instead I will explain: The common concepts How they link together This should help if you: Read papers Talk to experts

37 Feature extraction Images contain a lot of information This one is 900K

38 Feature extraction Instead of using pixels directly extract high level properties of groups of pixels Result in less data which is more relevant to the problem at hand

39 Feature extraction PS3 Demo: Basic image PS3 Demo: Canny edge detector Invariant to lighting changes Store additional gradient info PS3 Demo: Motion Used in all our camera games PS3 Demo: Feature points Store image patch for each one Can match them frame to frame

40 Likelihood functions Given that we have observed these features, what is the probability that we are observing what we modelled Conditional probability Bayesian statistics underpins most vision algorithm

41 Cost functions = Likelihood functions Some terminology Sometimes you will here about Cost functions They are the same concept: Likelihood goes up with a good match Cost goes down One is (conceptually) the inverse of the other

42 Cost functions Sum of Squared Differences (SSD) SSD High cost = bad match Low cost = good match

43 Cost functions Sum of Squared Differences (SSD)

44 Classifiers Compares observed features to a number of models Tells you which model fits the features best Which model fits best

45 Classifiers: Face example Is this a face?

46 Classifiers: Face example Classic detector (Voila-Jones) Models are trained on example images Classifier

47 Classifiers: Face example PS3 demo

48 Detectors We have a model (with associated state) Given some observed features Detector returns: Is the object present? What its state? Its state (X,Y position/rotation/Human pose)

49 Detectors: Faces again Viola-Jones face detector: Scans a box over the image Different positions and sizes Runs the classifier and returns any positives Recall face detection demo

50 Trackers We have a model, some observed features and the previous state Tracker returns the next state

51 Trackers: Face example PS3 Demo: SSD tracker PS3 Demo: Wand game If we move quickly the tracker gets stuck in a local minimum

52 Learning more Computer Vision Conferences ICCV CVPR ECCV Read papers accepted by conferences Get friendly with an academic Or hire one!

53 Robust Head Tracking

54 Track rotation and scale The SSD based tracker did not track rotation and scale Next iteration of tracker does: X, Y position Scale θ : in plane rotation PS demo: Hager tracker (swap demo)

55 Track rotation and scale Tracked more types of movement But very fragile Problem: A 2D image patch is not a good model of a head

56 Track rotation and scale Does not deal with out-of-plane rotation

57 Track rotation and scale Even in-plane rotation is not right

58 Colour histograms Lets move away from comparing pixels and think about features Consider these images of the same objects

59 Colour histograms If we compared them pixel for pixel they would seem very different But look at a histogram of the colours that appear in them and they look the same

60 Colour histograms Histograms are a feature that throw away all spatial information

61 Where we are now Current system uses: Colour histograms Keeps approximate spatial information

62 Where we are now It has a foreground and a background model – each with its own histograms

63 Where we are now PS3 Demo

64 Marker based Augment Reality (AR)

65 Marker based AR Marker based AR is in a published game: EyePet

66 Camera setup Topics to discuss Camera based games What is EyePet? Improving the tech Future research

67 What the player sees on the TV Virtual Real Topics to discuss Camera based games What is EyePet? Improving the tech Future research

68 Marker based AR We shipped a magic card with the game Allows the players to manipulate virtual objects in 3D

69 Finding the marker Input image Topics to discuss Camera based games What is EyePet? Improving the tech Future research

70 Finding the marker Threshold Topics to discuss Camera based games What is EyePet? Improving the tech Future research

71 Finding the marker Trace outlines Topics to discuss Camera based games What is EyePet? Improving the tech Future research

72 Finding the marker Test for quad shapes Topics to discuss Camera based games What is EyePet? Improving the tech Future research

73 Finding the marker Actually, just keep pairs of quads Topics to discuss Camera based games What is EyePet? Improving the tech Future research

74 Finding the marker Take corner positions Calculate a 2D transform Topics to discuss Camera based games What is EyePet? Improving the tech Future research

75 Finding the marker Match the pattern Topics to discuss Camera based games What is EyePet? Improving the tech Future research

76 Finding the marker Match the pattern Topics to discuss Camera based games What is EyePet? Improving the tech Future research

77 Finding the marker Match the pattern Topics to discuss Camera based games What is EyePet? Improving the tech Future research

78 Finding the marker Match the pattern Topics to discuss Camera based games What is EyePet? Improving the tech Future research

79 Finding the marker Match the pattern Topics to discuss Camera based games What is EyePet? Improving the tech Future research

80 Finding the marker Match the pattern (Yes!) Topics to discuss Camera based games What is EyePet? Improving the tech Future research

81 Finding the marker Decompose the 2D transform Camera projection Model view matrix Use a Kalman filter Topics to discuss Camera based games What is EyePet? Improving the tech Future research

82 Finding the marker And were done….. Topics to discuss Camera based games What is EyePet? Improving the tech Future research

83 Problems we faced

84 Picking the right threshold Threshold to find black and white regions But which one? Many clever solutions – didnt work Brute force approach Try lots (around 60) thresholds

85 Picking the right threshold PS3 Demo: Thresholds PS3 Demo: AR Thresholds

86 Light sensitive matching Pattern matching used Sum of Square Differences (SSD) SSD = 2242 SSD = 14 SSD = 874 Brightness of image affected the score

87 Light sensitive matching Use Normalised Cross Correlation (NCC) instead SSD = 0.8 SSD = 0.9

88 Light sensitive matching New way to look at images An image is an array of numbers We can list out every number and it becomes a vector ,000

89 Light sensitive matching This is a co-ordinate vector in image space Every 100X100 image corresponds to a single unique point in image space

90 Light sensitive matching This is a co-ordinate vector in image space Every 100X100 image corresponds to a single unique point in image space Brightening an image corresponds to scaling the position vector

91 Light sensitive matching When comparing two images SSD corresponds to the distance between them in image space NCC corresponds to their angle SSD θ θ

92 Light sensitive matching Linear algebra is the other pillar of computer vision Feature extracting is just a transformation from one space to another Image space -> Feature space Classifiers are often just planes which divide up the space (e.g. into a region that contains faces and a region that doesnt)

93 Occlusion It is easy to occlude the marker with your fingers

94 Occlusion Put big red handle on and instruct the player to hold it Also put handle on the back

95 Occlusion: Another approach (still in research phase) Edge based tracking Uses AR Marker to initialise Then tracks using edge features PS3Demo (load EyePet)

96 False positives When not occluded, we find the marker (almost) all the time Our home videos showed this False positives were a problem Not represented in our videos Added some Hollywood films to the video tests We knew that no markers were present

97 False positives Saved out all spurious frames

98 False positives Made a number of tweaks to algorithm E.g. Pattern matching whole marker, not just the centre pattern 20 times less false detections

99 EyePet Demo

100 Use motion detection for normal interaction Call Jump Stroke Use AR card for health monitor Screen-facing case Needs stimulation Trampoline Finally Give him a shower

101 Summary What we do and why The development process Testing and videos Computer Vision Concepts A robust head tracker Marker based Augmented Reality The problems we faced A demo of EyePet

102 The End (please fill out your questionnaires)


Download ppt "Making robust computer vision in games Presented by: Diarmid Campbell."

Similar presentations


Ads by Google