Presentation is loading. Please wait.

Presentation is loading. Please wait.

SIGGRAPH 2010 Dmitry Andreev (AND) Real-time Frame Rate Up-conversion for Video Games or how to get from 30 to 60 fps for “free”

Similar presentations


Presentation on theme: "SIGGRAPH 2010 Dmitry Andreev (AND) Real-time Frame Rate Up-conversion for Video Games or how to get from 30 to 60 fps for “free”"— Presentation transcript:

1

2 SIGGRAPH 2010 Dmitry Andreev (AND) dandreev@LucasArts.com Real-time Frame Rate Up-conversion for Video Games or how to get from 30 to 60 fps for “free”

3 Agenda Stories, Ideas, Possibilities Excitement, Inspiration Better Games Happy Players … Technical Details are Simplified –Only current real-time implementation is discussed

4 Real-time DEMO

5 Frame Rate Up-conversion Natural Motion of Objects In Software –AVISynth + MSU FRC –WinDVD (Trimension) In Hardware –Display devices (120Hz HDTVs)

6 Motion Estimation Block-matching –Diamond Search Optical Flow Phase Correlation MAP/MRF or take it from the MPEG stream :) AB Motion Vectors

7 Video Frame Interpolation Build Inner frames from outer frames – Using Motion Vectors – High-res estimation is required Scaling, rotation, transparency, dynamic lighting ??? ABA1A2A3

8 Motion Vectors vs Velocities Estimated vs Rendered Motion Vectors Velocity Buffer

9 No Need for Motion Estimation The Source of Motion is Known –Objects –Camera Handle Special Cases Differently –Transparency –Shadows, reflections, refractions –Characters ?

10 Running at 30 fps Game –Extra 16.6 ms –More advanced rendering techniques Deferred / Screen-space, etc… –Less strict production process (easier to make)

11 Running at 30 fps Game –Extra 16.6 ms –More advanced rendering techniques Deferred / Screen-space, etc… –Less strict production process (easier to make) User –Exposed to the same frame Twice

12 Motion Eye-tracking (60 fps) Observed @ 60Hz Display A, B, C,... Temporal Functions –Display –Eye 01/60 s 0

13 Motion Eye-tracking (30 fps) Observed @ 60Hz Display A, A, B, B, C, C,... Temporal Functions –Display –Eye 01/60 s1/30 s 01/60 s1/30 s

14 Motion Blur Helps to Remedy “Flickering”, but… Happens in the Eye… –Not the camera (virtual) –Should not be used at all At eye-trackable speeds Observed @ 60Hz

15 Motion Blur In Games

16

17 The Idea is … Render Velocity Buffer

18 P n-½ The Idea is simple … Render Velocity Buffer Interpolate the Middle Frame P n-1 PnPn

19 P n-½ The Idea is very simple … Render Velocity Buffer Interpolate the Middle Frame Present at the Right Moment in Time P n-1 PnPn VBlank 1/60 s

20 Ideally (to be “free”) Reuse as Much as Possible –Velocity buffers –Previous frames @ lower-res –Any other data available (e.g. animations) No Extra –Latency –Memory / performance hit

21 P n-1 PnPn Typical Rendering Pipeline Depth Buffer (~2-3 ms) Shadows, Lighting (deferred), SSAO Main Color Pass & Alpha Pass Post Effects (HDR, DOF, Motion blur, DLAA, etc...)

22 PnPn P n-½ Modified Rendering Pipeline Depth Buffer (~2-3 ms) Render Velocity / Interpolate / Schedule to swap Do the Rest of the Current Frame P n GPU is working on P n-1

23 One-frame Based Solution Pros –No extra latency –Little extra memory Cons –Issues with shadows and reflections Manageable –Harder to fix interpolation errors But possible ;)

24 INTERPOLATION

25 Very Basic Interpolation P n-½ = f( P n-1,-½·V n ) –Sample half way backwards –Previous frame P n-1

26 Original I

27 Interpolated Very basic interpolation II

28 Very Basic InterpolationIII Based on Current Velocity P n-½ = f( P n-1, -½·V n ) –Sample half way backwards –Previous frame P n-1 Two Types of Issues –Static geometry tails –Dynamic geometry tails

29 Camera Motions (TFU2)I 3 rd Person Camera Translation

30 Camera Motions (TFU2)II 3 rd Person Camera TranslationRotation

31 Artifact Minimization Static Geometry –Velocity buffer filtering Dynamic Geometry (Characters) –Render separately, re-render completely or... Use Previous Color & Velocity Frames –To fix-up invalid regions –Two-frame solution works best

32 Two-frame Based Solution Pros –Smoother shadows, alpha, reflections –More real data to handle overlaps Cons –Extra frame of latency –More memory and slower

33 Modified Velocity Rendering Re-render Characters –At interpolated positions Use other Channels –To help with interpolation and artifact detection –B – 8bit depth buffer –A – 8bit id masks (e.g. characters)

34 Velocity Merging Very Simple and Efficient V merged = V n-1 if ( Z n-1 + ∂ ) < Z n else V n –Z is embedded in V –Works well for background objects –Assuming continuity of motion

35 Original I

36 Interpolated Velocity Merging II

37 Character RemovalI Visual System’s Blind Spot Inspired

38 Character RemovalI Visual System’s Blind Spot Inspired Synthesize the Interior Assuming That –Horizontal remains horizontal –Vertical remains vertical –Texture is preserved –Shading changes gradually

39 Character Removal II Generate Mask

40 Character Removal II Generate Mask Leak Neighboring Image Patches

41 Character Removal II Generate Mask Leak Neighboring Image Patches –Duplicate and offsetUp

42 Character Removal II Generate Mask Leak Neighboring Image Patches –Duplicate and offsetUp Down

43 Character Removal II Generate Mask Leak Neighboring Image Patches –Duplicate and offsetUp Down Left

44 Character Removal II Generate Mask Leak Neighboring Image Patches –Duplicate and offsetUp Down Left Right –Merge all layers together

45 Character Removal III Repeat N Times Blur the Interior Slightly original mask pass 1 pass 2 pass 3 finalize

46 Character Removal IV Original Synthesized 3 passes @ 640x360 XBox360 ~ 0.4ms PS3 ~ 0.3ms (5 SPUs)

47 Improved Interpolation Based on V merged –Sample half way backwards –Previous real frame P n-1 –Synthesized frame P' n-1 Based on Masks in V n and P' n-1 –Select P' n-1, if it had a character where V n does not –Else select P n-1

48 Original I

49 Interpolated Velocity Merging and Character Removal II

50 Next Original In case of One-frame based Interpolation it is Unknown III

51 Typical Flipping Flip Normal Buffer –As soon as VBlank is received –Immediately, if VBlank passed (frame rate drops) waitrender wait render wait render

52 Modified Flipping Flip Interpolated Frame Based on Time –Normal conditions –Frame rate drops wait render P PPP PP

53 Custom flipping on XBox360 Bypass Direct3D Present with... Asynchronous Swaps –Multiple front buffers –SwapCallback, VerticalBlankCallback Manual Hardware Manipulation –D1-GRPH_PRIMARY_SURFACE_ADDRESS 0x7FC86110 (undocumented)

54 Real-time Implementation One-frame Based Interpolation of Dynamic Objects –Character removal Performance @ 1280x720 –XBox360 ~ 1.5 ms –PS3 (5 SPUs)~ 1.2 ms

55 Motion Blur In TFU2 Current Solution –Alpha “friendly”, simple edge condition –Better than plain 30 fps –But not as good as 60 fps Performance @ 1280x720 –XBox360~ 2.2 ± 0.4 ms (5-11 samples) –PS3 (5 SPUs)~ 1.4 ± 0.5 ms (16 samples)

56 Future Work Transparency Customizations –Re-render lightsabers, HUD –Multi-layer interpolation Up-scale from Lower Frame Rates –E.g. render environment at 20 or 15 fps Interpolate Shadows and Reflections (or blur) Predictive Performance Adjustment

57 Acknowledgments Szymon Swistun Ruslan Abdikeev Cory Bloyd Cedrick Collomb Axel Wefers

58 Words of Wisdom

59 Thank You

60 Questions? dandreev@LucasArts.com


Download ppt "SIGGRAPH 2010 Dmitry Andreev (AND) Real-time Frame Rate Up-conversion for Video Games or how to get from 30 to 60 fps for “free”"

Similar presentations


Ads by Google