1. THINC: A Virtual and Remote Display Architecture for Desktop Computing Ricardo A. Baratto Network Computing Laboratory Columbia University

2. isolation...
The advent of network technology has allowed us to move from a world of complete computer isolation ...

3. ... connectivity
to a world of connectivity with devices of all kinds are able to communicate with each other.
Furthermore, as network capacity and ubiquity have increased, we have come to witness a phenomenon that has been denominated...
Source: Internet Mapping Project (

4. dis-integration of the computer clusters and grid computing
network storage
the dis-integration of the computer where our computing environment is no longer confined to a single box on our desktop, but instead it's spread out all around us, and networks replace what once were internal communication paths in the computer.
two of the most popular examples of this phenomenon are <click> network storage and <click> clusters and grid computing
a third example of the dis-integration of the computer is...

5. remote display display updates input ...remote display.
remote display is a client/server architecture that decouples a computer from the devices used to interact with it, such as the monitor, keyboard, and mouse, and <click> uses the network to provide a communication channel between these devices and the computer.
the mode of operation is simple: graphical output from the computer's desktop that would normally be sent to the local video hardware, is instead <click> redirected over the network to a client to be displayed. similarly, in response to the user interacting with the desktop, input events are generated and <click> forwarded from the client back to the server.

6. benefits
given the number of benefits associated with it, it's not surprising the popularity that remote display systems enjoy today.

7. ubiquitous access
quick: for example, it enables ubiquitous access to our desktop. all we need is a network connection and a simple client device.
complete: for example, it enables ubiquitous access to our desktop, and all we need is a network connection and a client. and since the required client functionality is so basic, it can be provided almost everywhere, using anything from a simple viewer application to specialized access devices

8. remote collaboration
quick: you can collaborate remotely by sharing access to the same computer
sosp: as the display output is redirected over the network, it can also be replicated and forwarded to multiple clients simultaneously. in this manner, groups of globally distributed users can collaborate by sharing access to a single computer and the applications and instruments on it.

9. online help
in a similar manner, remote display creates a new paradigm for providing live computer help and technical support. sharing the desktop provides the perfect environment for either showing somebody else how to perform a computer task, or help them troubleshoot a problem.

10. application processing
thin clients
application processing
and data
secure server room
stateless client
quick: and finally, they're the key technology behind thin clients, where all application logic and data are moved to a secure location and access is done through very simple, stateless devices.
sosp: finally, remote display is the core technology enabling thin clients. A thin client system moves all application processing and data to centralized servers and secure server rooms, and using a remote display protocol provides access to these servers from simple and stateless devices.
the major benefit of the thin-client model is the reduced management costs associated with it. this reduction comes from the fact that the edges of the network, where management is more costly, are composed of simple devices that require little or no maintenance.

11. existing systems
remote display has been around for some time now and as a result a number of remote display systems exist today. among the most popular are ...
however, the remote display world as it stands today is suffering from a fundamental performance problem <click>

12. what's wrong?
problem:
focused on office applications on LAN and reducing data transfer on low bandwidth links
therefore:
poor support for display intensive interactive applications
poor support for higher latency WAN environments
Most existing remote display systems have focused on supporting office applications on local area network environments or reducing data transfer for low bandwidth links.
as a result, they are unable to provide a high fidelity visual and interactive experience for display intensive applications which form the basis of today's desktop computing
and they are unable to efficiently support access on higher latency wide area network enviroments

13. virtual and remote display architecture for desktop computing
THINC
virtual and remote display architecture
for desktop computing
In this context this dissertation introduces THINC, a virtual and remote display architecture for desktop computing designed to address the challenges facing current remote display technology

14. contributions high performance remote display system [SOSP 05]
first to natively support multimedia applications
superior remote display support for mobile devices [WWW 06, SCC 06]
in particular, this dissertation introduces the following contributions.
- first, we introduce a high performance remote display system for LAN and WAN environments. our system differs from existing technologies in that we focus on the architecture of the system and not just on the remote display protocol, as a mechanism to improve performance..
- second, we introduce the first remote display system to natively support multimedia content in a way that is both transparent to applications and format independent
- third we present a system to deliver superior remote display support for mobile devices, both in terms of performance and usability. our system also provides a competitive alternative to native mobile applications

15. contributions II
MobiDesk: desktop utility computing infrastructure [MobiCom 04, best student paper award]
A2M: protect MobiDesk from DDoS attacks
beyond remote display: desktop recording [SOSP 07]
fourth, we introduce mobidesk, a desktop utlity computing infrastructure that enables service providers to host desktop sessions in fully virtualized environments. hosted sessions can be remotely accessed using THINC's remote display, and they can be migrated across computers to provide high-availability
- fifth, we present A2M, a system to protect MobiDesk type infrastructures from DDoS attacks by combining an indirection based network and THINC's remote display architecture. A2M is able to provide protection with low overhead by exploiting the asymmetric characteristics of remote display, where the uplink...
- finally, we move beyond remote display and show how THINC's architecture can be used to provide continuous, low overhead recording of a desktop. we also introduce a novel way to leverage desktop a11y services to allow users to search recording based on captured text content.

16. THINC remote display simple display protocol
COPY, Solid Fill, Tile Fill, BITMAP, RAW
focus on architecture of the system to improve performance
interception
translation
delivery

17. interception: traditional display pipeline
applications
window
system
device
driver
framebuffer
our first architectural component is the point at which graphical output from the desktop is intercepted so it can be redirected over the network.
Pictured here is the display pipeline of a typical desktop system. At the top are desktop applications, such as web browser/ . Below it is the window system, which is in charge of managing display resources and maintaining the state of the desktop. As application requests are generated, the window system translates them into commands suitable for the display hardware. These commands are passed on to the device driver, a piece of code that knows how to drive a particular hardware device. finally at the bottom is the framebuffer, which contains the contents of the desktop which are displayed on the screen.
given this display pipeline there are a number of places we can perform interception.

18. interception applications window system stateful client hurts mobility
device
driver
framebuffer
high-level requests
stateful client hurts mobility
app – window system synchronization
the first option is to intercept high-level requests between the applications and the window system , an approach used by X and X-based proxies such as NX. In this model, everything below the window system is executed on the client. Intercepting at this level has the advantage of giving the rd system complete knowledge and control over the system. unfortunately it all has two serious drawbacks. first, since the window system is at the client, ...
second, since an application's logic and its user interface are typically tightly coupled, running the ui on the client and the logic in the client may result in a need for continuous synchronization over the network. In high-latency wide are network environments this kind of synchronization results in substantial performance degradation.

19. interception applications window system
framebuffer
device
driver
high-level requests
lose semantics: expensive encoding
at the other end of the
raw pixels

20. virtual display architecture
...
this model has a number of benefits

21. benefits
first, using the standard device driver interface allows THINC to provide remote display in a manner that is transparent to the applications and the window system.

22. benefits
second, since the device driver sits below the window system it can leverage existing window system technology instead of having to reimplement, and it can readily take advantage of improvements in commodity window systems

23. benefits
third, it enables us to have a simple, low-level protocol that closely mimics the kind of low-level requests delivered to a traditional device driver.

24. benefits
finally, the simple low level protocol means that clients can be simple and stateless. in most cases, all they are required to do is read commands from the network and pass them down to their local hardware.

25. use and preserve semantic information for efficient translation
how do we translate from application commands to the display protocol?

26. translation
use semantic information when doing translation

27. use request semantics to generate update
req: fill window W, color C
application
window
system
req: fill [x,y,w,h] color C
THINC
update: solid fill
[x,y,w,h] color C

28. translation use semantic information when doing translation
preserve semantic information throughout the system

29. why preserve semantics: offscreen rendering draw offscreen regions
abcde
copy
abcde
display

30. offscreen rendering (cont) offscreen region command log
merge, clip, and discard commands as needed

31. maximize interactive response of the system
delivery
maximize interactive response of the system
how and when do we send display updates?

32. delivery transmit updates as soon as possible
merge, clip, and discard updates as needed
... not all display content created equal
in order to maximize the responsiveness of the system, the server transmits updates to the client as soon as possible, by pushing them on the network. this approach guarantees minimum delay plus it doesn't suffer as much as network latency increases
to properly adapt to varying network conditions and slow responding clients, coalesce + discard old updates

33. shortest remaining size first scheduler real time client buffer C1 C2
... Cn
. . .
queue 1
queue n
cmd size
srsf analogous to srpt
srpt optimal for minimizing mean response time
as updates are generated, sort them in queues.
the queues are flushed in increasing size
additional real time queue allows us to prioritize updates that provide feedback to user (e.g. cursor change)

34. experimental results X/Linux prototype web browsing performance
comparison to existing systems
performance on wide area networks
to demonstrate the effectiveness of THINC's rd arch, we implemented a prototype as a device driver for the X window system, and conducted an evaluation of its web browsing performance, first in a direct comparison with a number of state of the art remote display systems, and second by measuring its performance on real WAN environments.
we measured web browsing performance using Mozilla web browser, and a mechanical device that generates mouse clicks a regular intervals. each click causes a new web page to be loaded. our primary measure of performance is page download latency, defined to be the time elapsed between the mouse click and the page being completely rendered on the client.

35. web browsing performance ... up to 4.8 times better performance
the results for our direct comparison results are shown in this graph. the x axis .. the y axis.
- as can be seen from the graph, THINC outperforms all systems, and can provide up to 4.8 times better performance.
two additional points i want to show:
- comparing to VNC and X on the LAN provides validation for our interception approach, and shows that THINC can outperform both X's high level interception, and VNCs low-level pixel based approach.
- comparing to SunRay demonstrates the effectiveness of our translation mechanisms, in particular offscreen. as mentioned before, sunray and thinc's protocol is quite similar, however sunray incurs higher overhead because it lacks support for offscreen rendering which is heavily used by
- finally,
... up to 4.8 times better performance

36. WAN web browsing performance
- subsecond latency except in korea (really far away, 7k miles)
- provides support for our delivery mechanisms, work in high latency environments.

37. multimedia

38. existing systems

39. THINC multimedia native support for video playback bidirectional audio
synchronized audio/video playback
format independent and transparent to applications

40. video playback video player scaling for free YUV leverage client
virtual device
driver
video player
hw video
acceleration
interfaces
scaling for free
YUV
leverage client
hardware

41. audio applications OS audio daemon audio data virtual audio driver

42. experimental results

43. ... perfect playback and up to two orders of magnitude better
a/v playback quality
... perfect playback and up to two orders of magnitude better

44. mobile devices becoming ubiquitous have network connectivity
but limited environment and applications
- mobile devices pretty much ubiquitous. cellphones, pdas. would be nice if they could serve as extensions of our desktop.
- however they provide a very limited environment and limited applications. constrained by low power, need to conserve battery, limited storage.
- but they do have network, and graphics cards

45. pTHINC provide superior remote display support for mobile devices
competitive alternative to limited mobile applications

46. server-resized updates ?

47. user interface zoom, scroll, rotate
full screen by leveraging external controls

48. experimental results

49. PDA web browsing performance
... up to 17x faster than other systems, and up to 8x faster than native

50. native vs pTHINC

51. btw

52. limitations need better compression NX/SunRay/VNC, adaptive multimedia
too much bandwidth?
limited on mobile devices
flash video
no support for new generation of high- end desktops and 3D applications
TCP:
- we have no provisions for lost/corrupted data
- we need for non-blocking operation

53. impact Technology licensed for commercial use
VESA standard in progress

54. conclusions
high performance remote display system able to outperform existing solutions
first to provide full support for multimedia content, transparently to applications and independent of format
superior remote display support for mobile devices
This dissertation introduced THINC, a remote and virtual display architecture for desktop computing.
We presented the design and evaluation of a high performance remote display system that is able to outperform existing solutions by focusing our efforts, not only on encoding and protocol primitives, but also on the architecture of the system
We also presented the first system able to provide full support ...
Finally, we showed a solution which can provide superior remote display support for mobile devices, both in terms of performance and display quality, and which is also able to provide a better experience that native applications.

55. backup

56. Page-by-page comparison
THINC faster on all pages except image-only
Our approach works better than both low and high-level on non-image content

57. Microbenchmarks
It's not each individual component, but the combination
Not clear microbenchmark results can be extrapolated to overall performance
But do have some results:
No offscreen: ~70% (all RAW)
Only interception and redirection: ~225%

58. future work remote display standard (Net2Display) 3D
indexing and searching improvements

59. list of publications
MobiDesk: Mobile Virtual Desktop Computing [MobiCom 04]
THINC: A Virtual Display Architecture for Thin-Client Computing [SOSP 05]
Remotely Keyed Cryptographics: Secure Remote Display Access Using (Mostly) Untrusted Hardware [ICICS 05]
pTHINC: A Thin-Client Architecture for Mobile Wireless Web [WWW 06]
An Application Streaming Service for Mobile Handheld Devices [SCC 06]
Net2Display: A Proposed VESA Standard for Remoting Displays and I/O Devices over Networks [ADEAC 06]
DejaView: A Personal Virtual Computer Recorder [SOSP 07]

60. beyond remote display: desktop recording
provide recording of all desktop output
allow recording to be searched
we want to:
- provide users with a recording of all output that is generated on their desktop, no matter if they saved it to disk, bookmarked it, etc. idea is no need for that.
- the recording should behave like a movie. can pause, rewind, fast forward, random access to any point in time
- since it will be a large amount of data need to provide a mechanism to search the information

61. desktop recording accessibility applications services virtual device
driver
visual
output
accessibility
services
text
output
text capture
daemon
visual
record
client/
viewer
index

62. recording runtime overhead

63. storage growth
desktop most important: only 32 KB/s on average
32KB/s

64. wan a/v playback quality
perfect except when connecting from korea (really far away, 7k miles)

65. mouth-to-ear latency overhead

66. PDA video playback quality

67. web browsing data transfer

68. A/V data transfer

69. PDA web browsing data transfer

70. PDA video data transfer

71. browse and search latency

72. playback speedup

73. web browsing performance ... up to 4.8 times better performance
- almost everybody performs well (subsecond) better than everybody in both LAN and WAN (obvious).
- compared to SunRay shows the effectiveness of our translation mechanisms, in particular offscreen. we have also measured just for THINC and found lack of offscreen caused a 70% increase in latency.
- compared to VNC shows the
... up to 4.8 times better performance

74. existing performance problem
This graph shows the results of displaying a video clip on the most popular remote display systems, and measuring the playback quality as perceived by the client.
As you can see, none of the systems is able to provide performance that is even close to that of today's desktop computer.

75. implementation X/Linux and windows display server
Linux audio driver (alsa) + daemon
X/Linux windows, PDA, and Java clients
X/Linux desktop recording
capture using ATK (Gnome)
index using Postgres + Tsearch2

76. offscreen drawing draw offscreen regions abcde copy abcde display

77. command queues offscreen region command queue

78. copy onscreen client queue 1 2 3 1 2 3

79. how? applications client hardware caps video

80. how we deliver updates display updates client buffer C1 C2 C3 ... Cn

81. desktop virtualization
decouple desktop from underlying video hardware
encapsulate window and graphical state
can move around:
carry in pocket
around hosting servers

82. MobiDesk

83. A2M: Access Assured Mobile Desktop Computing

84. old slides

85. contributions high performance remote display system [SOSP 05]
focus on system architecture to improve performance
outperforms existing systems
first to natively support multimedia applications
transparent and format-independent
wireless mobile device support [WWW 06, SCC 06]
server-side display scaling for improved display quality and performance
user interface tailored for constrained environment
in particular, this dissertation:
- introduces a remote display architecture for high performance remote display on LAN and WAN environments.

86. contributions II
MobiDesk: desktop utility computing infrastructure [MobiCom 04]
fully virtualized environment for hosting desktops
hosted sessions can be migrated for high availability
A2M: protect MobiDesk from DDoS attacks
indirection-based network + remote display
exploit asymmetric traffic to minimize overhead
beyond remote display: desktop recording [SOSP 07]
low overhead recording, and efficient access
novel use of accessibility services for indexing and searching

87. display protocol Inspired on Sun Ray protocol 2D Primitives copy
solid and tile fill
bitmap fill
raw

88. demo

89. video leverage existing hardware acceleration interfaces
YUV (luminance-chrominance) color space
format independence
client hardware acceleration (scaling for free)

90. YUV Standard hardware interface Format independence
Hardware acceleration (fullscreen for free!)

91. isolation...
The advent of network technology has allowed us to move from a world of complete computer isolation ...

92.                                             
... and a PC

93. system architecture as important as protocol and encoding

94. goals
minimize latency
simple and portable
transparent operation

95. application requests display updates translate commands deliver THINC
Translation decoupled from delivery
THINC

96. basic static translation Draw API standard device driver commands
THINC commands

97. THINC high performance remote display LAN and WAN environments
transparent operation in exisiting desktop systems
full screen, full motion audio/video playback

98. server-resized updates

99. system architecture
so what does THINC's architecture look like?

100. two key problems
how do we translate from application commands to the display protocol?
how and when do we send display updates?

101. but... performance problems
issues supporting display intensive interactive applications
issues coping with higher latency network environments
lack of “features”
limited or no support for multimedia content
subpar support for mobile devices

102. system architecture is key
interception
translation
delivery