Semantic multimedia remote display for mobile thin clients

Abstract

Current remote display technologies for mobile thin clients convert practically all types of graphical content into sequences of images rendered by the client. Consequently, important information concerning the content semantics is lost. The present paper goes beyond this bottleneck by developing a semantic multimedia remote display. The principle consists of representing the graphical content as a real-time interactive multimedia scene graph. The underlying architecture features novel components for scene-graph creation and management, as well as for user interactivity handling. The experimental setup considers the Linux X windows system and BiFS/LASeR multimedia scene technologies on the server and client sides, respectively. The implemented solution was benchmarked against currently deployed solutions (VNC and Microsoft-RDP), by considering text editing and WWW browsing applications. The quantitative assessments demonstrate: (1) visual quality expressed by seven objective metrics, e.g., PSNR values between 30 and 42 dB or SSIM values larger than 0.9999; (2) downlink bandwidth gain factors ranging from 2 to 60; (3) real-time user event management expressed by network round-trip time reduction by factors of 4–6 and by uplink bandwidth gain factors from 3 to 10; (4) feasible CPU activity, larger than in the RDP case but reduced by a factor of 1.5 with respect to the VNC-HEXTILE.

Appendices

Appendix 1: Thin client application panorama

In practice, the VNC and RDP technological supports are exploited by a large variety of ready-to-use applications. While an exhaustive list of such applications is practically impossible to be done and is also out of the scope of our paper, consider for instance:

• VNC-based applications: Apple Remote Desktop [67], Cendio ThinLinc [68], Chicken [69], ChunkVNC [70], Crossloop [71], EchoVNC [72], Ericom [73], Goverlan Remote Control [74], NoMachine NX [75], iTALC [76], KRDC [77], Mac HelpMate [78], N-central [79], noVNC [80], RealVNC [81], RapidSupport [82], Remote Desktop Manager [83], TigerVNC [84], TightVNC [85], TurboVNC[86], UltraVNC [87], X11vnc [88];

• RDP-based applications: AnywhereTS [89], Citrix XenApp [90], CoRD [91], DualDesk [92], Ericom [73], FreeRDP [93], NoMachine NX [75], KRDC [77], N-central [79], rdesktop [94], Remote Desktop Manager [83], Techinline [95], xrdp [96], XP/VS Server [97].

Moreover, proprietary (undisclosed) remote viewers’ technological support and applications are also available. For instance, TeamViewer [98] exploits NAT (Network Address Translation) for establishing a connection, based on the RFB (VNC) and RDP protocols. GoToMyPC [99] exploits the Citrix ICA (Independent Computing Architecture) proprietary protocol, but also supports VNC (RFB) and RDP. PhoneMyPC [100] exploits its proprietary protocol, without exposing any specification detail. Oracle and Sun Microsystems offer Virtual Desktop Infrastructure (VDI) [101] solution based on the proprietary Appliance Link Protocol (ALP). Unfortunately, all these solutions are not open for research (undisclosed specification) and development (no source code); consequently, they cannot be objectively benchmarked in our research study.

All these applications may provide additional levels of functionalities, like built-in encryption, file transfer, audio support, multiple sessions, seamless window, NAT pass-through, IpV6 support, video or 3D. The study, reported in the present paper, is placed at the technological support level; consequently, the actual application peculiarities will not be further discussed.

Appendix 2: Semantic content management

When trying to optimize content transmission from server toward mobile thin clients, three directions can be exploited: visual content re-usage, client memory control and network adaptability.

Regardless of its type (text editing, WWW browsing, etc.), each X application can periodically generate identical visual content. Such a case does not only occur when refreshing the screen but also when dealing with some fixed items (frequent letters/words typing, icons or menus displaying, etc.) or with repeated user actions (mouse over, open, save, etc.). Consequently, significant bandwidth reduction is a priori likely to be obtained by re-using that content directly on the client side, instead of resending it through the network. However, to take practical advantage of this concept, a tool for automatically detecting the repetition of the visual content and for its particular management should be provided.

The thin client memory is limited by its hardware resources. While content re-use would suppose, as a limit case, the caching of all the visual content previously generated in a session, the limited memory resources requires a mechanism for dynamically adjusting the cached information. Several implementation choices are available, from a fixed time window to more elaborated decisions based on actual frequency of usage or on the content semantic.

For mobile clients, the network connection is prone to dynamic changes (bandwidth variation, random errors, etc.). Moreover, the mobile user might also want to switch the terminal during a work session (e.g., from a smartphone to a tablet). Consequently, the encoding parameters should be real-time adapted to the client/network conditions. Unfortunately, the nowadays MPEG-4 scene representation technologies do not provide a direct solution to this issue. Moreover, the flexibility requirement set on the Compression & Transmission block forced us to map this functionality to the Scene-graph Manager level.

In the sequel, a functional solution jointly addressing the three above-mentioned aspects is illustrated for the particular case of image content by the flowchart in Fig. 12.

Fig. 12

Flowchart for image management

The scene updating starts by detecting an image generated by the application and by reading some external network/client parameters. The image, its semantic information and the network/client parameters are combined so as to establish some encoding parameters (e.g., a low bandwidth and a low-resolution display would lead to a low quality factor for a JPEG compression).

Then, the existence of this image in the scene graph is checked. This task is achieved by computing the MD5 hash of the image and by searching it into a list containing the hashes of all images already used in the scene graph. According to the way in which this list is organized (from a simple hash record to more sophisticated relations between hash, its usage, its time stamp, etc.), different functionalities can be provided by this module.

In the case when the image already exists in the scene, a simple reference (pointer) to the corresponding image is created. Otherwise, the hash record list is updated and the new image with its encoding parameters is placed in a new node (or, in several nodes) in the scene graph.

As a side effect of this mechanism, the memory resources required by the client are increased. Hence, for thin clients, the image re-using should be restricted in time. In our implementation, we combined some temporal and spatial information about the cached images: assuming some images in the scene are not visible (i.e., they are covered by other visual elements) for more than τ seconds (in the experiments, τ = 180 s), they are removed from the scene and the hash record list is updated accordingly. Of course, different decision making rules can be implemented here: while directly impacting the system performances, they would not affect the architecture generality.

Finally, the BiFS/LASeR scene is updated so as to take into account of these changes: adding a new image/pointer to an image and remove some old images.