Skip to main content

Semantic multimedia remote display for mobile thin clients

Abstract

Current remote display technologies for mobile thin clients convert practically all types of graphical content into sequences of images rendered by the client. Consequently, important information concerning the content semantics is lost. The present paper goes beyond this bottleneck by developing a semantic multimedia remote display. The principle consists of representing the graphical content as a real-time interactive multimedia scene graph. The underlying architecture features novel components for scene-graph creation and management, as well as for user interactivity handling. The experimental setup considers the Linux X windows system and BiFS/LASeR multimedia scene technologies on the server and client sides, respectively. The implemented solution was benchmarked against currently deployed solutions (VNC and Microsoft-RDP), by considering text editing and WWW browsing applications. The quantitative assessments demonstrate: (1) visual quality expressed by seven objective metrics, e.g., PSNR values between 30 and 42 dB or SSIM values larger than 0.9999; (2) downlink bandwidth gain factors ranging from 2 to 60; (3) real-time user event management expressed by network round-trip time reduction by factors of 4–6 and by uplink bandwidth gain factors from 3 to 10; (4) feasible CPU activity, larger than in the RDP case but reduced by a factor of 1.5 with respect to the VNC-HEXTILE.

This is a preview of subscription content, access via your institution.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Notes

  1. All acronyms used throughout the paper are listed in the List of Abbreviations.

  2. The usage of the word semantic in this definition follows the MPEG-4 standard specification [13] and the principles in some related studies [14, 15].

  3. The X window system terminology may be confusing, the user’s terminal being the XServer and the server application being the XClient [3].

  4. The result of the conversion is presented in MPEG-4 BiFS Textual format (BT); an equivalent and alternative way of representing uncompressed BiFS content would be the XMT-A (eXtensible MPEG-4 Textual) format, an XML-based representation defined in [13].

  5. The alternative usage of UDP and TCP with the AJAX HttpRequests for handling the user interaction was already presented in the authors’ previous study [51]; however, the experiments in [51] did not consider the MPEG-4 BiFS ServerCommand.

  6. For other image processing applications, like the quality of television pictures, ITU elaborated recommendations (ITU-R BT 500-12 and BT 1438) [52] precising all the testing conditions and interpretation of the results.

  7. When computing the confidence intervals, the correlation between the images corresponding to successive scene updates was neglected; however, because of the very small variance of the values corresponding to each and every quality metric, the practical relevance of the results is not affected by this approximation.

Abbreviations

AAD:

Absolute Average Difference

AJAX HttpRequest:

Asynchronous JavaScript and XML HyperText Transfer Protocol Request

ALP:

Appliance Link Protocol

AVC:

Advanced Video Coding

BiFS:

Binary Format for Scene

CPU:

Central Processing Unit

CQ:

Correlation Quality

ECMA:

European Computer Manufacturer Association

FLV:

FLash Video

GDI:

Graphical Device Interface

HTML:

HyperText Markup Language

IF:

Image Fidelity

I/O:

Input/Output

iOS:

iPhone Operating System

ISO:

International Organization for Standardization

LASeR:

Lightweight Application Scene Representation

Mac OS:

Apple Operating System

MPEG:

Moving Picture Expert Group

NCC:

Normalized Cross Correlation

OS:

Operating System

PC:

Personal Computer

PDA:

Personal Digital Assistant

png:

Portable Network Graphics

ppm:

Portable Pixel Map

PSNR:

Peak Signal-to-Noise Ratio

QoE:

Quality of Experience

RDP:

Remote Desktop Protocol

RFB:

Remote FrameBuffer

RIM:

Research in Motion

RTP:

Real-time Transport Protocol

RTSP:

Real-time Streaming Protocol

SC:

Structural Content

SMIL:

Synchronized Multimedia Integration Language

SVG:

Scalable Vector Graphics

SWF:

ShockWave Flash

TCP:

Transmission Control Protocol

UDP:

User Datagram Protocol

VDI:

Virtual Desktop Interface

VM:

Virtual Machine

VNC:

Virtual Network Computing

VRML:

Virtual Reality Modeling Language

Wi-Fi:

Wireless Fidelity

xHTML:

eXtensible HyperText Markup Language

XML:

eXtensible markup language

XMT:

eXtensible MPEG-4 textual

References

  1. Schlosser, D., Binzenhofer, A., Staehle, B.: Performance comparison of windows-based thin-client architectures. In: 2007 Australasian Telecommunication Networks and Applications Conference, Christchurch, New Zealand, 2–5 Dec 2007

  2. Simoens, P., Praet, P., Vankeirsbilck, B., De Wachter, J., Deboosere, L., De Turck, F., Dhoedt, B., Demeester, P.: Design and implementation of a hybrid remote display protocol to optimize multimedia experience on thin client devices. In: ATNAC 2008, 7–10 Dec 2008

  3. X Window System description. http://www.x.org

  4. VNC, Virtual Network Computing description. http://www.realvnc.com

  5. NX, NoMachine description. http://www.nomachine.com/

  6. RDP, Microsoft—Remote Desktop Protocol: basic connectivity and graphics remote specification. http://msdn.microsoft.com/en-us/library/cc240445

  7. Tan, K.-J., Gong, J.-W., Wu, B.-T., Chang, D.-C., Li1, H.-Y., Hsiao, Y.-M., Chen, Y.-C., Lo, S.-W., Chu, Y.-S., Guo, J.-I.: A remote thin client system for real time multimedia streaming over VNC. In: ICME 2010, Singapore, 19–23 July 2010

  8. Shen, H., Lu, Y., Wu, F., Li, S.: A high-performance remote computing platform. In: PerCom 2009, Galveston, Texas, 9–13 March 2009

  9. Yasser, P.-F., Kambiz, A.-H., Alnuweiri, H.: Internet delivery of MPEG-4 object-based multimedia. In: IEEE MultiMedia, April–June 2004

  10. Lethanhman, C., Isokawa, H., Kato, T.: Multipath data transmission for wireless thin clients. In: UBICOMM 2009, Sliema, Malta, 11–16 Oct 2009

  11. Bannan, M., Kellogg, D.: All about android, Nielsen (2011)

  12. Yan, L., Shipeng, L., Huifeng, S.: Virtualized screen: a third element for cloud mobile convergence. IEEE MultiMedia 18(2), 4–11 (2011)

    Article  Google Scholar 

  13. MPEG-4 BiFS standard specification, ISO/IEC JTC1/SC29/WG11 14496-11

  14. Asadi, M.K., Dufourd, J.-C: Context-aware semantic adaptation of multimedia presentations. In: IEEE International Conference on Multimedia and Expo, 6–8 July 2005, Amsterdam, Holland

  15. Izquierdo, E., Casas, J.R., Leonardi, R., Migliorati, P., O’Connor, N.E., Kompatsiaris, I., Strintzis, M.G.: Advanced content-based semantic scene analysis and information retrieval: the chema project. In: Workshop on Image Analysis for Multimedia Interactive Services, 9–11 April 2003, London, UK

  16. Carroll, A., Heiser, G.: An analysis of power consumption in a smartphone. In: USENIX 2010, 22–25 June 2010

  17. Microsoft® RemoteFX™. http://technet.microsoft.com/en-us/library/ff817578(WS.10).aspx

  18. Forecasts for smartphone operating system sales until 2015, Gartner. http://www.rossdawsonblog.com

  19. Battista, S., Casalino, F., Lande, C.: MPEG-4: a multimedia standard for the third millennium, part 1. IEEE MultiMedia 6, 74–83 (1999)

    Article  Google Scholar 

  20. Battista, S., Casalino, F., Lande, C.: MPEG-4: a multimedia standard for the third millennium, part 2. IEEE MultiMedia 7, 76–84 (2000)

    Article  Google Scholar 

  21. MPEG-4 LASeR standard specification. ISO/IEC°JTC1/SC29/WG11 14496-20

  22. Dufourd, J.-C., Avaro, O., Concolato, C.: An MPEG standard for rich media services. IEEE MultiMedia 12, 60–68 (2005)

    Article  Google Scholar 

  23. Adobe Flash. http://www.adobe.com/

  24. The Java Language Specification. http://java.sun.com/docs/books/jls/third_edition/html/j3TOC.html

  25. W3C, Scalable Vector Graphics Specification. http://www.w3.org/TR/2002/WD-SVG12-20021115

  26. W3C, Timed Text Markup Language Specification. http://www.w3.org/TR/ttaf1-dfxp/

  27. xHTML specification. http://www.w3.org/TR/xhtml-basic/

  28. GZIP file format specification version 4.3, rfc 1952. http://tools.ietf.org/html/rfc1952

  29. Liu, Z., Saifullah, Y., Greis, M., Sreemanthula, S.: HTTP compression technique. In: Proceedings of the 2005 IEEE Wireless Communications and Networking Conference, vol. 4, pp. 2495–2500, New Orleans, USA, 13–17 March 2005

  30. Song, J., Lee, B.-D.: Mobile rich media technologies: current status and future directions. KSII Trans Internet Inf Syst 5(2):359–373 (2011)

    Google Scholar 

  31. Ecma-262, EcmaScript documentation. http://www.ecmascript.org/docs.php

  32. Bruno, E.J.: Ajax: asynchronous JavaScript and XML. Dr. Dobb’s J. 31(2), 32–35 (2006)

    MathSciNet  Google Scholar 

  33. Mitrea, M., Simoens, P., Joveski, B., Marshall, I.J., Tanguengayte, A., Preteux, F., Dhoed, B.: BiFS based approaches to remote display for mobile thin clients. Proc. SPIE 7444, 74440F (2009). doi:10.1117/12.828152

    Article  Google Scholar 

  34. Joveski, B., Mitrea, M., Preteux, F.: MPEG-4 LASeR-based thin client remote viewer. In: EUVIP2010—European Workshop on Visual Information Processing, Paris, July 2010

  35. Joveski, B., Simoens, P., Gardenghi, L., Marshall, J., Mitrea, M., Vankeirsbilck, B., Prêteux, F., Dhoed, B.: Towards a multimedia remote viewer for mobile thin clients. Proc. SPIE 7881, 788102 (2011). doi:10.1117/12.876279

    Article  Google Scholar 

  36. GPAC—Open Source Multimedia Framework. http://gpac.sourceforge.net/index.php

  37. Concolato, C., Le Feuvre, J., Moissinac, J.-C.: Design of an efficient scalable vector graphics player for constrained devices. IEEE Trans. Consumer Electron. 54(2), 895–903 (2008)

    Article  Google Scholar 

  38. Live555 Streaming Media, source code libraries. http://www.live555.com

  39. UDP— User Datagram Protocol, RFC: 768. http://tools.ietf.org/html/rfc768

  40. TCP—Transmission Control Protocol specification, RFC: 793. http://www.ietf.org/rfc/rfc793.txt

  41. MPEG Media Transport (MMT). http://mpeg.chiariglione.org/working_documents/mpeg-h/mmt/mmt_co.zip

  42. MPEG Dynamic Adaptive Streaming over HTTP (DASH). http://mpeg.chiariglione.org/working_documents/mpeg-dash/MPEG-DASH-Tutorial.pdf

  43. Hypertext Transfer Protocol specification, RFC2616. http://www.http-compression.com/rfc2616.txt

  44. GPAC publication page. http://gpac.wp.institut-telecom.fr/about/references/

  45. Beg, M.S., Muslim, A., Chang, Y.C., Tang, T.F.: Performance evaluation of error resilient tools for MPEG-4 video transmission over a mobile channel. In: Conference on Personal Wireless Communications, 15–17 Dec 2002

  46. Galluccio, L.: Transmission of adaptive MPEG video over time-varying wireless channels: modeling and performance evaluation. IEEE Trans. Wirel. Commun. 4(6), 2777–2788 (2005)

    Article  Google Scholar 

  47. Basso, A., Kim, Y.-J., Jiang, Z.: Performance evaluation of MPEG-4 video over realistic EDGE wireless networks. In: The 5th International Symposium on Wireless Personal Multimedia Communications, vol. 3, pp. 1118–1122 (2002)

  48. gEdit official text editor for the GNOME desktop environment. http://projects.gnome.org/gedit/

  49. Epiphany web browser for the GNOME desktop environment. http://projects.gnome.org/epiphany/

  50. XRDP— Open source project for Remote Desktop Protocol for Linux version 0.4.2. http://sourceforge.net/projects/xrdp/files/xrdp/0.4.2/

  51. Joveski, B., Gardenghi, L., Mitrea, M., Prêteux, F.: Towards collaborative MPEG-4 BiFS mobile thin remote viewer. In: 15th IEEE International Symposium on Consumer Electronics, ISCE2011, Singapore, 14–17 June 2011

  52. Recommendation ITU-R BT.500: Methodology for the subjective assessment of the quality of television pictures, 09/2009, ITU-T Tutorial, Video Quality Experts Group (VQEG). Objective perceptual assessment of video quality: Full reference television (2004)

  53. ISO ISO/IEC JTC1/SC29/WG11 MPEG2010/N11275 Dresden, D.E., Baroncini, V., Ohm, J.-R., Sullivan, G.J.: Report of Subjective Test Results of Responses to the Joint Call for Proposals (CfP) on Video Coding Technology for High Efficiency Video Coding (HEVC) (2010)

  54. Rahmoune, A., Vandergheynst, P., Frossard, P.: Flexible motion-adaptive video coding with redundant expansions. IEEE Trans. Circuits Syst. Video Technol. 16(2), 178–190 (2006)

    Article  Google Scholar 

  55. Skodras, A., Christopoulos, C., Ebrahimi, T.: The JPEG 2000 still Image compression standard. IEEE Signal Process. Mag. 18(5), 36–58 (2001)

    Article  Google Scholar 

  56. Petrazzuoli, G., Cagnazzo, M., Pesquet-Popescu, B.: High order motion interpolation for side information improvement in DVC. In: IEEE International Conference on Acoustics, Speech, and Signal Processing, ICASSP, April 2010

  57. Shiang, H.-P., van der Schaar, M.: Multi-user video streaming over multi-hop wireless networks: a distributed, cross-layer approach based on priority queuing. IEEE J. Sel. Areas Commun. 25(4), 770–785 (2007)

    Article  Google Scholar 

  58. De Simone, F., Naccari, M., Tagliasacchi, M., Dufaux, F., Tubaro, S., Ebrahimi, T.: Subjective quality assessment of H.264/AVC video streaming with packet losses. EURASIP J. Image Video Process. 2011, 190431 (2011)

    Article  Google Scholar 

  59. Fry, T.C.: Probability and Its Engineering Use. D van Nostrand, Princeton (1965)

    Google Scholar 

  60. Walpole, R.E., Myers, R.H.: Probability and Statistics for Engineers and Scientists, 4th edn. MacMillan Publishing, New York (1989)

    Google Scholar 

  61. Baliga, J., Ayre, R.W.A., Hinton, K., Tucker, R.S.: Green cloud computing: balancing energy in processing, storage, and transport. Proc. IEEE 99(1), 149–167 (2011)

    Article  Google Scholar 

  62. Marshall, I.J., Mitrea, M., Joveski, B., Gardenghi L., Preteux, F.: Requirements on MPEG Scene Technology for Collaborative Applications, ISO/IEC JTC 1/SC 29/WG 11, No: N12206, Torino, Italy, 18–23 July 2011

  63. Marshall, I.J., Mitrea, M., Joveski, B., Gardenghi L., Preteux, F.: Context and Objectives for MPEG Scene Technology with Collaborative Applications, ISO/IEC JTC 1/SC 29/WG 11, No: N12207, Torino, Italy, 18–23 July 2011

  64. Marshall, I.J., Mitrea, M., Joveski, B., Preteux, F.: Use Cases for Collaborative Applications, ISO/IEC JTC 1/SC 29/WG 11, No: N12208, Torino, Italy, 18–23 July 2011

  65. Marshall, I.J., Mitrea, M., Joveski, B., Preteux, F.: Draft Call for Proposals for Scene Technologies for Collaborative Applications, ISO/IEC JTC 1/SC 29/WG 11, No: N12209, Torino, Italy, 18–23 July 2011

  66. HTML5 working draft by W3C. http://www.w3.org/TR/html5/

  67. Apple Remote Desktop. http://www.apple.com/remotedesktop/

  68. Cendio ThinLinc. http://www.cendio.com/products/thinlinc/

  69. Chicken. http://sourceforge.net/projects/cotvnc/

  70. ChunkVNC. http://www.chunkvnc.com/

  71. Crossloop. http://www.crossloop.com/

  72. EchoVNC. http://sourceforge.net/projects/echovnc/

  73. Ericom. http://www.ericom.com/

  74. Govern Remote Control. http://www.pjtec.com/

  75. NoMachine. http://www.nomachine.com/

  76. iTALC. http://italc.sourceforge.net/

  77. KRDC. http://www.kde.org/applications/internet/krdc/

  78. MAC HelpMate. http://www.machelpmate.com/

  79. N-central. http://www.n-able.com/products/n-central/

  80. noVNC. http://kanaka.github.com/noVNC/

  81. RealVNC. http://www.realvnc.com/

  82. RappidSupport. http://www.rapidsupport.net/pages/screenshots.php

  83. Remote Desktop Manager. http://remotedesktopmanager.com/

  84. TigerVNC. http://sourceforge.net/apps/mediawiki/tigervnc/index.php?title=Main_Page

  85. TightVNC. http://tightvnc.com/

  86. TurboVNC. http://www.virtualgl.org/Downloads/TurboVNC

  87. UltraVNC. http://www.uvnc.com/

  88. X11vnc. http://www.karlrunge.com/x11vnc/

  89. AnywhereTC. http://anywherets.com/

  90. Citrix XenApp. http://www.citrix.com/English/ps2/products/product.asp?contentID=186

  91. CoRD. http://cord.sourceforge.net/

  92. DualDesk. http://www.dualdesk.com/features/index.html

  93. FreeRDP. http://www.freerdp.com/

  94. RDesktop. http://www.rdesktop.org/

  95. Techinline. http://techinline.com/

  96. Xrdp. http://www.xrdp.org/

  97. XP/VS Server. http://www.thinstuff.com/faq/index.php?action=artikel&cat=9&id=97&artlang=en

  98. TeamViewer. http://www.teamviewer.com/en/index.aspx

  99. GoToMyPC. http://www.gotomypc.com/remote_access/remote_access

  100. PhoneMyPC. http://softwareforme.com/?page_id=6

  101. Virtual Desktop Interface by Oracle. http://www.oracle.com/us/technologies/virtualization/oraclevm/061153.html

Download references

Acknowledgments

This research was initiated under the framework of the FP7 MobiThin project. The work was partly founded by the ITEA2 SPY project.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to M. Mitrea.

Additional information

Communicated by M. Zink.

Appendices

Appendix 1: Thin client application panorama

In practice, the VNC and RDP technological supports are exploited by a large variety of ready-to-use applications. While an exhaustive list of such applications is practically impossible to be done and is also out of the scope of our paper, consider for instance:

  • VNC-based applications: Apple Remote Desktop [67], Cendio ThinLinc [68], Chicken [69], ChunkVNC [70], Crossloop [71], EchoVNC [72], Ericom [73], Goverlan Remote Control [74], NoMachine NX [75], iTALC [76], KRDC [77], Mac HelpMate [78], N-central [79], noVNC [80], RealVNC [81], RapidSupport [82], Remote Desktop Manager [83], TigerVNC [84], TightVNC [85], TurboVNC[86], UltraVNC [87], X11vnc [88];

  • RDP-based applications: AnywhereTS [89], Citrix XenApp [90], CoRD [91], DualDesk [92], Ericom [73], FreeRDP [93], NoMachine NX [75], KRDC [77], N-central [79], rdesktop [94], Remote Desktop Manager [83], Techinline [95], xrdp [96], XP/VS Server [97].

Moreover, proprietary (undisclosed) remote viewers’ technological support and applications are also available. For instance, TeamViewer [98] exploits NAT (Network Address Translation) for establishing a connection, based on the RFB (VNC) and RDP protocols. GoToMyPC [99] exploits the Citrix ICA (Independent Computing Architecture) proprietary protocol, but also supports VNC (RFB) and RDP. PhoneMyPC [100] exploits its proprietary protocol, without exposing any specification detail. Oracle and Sun Microsystems offer Virtual Desktop Infrastructure (VDI) [101] solution based on the proprietary Appliance Link Protocol (ALP). Unfortunately, all these solutions are not open for research (undisclosed specification) and development (no source code); consequently, they cannot be objectively benchmarked in our research study.

All these applications may provide additional levels of functionalities, like built-in encryption, file transfer, audio support, multiple sessions, seamless window, NAT pass-through, IpV6 support, video or 3D. The study, reported in the present paper, is placed at the technological support level; consequently, the actual application peculiarities will not be further discussed.

Appendix 2: Semantic content management

When trying to optimize content transmission from server toward mobile thin clients, three directions can be exploited: visual content re-usage, client memory control and network adaptability.

Regardless of its type (text editing, WWW browsing, etc.), each X application can periodically generate identical visual content. Such a case does not only occur when refreshing the screen but also when dealing with some fixed items (frequent letters/words typing, icons or menus displaying, etc.) or with repeated user actions (mouse over, open, save, etc.). Consequently, significant bandwidth reduction is a priori likely to be obtained by re-using that content directly on the client side, instead of resending it through the network. However, to take practical advantage of this concept, a tool for automatically detecting the repetition of the visual content and for its particular management should be provided.

The thin client memory is limited by its hardware resources. While content re-use would suppose, as a limit case, the caching of all the visual content previously generated in a session, the limited memory resources requires a mechanism for dynamically adjusting the cached information. Several implementation choices are available, from a fixed time window to more elaborated decisions based on actual frequency of usage or on the content semantic.

For mobile clients, the network connection is prone to dynamic changes (bandwidth variation, random errors, etc.). Moreover, the mobile user might also want to switch the terminal during a work session (e.g., from a smartphone to a tablet). Consequently, the encoding parameters should be real-time adapted to the client/network conditions. Unfortunately, the nowadays MPEG-4 scene representation technologies do not provide a direct solution to this issue. Moreover, the flexibility requirement set on the Compression & Transmission block forced us to map this functionality to the Scene-graph Manager level.

In the sequel, a functional solution jointly addressing the three above-mentioned aspects is illustrated for the particular case of image content by the flowchart in Fig. 12.

Fig. 12
figure 12

Flowchart for image management

The scene updating starts by detecting an image generated by the application and by reading some external network/client parameters. The image, its semantic information and the network/client parameters are combined so as to establish some encoding parameters (e.g., a low bandwidth and a low-resolution display would lead to a low quality factor for a JPEG compression).

Then, the existence of this image in the scene graph is checked. This task is achieved by computing the MD5 hash of the image and by searching it into a list containing the hashes of all images already used in the scene graph. According to the way in which this list is organized (from a simple hash record to more sophisticated relations between hash, its usage, its time stamp, etc.), different functionalities can be provided by this module.

In the case when the image already exists in the scene, a simple reference (pointer) to the corresponding image is created. Otherwise, the hash record list is updated and the new image with its encoding parameters is placed in a new node (or, in several nodes) in the scene graph.

As a side effect of this mechanism, the memory resources required by the client are increased. Hence, for thin clients, the image re-using should be restricted in time. In our implementation, we combined some temporal and spatial information about the cached images: assuming some images in the scene are not visible (i.e., they are covered by other visual elements) for more than τ seconds (in the experiments, τ = 180 s), they are removed from the scene and the hash record list is updated accordingly. Of course, different decision making rules can be implemented here: while directly impacting the system performances, they would not affect the architecture generality.

Finally, the BiFS/LASeR scene is updated so as to take into account of these changes: adding a new image/pointer to an image and remove some old images.

Rights and permissions

Reprints and Permissions

About this article

Cite this article

Joveski, B., Mitrea, M., Simoens, P. et al. Semantic multimedia remote display for mobile thin clients. Multimedia Systems 19, 455–474 (2013). https://doi.org/10.1007/s00530-013-0304-6

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00530-013-0304-6

Keywords

  • Semantic multimedia remote display
  • Mobile thin client
  • MPEG-4 multimedia scene (BiFS, LASeR)
  • X Window System
  • VNC-HEXTILE
  • RDP