International Journal of Information Security

, Volume 13, Issue 2, pp 171–189 | Cite as

On detecting co-resident cloud instances using network flow watermarking techniques

  • Adam Bates
  • Benjamin Mood
  • Joe Pletcher
  • Hannah Pruse
  • Masoud Valafar
  • Kevin Butler


Virtualization is the cornerstone of the developing third-party compute industry, allowing cloud providers to instantiate multiple virtual machines (VMs) on a single set of physical resources. Customers utilize cloud resources alongside unknown and untrusted parties, creating the co-resident threat—unless perfect isolation is provided by the virtual hypervisor, there exists the possibility for unauthorized access to sensitive customer information through the exploitation of covert side channels. This paper presents co-resident watermarking, a traffic analysis attack that allows a malicious co-resident VM to inject a watermark signature into the network flow of a target instance. This watermark can be used to exfiltrate and broadcast co-residency data from the physical machine, compromising isolation without reliance on internal side channels. As a result, our approach is difficult to defend against without costly underutilization of the physical machine. We evaluate co-resident watermarkingunder a large variety of conditions, system loads and hardware configurations, from a local laboratory environment to production cloud environments (Futuregrid and the University of Oregon’s ACISS). We demonstrate the ability to initiate a covert channel of 4 bits per second, and we can confirm co-residency with a target VM instance in \(<\)10 s. We also show that passive load measurement of the target and subsequent behavior profiling is possible with this attack. We go on to consider the detectability of co-resident watermarking, extending our scheme to create a subtler watermarking attack by imitating legitimate cloud customer behavior. Our investigation demonstrates the need for the careful design of hardware to be used in the cloud.


Cloud security Traffic analysis  Covert channel 



We would like to thank Allen D. Malony, Chris Hoge, and the ACISS staff for their assistance and support. Through our use of Futuregrid, this material is based upon work supported in part by the National Science Foundation Under Grant No. 0910812 to Indiana University for “FutureGrid: An Experimental, High-Performance Grid Test-bed.” and Grant CNS-1118046.


  1. 1.
    Amazon EC2 Service Level Agreement.
  2. 2.
    Amazon. Amazon Elastic Compute Cloud (EC2).
  3. 3.
    Armbrust, M., Fox, A., Griffith, R., Joseph, A., Katz, R., et al.: Above the Clouds: A Berkeley View of Cloud Computing. Technical Report UCB/EECS-2009-28, University of California, Berkeley (2009)Google Scholar
  4. 4.
    Barham, P., Dragovic, B., Fraser, K., Hand, S., Harris, T., Ho, A., Neugebauer, R., Pratt, I., Warfield A.: Xen and the art of virtualization. In: Proceedings of 19th ACM Symposium on Operating Systems Principles, SOSP ’03, New York, pp. 164–177. ACM (2003)Google Scholar
  5. 5.
    Barker, S., Shenoy P.: Empirical evaluation of latency-sensitive application performance in the cloud In: Proceedings of 1st SCM SIGMM Conference on Multimedia Systems, MMSys ’10, New York, pp. 34–46. ACM (2010)Google Scholar
  6. 6.
    Bernstein D.J.: Cache-timing attacks on AES. Compute (2005)Google Scholar
  7. 7.
    Blum, A., Song, D., Venkataraman, S.: Detection of interactive stepping stones: algorithms and confidence bounds. In: Proceedings of Recent Advances in Intrusion Detection (RAID) (2004)Google Scholar
  8. 8.
    Bowers, K.D., van Dijk, M., Juels, A., Oprea, A., Rivest R.L.: How to tell if your cloud files are vulnerable to drive crashes. In: CCS ’11: Proceedings of 18th ACM Conference on Computer and Communications Security, Chicago, pp. 501–514 (2011)Google Scholar
  9. 9.
    Brodkin J.: VMware confirms source code leak, LulzSec -affiliated hacker claims credit.
  10. 10.
    Butt, S., Lagar-Cavilla, H.A., Srivastava, A., Ganapathy V.: Self-service cloud computing. In: Proceedings of 2012 ACM Conference on Computer and Communications Security, Raleigh (2012)Google Scholar
  11. 11.
    Cabuk, S., Brodley, C.E., Shields C.: Ip covert timing channels: design and detection. In: Proceedings of 11th ACM Conference on Computer and Communications Security, CCS ’04, New York, pp. 178–187. ACM (2004)Google Scholar
  12. 12.
    Cabuk, S., Brodley, C.E., Shields C.: IP Covert Channel Detection. ACM Trans. Inf. Syst. Secur. (TISSEC) 12(4): 1–29 (2009)Google Scholar
  13. 13.
    Chinni, S., Hiremane, R.: Virtual machine device queues. White paper, Intel Corporation (2007)Google Scholar
  14. 14.
    Coskun, B., Memon, N.: Online sketching of network flows for real-time stepping-stone detection. In: Proceedings of 2009 Annual Computer Security Applications Conference, ACSAC ’09, Washington, pp. 473–483. IEEE Computer Society (2009)Google Scholar
  15. 15.
    CVE-2007-4993. pygrub (tools/pygrub/src/ in xen 3.0.3.
  16. 16.
    CVE-2007-5497. Multiple integer overflows in libext2fs.
  17. 17.
    CVE-2010-2240. The do\_anonymous\_page function in mm/ memory.c.
  18. 18.
    Dong, Y., Yu, Z., Rose, G.: SR-IOV networking in Xen: architecture, design and implementation. In: Proceedings of First Conference on I/O Virtualization, WIOV’08, Berkeley, p. 10. USENIX Association (2008)Google Scholar
  19. 19.
    Gamage, S., Kangarlou, A., Kompella, R.R., Xu, D.: Opportunistic flooding to improve TCP transmit performance in virtualized clouds. In: Proceedings of 2nd ACM Symposium on Cloud Computing, SOCC ’11, New York, pp. 1–14. ACM (2011)Google Scholar
  20. 20.
    Gianvecchio, S., Wang, H.: Detecting covert timing channels: an entropy-based approach. In: Proceedings of 14th ACM Conference on Computer and Communications Security (CCS’07), Alexandria (2007)Google Scholar
  21. 21.
    Gupta, D., Cherkasova, L., Gardner, R., Vahdat, A.: Enforcing performance isolation across virtual machines in Xen. In: Middleware (2006)Google Scholar
  22. 22.
    Habib, I.: Virtualization with KVM. Linux J. 166: 8(2008)Google Scholar
  23. 23.
    Houmansadr, A., Borisov, N.: SWIRL: a scalable watermark to detect correlated network flows. In: Proceedings of 18th ISOC Symposium on Network and Distributed Systems Security (NDSS ’11), San Diego (2011)Google Scholar
  24. 24.
    Houmansadr, A., Kiyavash, N., Borisov, N.: RAINBOW: a robust and invisible non-blind watermark for network flows. In: Proceedings of 16th Network and Distributed System Security Symposium (NDSS’09) (2009)Google Scholar
  25. 25.
    Keller, E., Szefer, J., Rexford, J., Lee, R.B.: Eliminating the hypervisor attack surface for a more secure cloud. In: Proceedings of ACM Conference on Computer and Communications, Security (CCS’11) (2011)Google Scholar
  26. 26.
    Keramidas, G., Antonopoulos, A., Serpanos, D., Kaxiras, S.: Non deterministic caches: a simple and effective defense against side channel attacks. Design Autom. Embed. Syst. 12: 221–230 (2008)Google Scholar
  27. 27.
    Kiyavash, N., Houmansadr, A., Borisov, N.: Multi-flow attacks against network flow watermarking schemes. In: Proceedings of 17th USENIX Security Symposium, San Jose (2008)Google Scholar
  28. 28.
    Kocher, P.C.: Timing attacks on implementations of Diffie–Hellman, RSA, DSS, and other systems. In: CRYPTO, pp. 104–113 (1996)Google Scholar
  29. 29.
    Kocher, P.C., Jaffe, J., Jun, B.: Differential power analysis. In: CRYPTO, pp. 388–397 (1999)Google Scholar
  30. 30.
    Kutch, P.: PCI-SIG SR-IOV Primer. Technical report, Intel Corporation (2011)Google Scholar
  31. 31.
    Law, A.M., Kelton, D.W.: Simulation Modeling and Analysis. McGraw-Hill, Boston (2000)Google Scholar
  32. 32.
    Luo, X., Chan, E., Chang, R.: Cloak: A ten-fold way for reliable covert communications. In: Proceedings of European Symposium on Research in Computer Security ESORICS (2007)Google Scholar
  33. 33.
    Luo, X., Zhang, J., Perdisci, R., Lee, W.: On the secrecy of spread-spectrum flow watermarks. In: Proceedings of European Symposium on Research in Computer Security ESORICS (2010)Google Scholar
  34. 34.
    Luo, X., Zhou, P., Zhang, J., Perdisci, R., Lee, W., Chang, R.K.C.: Exposing invisible timing-based traffic watermarks with BACKLIT. In: Proceedings of 27th Annual Computer Security Applications Conference, ACSAC ’11, Orlando (2011)Google Scholar
  35. 35.
    McCune, J.M., Li, Y., Qu, N., Zhou, Z., Datta, A., Gligor, V., Perrig, A.: TrustVisor: efficient TCB reduction and attestation. In: Proceedings of 2010 IEEE Symposium on Security and Privacy, Oakland (2010)Google Scholar
  36. 36.
    Murdoch, S., Danezis, G.: Low-cost traffic analysis of Tor. In: Proceedings of 2005 IEEE Symposium on Security and Privacy. Oakland (2005)Google Scholar
  37. 37.
    Okamura, K., Oyama, Y.: Load-based covert channels between Xen virtual machines. In: Proceedings of 2010 ACM Symposium on Applied Computing, SAC ’10, Sierre (2010)Google Scholar
  38. 38.
    Peng, P., Ning, P., Reeves, D.S.: On the secrecy of timing-based active watermarking trace-back techniques. In: Proceedings of 2006 IEEE Symposium on Security and Privacy, Oakland (2006)Google Scholar
  39. 39.
    Percival, C.: Cache missing for fun and profit. In: BSDCan (2005)Google Scholar
  40. 40.
    Pettitt, A.N., Stephens, M.A.: The Kolmogorov–Smirnov goodness-of-fit statistic with discrete and grouped data. Technometrics 19(2), 205–210 (1977)CrossRefzbMATHGoogle Scholar
  41. 41.
    Raj, H., Nathuji, R., Singh, A., England, P.: Resource management for isolation enhanced cloud services. In: Proceedings of 2009 ACM Workshop on Cloud Computing Security, CCSW ’09, Chicago (2009)Google Scholar
  42. 42.
    Ram, K.K., Santos, J.R., Turner, Y., Cox, A.L., Cox, A.L., Rixner, S.: Achieving 10 GB/s using Xen para-virtualized network drivers. Xen Summit (2009)Google Scholar
  43. 43.
    Ristenpart, T., Tromer, E., Shacham, H., Savage, S.: Hey, You, Get off of my cloud: exploring information leakage in third-party compute clouds. In: CCS’09: Proceedings of 16th ACM Conference on Computer and Communications Security, Chicago (2009)Google Scholar
  44. 44.
    Schad, J., Dittrich, J., Quiané-Ruiz, J.-A.: Runtime measurements in the cloud: observing, analyzing, and reducing variance. Proc. VLDB Endow. 3(1–2), 460–471 (2010)Google Scholar
  45. 45.
    Seshadri, A., Luk, M., Qu, N., Perrig, A.: SecVisor: a tiny hypervisor to provide lifetime kernel code integrity for commodity OSes. In: SOSP’07: Proceedings of 21st ACM Symposium on Operating Systems Principles, Stevenson (2007)Google Scholar
  46. 46.
    Singh, A., Korupolu, Aameek M., Mohapatra, D.: Server-storage virtualization: integration and load balancing in data centers. In: Proceedings of 2008 ACM/IEEE Conference on Supercomputing, Austin (2008)Google Scholar
  47. 47.
    Stevens, W.R.: TCP/IP Illustrated: The Protocols, vol. 1. Addison-Wesley Longman Publishing Co. Inc., Boston (1993)zbMATHGoogle Scholar
  48. 48.
    Varadarajan, V., Kooburat, T., Farley, B., Ristenpart, T., Swift, M.M.: Resource-freeing attacks: improve your cloud performance (at Your Neighbor’s Expense). In: Proceedings of 2012 ACM Conference on Computer and Communications Security, Raleigh (2012)Google Scholar
  49. 49.
    VMSA-2008-0008. Updates to VMware workstation, VMware player, VMware ACE, VMware fusion resolve critical security issues.
  50. 50.
    Wang, X., Chen, S., Jajodia, S.: Network flow watermarking attack on low-latency anonymous communication systems. In: Proceedings of 2007 IEEE Symposium on Security and Privacy, Oakland (2007)Google Scholar
  51. 51.
    Wang, X., Reeves, D.S.: Robust correlation of encrypted attack traffic through stepping stones by manipulation of interpacket delays. In: Proceedings of 10th ACM Conference on Computer and Communications Security, CCS ’03, New York, pp. 20–29. ACM (2003)Google Scholar
  52. 52.
    Whiteaker, J., Schneider, F., Teixeira, R.: Explaining packet delays under virtualization. SIGCOMM Comput. Commun. Rev. 41: 38–44 (2011)Google Scholar
  53. 53.
    Wood, T., Shenoy, P., Venkataramani, A., Yousif, M.: Black-box and gray-box strategies for virtual machine migration In: Proceedings of 4th USENIX Conference on Networked Systems Design and Implementation, Cambridge (2007)Google Scholar
  54. 54.
    Xu, Y., Bailey, M., Jahanian, F., Joshi, K., Hiltunen, M., Schlichting R.: An exploration of L2 cache covert channels in virtualized environments. In: Proceedings of 3rd ACM Workshop on Cloud Computing, Security (CCSW’11) (2011) Google Scholar
  55. 55.
    Yao, Y.: Network speed test (IPerf) in KVM (Virtio-net, emulated, vt-d). (2004)
  56. 56.
    Yu, W., Fu, X., Graham, S., Xuan, D., Zhao, W.: DSSS-based flow marking technique for invisible traceback. In: Proceedings of 2007 IEEE Symposium on Security and Privacy (2007)Google Scholar
  57. 57.
    Zhang, Y., Juels, A., Oprea, A., Reiter, M.K.: HomeAlone: Co-Residency Detection in the Cloud via Side-Channel Analysis. In: Proceedings of 2011 IEEE Symposium on Security and Privacy, Berkeley (2011)Google Scholar
  58. 58.
    Zhang, Y., Juels, A., Reiter, M.K., Reiter, M., Ristenpart, T.: Cross-VM side channels and their use to extract private keys. In: Proceedings of 2012 ACM Conference on Computer and Communications Security, Raleigh (2012)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2013

Authors and Affiliations

  • Adam Bates
    • 1
  • Benjamin Mood
    • 1
  • Joe Pletcher
    • 1
  • Hannah Pruse
    • 1
  • Masoud Valafar
    • 1
  • Kevin Butler
    • 1
  1. 1.Department of Computer and Information ScienceUniversity of OregonEugeneUSA

Personalised recommendations