Computational Economics

, Volume 33, Issue 2, pp 131–154 | Cite as

Measuring the Efficiency of the Intraday Forex Market with a Universal Data Compression Algorithm

  • Armin Shmilovici
  • Yoav Kahiri
  • Irad Ben-Gal
  • Shmuel Hauser


Universal compression algorithms can detect recurring patterns in any type of temporal data—including financial data—for the purpose of compression. The universal algorithms actually find a model of the data that can be used for either compression or prediction. We present a universal Variable Order Markov (VOM) model and use it to test the weak form of the Efficient Market Hypothesis (EMH). The EMH is tested for 12 pairs of international intra-day currency exchange rates for one year series of 1, 5, 10, 15, 20, 25 and 30 min. Statistically significant compression is detected in all the time-series and the high frequency series are also predictable above random. However, the predictability of the model is not sufficient to generate a profitable trading strategy, thus, Forex market turns out to be efficient, at least most of the time.


Efficient Market Hypothesis Universal prediction Forex Intra-day trading Variable Order Markov 

JEL Classification

G14 C22 C53 C49 C63 

Mathematics Subject Classification (2000)

62P05 91B84 62M20 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Alon-Brimer, Y. (2002). Measuring the efficiency of the Israeli stock market by the context tree model. M.Sc. dissertation (in Hebrew), Ben-Gurion University, Israel.Google Scholar
  2. Annette, M. G. (1997). Kappa statistics for multiple raters using categorical classifications. In Proceedings of the Twenty-Second Annual SAS Users Group International Conference (available online), San Diego, CA, March.Google Scholar
  3. Baetaens D.J.E., van den Berg W.M., Vaudrey H. (1996) Market inefficiencies, technical trading and neural networks. In: Dunis C. (eds) Forecasting financial markets, financial economics and quantitative analysis. Wiley, Chichester, pp 245–260Google Scholar
  4. Bahmani-Oskooee, M., Kutan, A. M., & Zhou, S. (2006). Do real exchange rates follow a non-linear mean reverting process in developing countries. EMG Working Paper Series, WP-EMG-02-2006.Google Scholar
  5. Begleiter R., El-Yaniv R., Yona G. (2004) On prediction using variable order Markov models. Journal of Artificial Intelligence 22: 385–421Google Scholar
  6. Bejerano G., Yona G. (2001) Variations on probabilistic suffix trees: Statistical modeling and prediction of protein families. Bioinformatics 17: 23–43 doi:10.1093/bioinformatics/17.1.23 CrossRefGoogle Scholar
  7. Bellgard, C. D. (2002). Diffusion of forecasting technology: Nonlinearity and the information advantage effect. Ph.D. dissertation, The University of Western Australia.Google Scholar
  8. Ben-Gal I., Morag G., Shmilovici A. (2003) CSPC: A monitoring procedure for state dependent processes. Technometrics 45(4): 293–311 doi:10.1198/004017003000000122 CrossRefGoogle Scholar
  9. Ben-Gal I., Shani A., Gohr A., Grau J., Arbiv S., Shmilovici A. et al (2005) Identification of transcription factor binding sites with variable-order Bayesian networks. Bioinformatics 21(11): 2657–2666 doi:10.1093/bioinformatics/bti410 CrossRefGoogle Scholar
  10. Boero G., Marrocu E. (2002) The performance of non linear exchange rate models: A forecasting comparison. Journal of Forecasting, 21: 513–542 doi:10.1002/for.837 CrossRefGoogle Scholar
  11. Buhlmann P., Wyner A.J. (1999) Variable length Markov chains. Annals of Statistics 27(2): 480–513 doi:10.1214/aos/1018031204 CrossRefGoogle Scholar
  12. Chen S.-H., Tan C.-W. (1996) Measuring randomness by Rissanen’s stochastic complexity: Applications to the financial data. In: Dowe D.L., Korb K.B., Oliver J.J. (eds) Information, statistics and induction in science. Singapore, World ScientificGoogle Scholar
  13. Chen, S.-H., & Tan, C.-W. (1999). Estimating the complexity function of financial time series: An estimation based on predictive stochastic complexity. In Proceedings of the 5th International Conference of the Society for Computational Economics, Boston, June 24–26. Also published in Journal of Management and Economics, 3(3) (an electronic journal).Google Scholar
  14. Choong, C. K., Poon, W. C., Habibullah, M. S., & Yusop, Z. (2003). The validity of PPP theory in asean five: Another look on cointegration and panel data analysis. Faculty of Accountancy and Management, University Tunku Abdul Rahman.Google Scholar
  15. Chung J., Hong Y. (2007) Model-free evaluation of directional predictability in foreign exchange markets. Journal of Applied Econometrics 22: 855–889 doi:10.1002/jae.965 CrossRefGoogle Scholar
  16. Cover T.M. (1991) Universal portfolios. Mathematical Finance 1(1): 1–29 doi:10.1111/j.1467-9965.1991.tb00002.x CrossRefGoogle Scholar
  17. Cover T.M., Thomas J.A. (1991) Elements of information theory. Wiley, New-YorkGoogle Scholar
  18. Dacorogna M., Gencay R., Muller U., Olsen R., Pictet O. (2001) An introduction to high-frequency finance. Academic Press, New YorkGoogle Scholar
  19. Deboeck G. (1994) Trading on the edge: Neural, genetic, and fuzzy systems for chaotic financial markets. Wiley, New YorkGoogle Scholar
  20. Dempster M.A.H., Payne T.W., Romahi Y., Thompson G.W.P. (2001) Computational learning techniques for intraday FX trading using popular technical indicators. IEEE Transactions on Neural Networks 12(4): 744–754 doi:10.1109/72.935088 CrossRefGoogle Scholar
  21. Fama E.F. (1991) Efficient capital markets: II. The Journal of Finance 46: 1575–1611 doi:10.2307/2328565 CrossRefGoogle Scholar
  22. Fama E.F. (1998) Market efficiency, long term returns, and behavioral finance. Journal of Financial Economics, 49: 283–306 doi:10.1016/S0304-405X(98)00026-9 CrossRefGoogle Scholar
  23. Feder, M., & Federovski, E. (1999), Universal prediction with finite memory. IT Workshop on Detection, Estimation, Classification and Imaging, Santa Fe, NM, USA, 24–26 February.Google Scholar
  24. Feder M., Merhav N. (1994) Relations between entropy and error probability. IEEE Transactions on Information Theory 38(1): 259–266 doi:10.1109/18.272494 CrossRefGoogle Scholar
  25. Feder M., Merhav N., Gutman M. (1992) Universal prediction of individual sequences. IEEE Transactions on Information Theory 38(4): 1258–1270 doi:10.1109/18.144706 CrossRefGoogle Scholar
  26. Federovsky E., Feder M., Weiss S. (1998) Branch prediction based on universal data compression algorithm. ACM SIGARCH Computer Architecture News 26(3): 62–72CrossRefGoogle Scholar
  27. Goldschmidt, P., & Bellgrad, C. (1999). Forecasting foreign exchange rates: Random walk hypothesis, linearity and data freqency. Department of Information Management and Marketing, The University of Western Australia.Google Scholar
  28. Hellstrom, T., & Holmstrom, K. (1998). Predicting the stock market. Technical report Ima-TOM-1997-07, Umea University, Sweden.
  29. Hutter, M. (2001). Convergence and error bounds for universal prediction of nonbinary sequences. In: Proceedings of the 12th European Conference on Machine Learning ECML-2001 (pp. 239–250).Google Scholar
  30. Jensen M. (1978) Some anomalous evidence regarding market efficiency. Journal of Financial Economics, 6: 95–101 doi:10.1016/0304-405X(78)90025-9 CrossRefGoogle Scholar
  31. Kaashoek J.F., Van Dijk H.K. (2002) Neural network pruning applied to real exchange rate analysis. Journal of Forecasting 21: 559–577 doi:10.1002/for.835 CrossRefGoogle Scholar
  32. Kahiri, Y. (2004). Measuring the efficiency of the Forex market via the context tree model. M.Sc. dissertation (in Hebrew), Ben-Gurion University, Israel.Google Scholar
  33. Kamruzzaman J., Sarker R.A. (2004) ANN based forecasting of foreign currency exchange rates. Neural Information Processing—Letters and Reviews 3(2): 49–58Google Scholar
  34. Lebaron B. (1999) Technical trading rule profitability and foreign exchange intervention. Journal of International Economics 49: 125–143CrossRefGoogle Scholar
  35. Lo A.W. (2007) Efficient markets hypothesis. In: Blume L., Durlauf S. (eds) The new palgrave: A dictionary of economics (2nd ed). Palgrave MacMillan, LondonGoogle Scholar
  36. Malkiel B.G. (2003) The efficient market hypothesis and its critics. The Journal of Economic Perspectives 17: 59–82 doi:10.1257/089533003321164958 CrossRefGoogle Scholar
  37. Merhav N., Feder M. (1998) Universal prediction. IEEE Transactions on Information Theory IT- 44: 2124–2147 doi:10.1109/18.720534 CrossRefGoogle Scholar
  38. Millman G.J. (1995) Around the world on a trillion dollars a day. Bantam Press, New YorkGoogle Scholar
  39. Mills, T. (eds) (2002) Forecasting financial markets. Edward Elgar, UK CheltenhamGoogle Scholar
  40. Neely, C. J. (1997). Technical analysis in the foreign exchange market: A Layman’s guide. Federal Reserve Bank of St. Louis.Google Scholar
  41. Orlov Y.L., Filippov V.P., Potapov V.N., Kolchanov N.A. (2002) Construction of stochastic context trees for genetic texts. In Silico Biology 2(3): 233–247Google Scholar
  42. Papageorgiou, C. P. (1997). High frequency time series analysis and prediction using Markov models. In Proceedings of the conference on Computational Intelligence for Financial Engineering IEEE/IAFE (pp. 182–188), 23–25 March, New York City, NY.Google Scholar
  43. Pring M.J. (1991) Technical analysis explained: The successful investor’s guide to spotting investment trends and turning points. McGraw-Hill, New-YorkGoogle Scholar
  44. Rissanen J. (1983) A universal data compression system. IEEE Transactions on Information Theory, 29(5): 656–664 doi:10.1109/TIT.1983.1056741 CrossRefGoogle Scholar
  45. Rissanen J. (1984) Universal coding, information, prediction, and estimation. IEEE Transactions on Information Theory 30(4): 629–636 doi:10.1109/TIT.1984.1056936 CrossRefGoogle Scholar
  46. Schwert G. (2003) Anomalies and market efficiency (Chap. 15). In: Constantinides G.M., Harris M., Stulz R. (eds) Handbook of economics of finance. Elsevier, Amsterdam, pp 939–960Google Scholar
  47. Shmilovici A., Alon-Brimer Y., Hauser S. (2003) Using a stochastic complexity measure to check the efficient market hypothesis. Computational Economics 22(3): 273–284 doi:10.1023/A:1026198216929 CrossRefGoogle Scholar
  48. Shmilovici, A., & Ben-Gal, I. (2006). Assessing the efficient market hypothesis in stock exchange markets via a universal prediction statistic. Unpublished technical report.Google Scholar
  49. Shmilovici A., Ben-Gal I. (2007) Using a VOM model for reconstructing potential coding regions in EST sequences. Computational Statistics 22(1): 49–69 doi:10.1007/s00180-007-0021-8 Google Scholar
  50. Sim J., Wright C.C. (2005) The kappa statistic in reliability studies: Use, interpretation, and sample size requirements. Physical Therapy 85: 257–268Google Scholar
  51. Sullivan R., Timmerman A., White H. (1999) Data-snooping, technical trading rule performance, and the bootstrap. The Journal of Finance 54: 1647–1692 doi:10.1111/0022-1082.00163 CrossRefGoogle Scholar
  52. Taylor M.P., Allen H. (1992) The use of technical analysis in the foreign exchange market. Journal of International Money and Finance 11: 304–314 doi:10.1016/0261-5606(92)90048-3 CrossRefGoogle Scholar
  53. Taylor A.M., Taylor M.P. (2004) The purchasing power parity debate. The Journal of Economic Perspectives 18: 135–158 doi:10.1257/0895330042632744 Google Scholar
  54. Timmermann A., Granger C.W.J. (2004) Efficient market hypothesis and forecasting. International Journal of Forecasting 20(1): 15–27 doi:10.1016/S0169-2070(03)00012-8 CrossRefGoogle Scholar
  55. Tino, P., Schittenkopf, C., & Dorffner, G. (2000). A symbolics dynamics approach to volatility prediction. In Y. S. Abu-Mustafa & B. LeBaron (Eds.), Proceedings of the 6th International Conference on Computational Finance (pp. 137–151), January 6–8 1999. Cambridge, MA: Leonard N. Stern School of Bussiness, MIT Press.Google Scholar
  56. Tsay R. (2002) Analysis of financial time series. Wiley, New YorkGoogle Scholar
  57. Vert J.P. (2001) Adaptive context trees and text clustering. IEEE Transactions on Information Theory 47(5): 1884–1901 doi:10.1109/18.930925 CrossRefGoogle Scholar
  58. Wu, W., & Shafer, G. (2007). Testing lead-Lag effects under game-theoretic efficient market hypotheses.
  59. Yu L., Wang S., Lai K.K. et al (2005) Adaptive smoothing neural networks in foreign exchange rate forecasting. In: Sunderametal V.S. (eds) ICCS, LNCS 3516. Springer, New York, pp 523–530Google Scholar
  60. Zaidenraise K.O.S., Shmilovici A., Ben-Gal I. (2007) Gene-finding with the VOM model. Journal of Computational Methods in Science and Engineering 7(1): 45–54Google Scholar
  61. Zhang X. (1994) Non-linear predictive models for intra-day foreign exchange trading. Intelligent Systems in Accounting, Finance and Management 3: 293–302Google Scholar
  62. Ziv J. (2001) A universal prediction lemma and applications to universal data compression and prediction. IEEE Transactions on Information Theory 47(4): 1528–1532 doi:10.1109/18.923732 CrossRefGoogle Scholar
  63. Ziv, J. (2002). An efficient universal prediction algorithm for unknown sources with limited training data.
  64. Ziv J., Lempel A. (1978) Compression of individual sequences via variable rate coding. IEEE Transactions on Information Theory 24: 530–536 doi:10.1109/TIT.1978.1055934 CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC. 2008

Authors and Affiliations

  • Armin Shmilovici
    • 1
  • Yoav Kahiri
    • 2
  • Irad Ben-Gal
    • 3
  • Shmuel Hauser
    • 4
  1. 1.Department of Information SystemsBen-Gurion UniversityBeer-ShevaIsrael
  2. 2.School of ManagementBen-Gurion UniversityBeer-ShevaIsrael
  3. 3.Department of Industrial EngineeringTel-Aviv UniversityRamat-Aviv, Tel-AvivIsrael
  4. 4.ONO Academic College and School of ManagementBen-Gurion UniversityBeer-ShevaIsrael

Personalised recommendations