Annals of Operations Research

, Volume 177, Issue 1, pp 151–183 | Cite as

Online stochastic optimization under time constraints

  • Pascal Van Hentenryck
  • Russell Bent
  • Eli Upfal


This paper considers online stochastic combinatorial optimization problems where uncertainties, i.e., which requests come and when, are characterized by distributions that can be sampled and where time constraints severely limit the number of offline optimizations which can be performed at decision time and/or in between decisions. It proposes online stochastic algorithms that combine the frameworks of online and stochastic optimization. Online stochastic algorithms differ from traditional a priori methods such as stochastic programming and Markov Decision Processes by focusing on the instance data that is revealed over time. The paper proposes three main algorithms: expectation E, consensus C, and regret R. They all make online decisions by approximating, for each decision, the solution to a multi-stage stochastic program using an exterior sampling method and a polynomial number of samples. The algorithms were evaluated experimentally and theoretically. The experimental results were obtained on three applications of different nature: packet scheduling, multiple vehicle routing with time windows, and multiple vehicle dispatching. The theoretical results show that, under assumptions which seem to hold on these, and other, applications, algorithm E has an expected constant loss compared to the offline optimal solution. Algorithm R reduces the number of optimizations by a factor |R|, where R is the number of requests, and has an expected ρ(1+o(1)) loss when the regret gives a ρ-approximation to the offline problem.


Stochastic optimization Online algorithms Dynamic vehicle routing 


Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.


  1. Benoist, T., Bourreau, E., Caseau, Y., & Rottembourg, B. (2001). Towards stochastic constraint programming: A study of online multi-choice knapsack with deadlines. In Proceedings of the seventh international conference on principles and practice of constraint programming (CP’01), London, UK (pp. 61–76). Berlin: Springer. Google Scholar
  2. Bent, R., & Van Hentenryck, P. (2004a). A two-stage hybrid local search for the vehicle routing problem with time windows. Transportation Science, 8(4), 515–530. CrossRefGoogle Scholar
  3. Bent, R., & Van Hentenryck, P. (2004b). Online stochastic and robust optimization. In Proceeding of the 9th Asian computing science conference (ASIAN’04), Chiang Mai University, Thailand, December 2004. Google Scholar
  4. Bent, R., & Van Hentenryck, P. (2004c). Regrets only. Online stochastic optimization under time constraints. In Proceedings of the 19th national conference on artificial intelligence (AAAI’04), San Jose, CA, July 2004. Google Scholar
  5. Bent, R., & Van Hentenryck, P. (2004d). Scenario based planning for partially dynamic vehicle routing problems with stochastic customers. Operations Research, 52(6). Google Scholar
  6. Bent, R., & Van Hentenryck, P. (2004e). The value of consensus in online stochastic scheduling. In Proceedings of the 14th international conference on automated planning & scheduling (ICAPS 2004), Whistler, British Columbia, Canada, 2004. Google Scholar
  7. Bent, R., & Van Hentenryck, P. (2005). Online stochastic optimization without distributions. In Proceedings of the 15th international conference on automated planning & scheduling (ICAPS 2005), Monterey, CA, 2005. Google Scholar
  8. Bent, R., Katriel, I., & Van Hentenryck, P. (2005). Sub-optimality approximation. In Eleventh international conference on principles and practice of constraint programming, Stiges, Spain, 2005. Google Scholar
  9. Bertsekas, D., & Tsitsiklis, J. (1996). Neuro-dynamic programming. Belmont: Athena Scientific. Google Scholar
  10. Birge, J. R., & Louveaux, F. (1997). Introduction to stochastic programming. New York: Springer. Google Scholar
  11. Blair, C. E., & Jeroslow, R. G. (1979). On the value function of an integer program. Mathematical Programming, 23. Google Scholar
  12. Borodin, A., & El-Yaniv, R. (1998). Online computation and competitive analysis. Cambridge: Cambridge University Press. Google Scholar
  13. Borodin, A., Kleinberg, J., Raghawan, P., Sudan, M., & Williamson, D. (2001). Adversarial queuing theory. Journal of the Association for Computing Machinery, 48(1), 13–38. Google Scholar
  14. Campbell, A., & Savelsbergh, M. (2002). Decision support for consumer direct grocery initiatives (Report TLI-02-09). Georgia Institute of Technology. Google Scholar
  15. Chang, H., Givan, R., & Chong, E. (2000). On-line scheduling via sampling. In Artificial intelligence planning and scheduling (AIPS’00) (pp. 62–71). Google Scholar
  16. Choi, J., Realff, M., & Lee, J. (2004). Dynamic programming in a heuristically confined state space: a stochastic resource-constrained project scheduling application. Computers and Chemical Engineering, 28, 2039–2058. CrossRefGoogle Scholar
  17. Csirik, J., Johnson, D. S., Kenyon, C., Orlin, J. B., Shor, P. W., & Weber, R. R. (2000). On the sum-of-squares algorithm for bin packing. In Proc. 32nd annual ACM symp. on theory of computing (STOCS’2000) (pp. 208–217). Google Scholar
  18. de Farias, D., Van Roy, B. (2002). Approximate linear programming for average-cost dynamic programming. In S. Becker, S. Thrun, & K. Obermayer (Eds.), NIPS (pp. 1587–1594). Cambridge: MIT Press. Google Scholar
  19. Dean, B., Goemans, M. X., & Vondrak, J. (2004). Approximating the stochastic knapsack problem: the benefit of adaptivity. In Proceedings of the 45th annual IEEE symposium on foundations of computer science (pp. 208–217). Rome, Italy, 2004. Google Scholar
  20. Fiat, A., & Woeginger, G. (1998). Online algorithms: the state of the art. Google Scholar
  21. Flaxman, A., Frieze, A., & Krivelevich, M. (2005). On the random 2-stage minimum spanning tree. In ACM-SIAM symposium on discrete algorithms (SODA-2005). Google Scholar
  22. Higle, J. L., & Sen, S. (1996). Stochastic decomposition. Dordrecht: Kluwer Academic. Google Scholar
  23. Kall, P., & Wallace, S. W. (1994). Stochastic programming. Chichester: Wiley. Google Scholar
  24. Karlin, A., Manasse, M., Rudolph, L., & Sleator, D. (1988). Competitive snoopy caching. Algorithmica, 3, 79–119. CrossRefGoogle Scholar
  25. Kesselman, A., Lotker, Z., & Mansour, Y. (2004). Buffer overflow management in QoQ switches. SIAM Journal on Computing, 33, 563–583. CrossRefGoogle Scholar
  26. Kleywegt, A., & Shapiro, A. (1999). The sample average approximation method for stochastic discrete optimization. SIAM Journal on Optimization, 15(1–2), 1–30. Google Scholar
  27. Koutsoupias, E., & Papadimitriou, C. H. (2000). Beyond competitive analysis. SIAM Journal of Computing, 30(1), 300–317. CrossRefGoogle Scholar
  28. Larsen, A., Madsen, O., & Solomon, M. (2002). Partially dynamic vehicle routing-models and algorithms. Journal of Operational Research Society, 53, 637–646. CrossRefGoogle Scholar
  29. Moehring, R. H., Radermacher, F. J., & Weiss, G. (2004). Stochastic scheduling problems. I: General strategies. ZOR—Zeitschrift für Operations Research, 28, 193–260. CrossRefGoogle Scholar
  30. Moehring, R. H., Schulz, A., & Uetz, M. (1999). Approximation in stochastic scheduling: the power of LP-based priority policies. Journal of the Association for Computing Machinery, 46(6), 924–942. Google Scholar
  31. Nikovski, D., & Branch, M. (2003). Marginalizing out future passengers in group elevator control. In Uncertainty in Artificial Intelligence (UAI’03), 2003. Google Scholar
  32. Powell, W. B., & van Roy, B. (2004). Approximate dynamic programming for high-dimensional dynamic resource allocation problems. In Handbook of learning and approximate dynamic programming (pp. 261–279). Google Scholar
  33. Puterman, M. (1994). Markov decision processes. New York: Wiley. CrossRefGoogle Scholar
  34. Romisch, W., & Schultz, R. (1993). Stability of solutions for stochastic programs with complete recourse. Mathematics of Operations Research, 18, 590–609. CrossRefGoogle Scholar
  35. Ross, S. (1997). A first course in probability. Englewood Cliffs: Prentice Hall. Google Scholar
  36. Schultz, R. (1993). Continuity properties of expectation functions in stochastic integer programming. Mathematics of Operations Research, 18(3), 578–589. CrossRefGoogle Scholar
  37. Schultz, R. (1995). On structure and stability in stochastic programs with random technology matrix and complete integer recourse. Mathematical Programming, 70(1), 73–89. CrossRefGoogle Scholar
  38. Schultz, R., Stougie, L., & van der Vlerk, M. H. (1996). Two-stage stochastic integer programming: a survey. Statistica Neerlandica. Journal of the Netherlands Society for Statistics and Operations Research, 50(3), 404–416. Google Scholar
  39. Schultz, R., Stougie, L., & van der Vlerk, M. H. (1998). Solving stochastic programs with integer recourse by enumeration: a framework using Gröbner basis reductions. Mathematical Programming, 83, 229–252. Google Scholar
  40. Shapiro, A. (2005). On complexity of multistage stochastic programs.
  41. Shapiro, A., & Homem-de Mello, T. (1998). A simulation-based approach to two-stage stochastic programming with recourse. Mathematical Programming, Ser. A, 81(3), 301–325. CrossRefGoogle Scholar
  42. Shaw, P. (1998). Using constraint programming and local search methods to solve vehicle routing problems. In Proceedings of fourth international conference on the principles and practice of constraint programming (CP’98) (pp. 417–431). Pisa, October 1998. Google Scholar
  43. Shmoys, D., & Swamy, C. (2004). Stochastic optimization is (almost) as easy as deterministic computation. In IEEE symp. foundations of computer science (FOCS-2004) (pp. 228–237). Google Scholar
  44. Skutella, M., & Uetz, M. (2005). Stochastic machine scheduling with precedence constraints. SIAM Journal of Computing, 34(4), 788–802. CrossRefGoogle Scholar
  45. Solomon, M. M. (1987). Algorithms for the vehicle routing and scheduling problems with time window constraints. Operations Research, 35(2), 254–265. CrossRefGoogle Scholar
  46. Stougie, L. (1985). Design and analysis of methods for stochastic integer programming. PhD thesis, University of Amsterdam. Google Scholar
  47. Stougie, L., & van der Vlerk, M. H. (1997). Stochastic integer programming. In M. Dell’Amico et al. (eds.) Annotated bibliographies in combinatorial optimization (pp. 127–141). New York: Wiley. Google Scholar
  48. van der Vlerk, M. H. (1995). Stochastic programming with integer recourse. PhD thesis, University of Groningen, The Netherlands. Google Scholar

Copyright information

© Springer Science+Business Media, LLC 2009

Authors and Affiliations

  • Pascal Van Hentenryck
    • 1
  • Russell Bent
    • 1
  • Eli Upfal
    • 1
  1. 1.Brown UniversityProvidenceUSA

Personalised recommendations