Grid Computing Using Terracotta

Abstract

The Master/Worker pattern is one of the most well-known and common patterns for parallelizing work, and it is also one of the core patterns in grid computing to such an extent that most grid-enabling technologies can be seen more or less as industrial implementations of this pattern. In this chapter, we will explain the characteristics of the pattern, walk through some possible implementation strategies, and consider what’s really behind all the buzz. We will briefly discuss what grids are really all about and how they use the Master/Worker pattern to solve very challenging real-world problems. We’ll also show you, step by step, how you can implement a very scalable and high performing grid-style data processing and workload management framework with Terracotta. By the end of this chapter, you will have implemented your own grid running on Terracotta. The use case will walk you through building a distributed web crawler by hand. The resulting system will be able to run on as many JVMs as you want, crawl any web site as deeply as you want (following four levels of HREF links, for example), and store all the web pages on that site into your Terracotta grid.