SCHEDULING STRATEGIES FOR CQs

In this chapter, we focus on run-time resource allocation problem of stream data processing. Scheduling, a mechanism for the allocation of CPU cycles, determines the order in which operators or operator paths are scheduled at each time slot in a multiple-CQ processing system. The irregular and bursty input characteristics of data streams and the near real-time response requirements from stream-based applications further entail DSMSs to carefully allocate the limited resources in the system. Improper resource allocation can cause DSMSs to fail in handling temporary bursty data streams and in providing timely responses. As we will show in this chapter, improper resource allocation can cause delayed responses which may not be acceptable to many applications. Improper allocation of resources can also cause steep increase in maximal memory needed by the system and may even exceed physical memory available in the system resulting in system failures. However, these failures can be avoided with proper resource allocation mechanisms in place and proper capacity planning (as discussed in Chapter 5). The long-running characteristic of CQs and the unbounded nature of inputs further makes the resource allocation a run-time problem as decisions may have to be changed during the course of execution of a CQ. In short, it is clear that a resource allocation mechanism is critical to the success of a DSMS.