A framework for performance monitoring, load balancing, adaptive timeouts and quality of service in digital libraries

Abstract.

In this paper, we investigate the issues of performance management in large scale, autonomous and federated digital library systems, performing the tasks of indexing, searching and retrieval of information objects. We have defined a management architecture and performance framework for measuring and monitoring the behavior of digital libraries as they operate. Our architecture and mechanisms are easily applicable to other digital library systems of similar flavor and architecture. We implemented this architecture over a testbed of Dienst servers using real data and workload from the operational NCSTRL system. We have defined the relevant parameters for investigating the performance of the servers and have developed visualization tools to monitor the parameters. In addition, our performance framework provides mechanisms for load-balancing search requests in a network of digital library servers. We have demonstrated this by building a testbed for investigating a few novel load balancing policies. Given that network delays and outages are unpredictable over the Internet, we have developed new adaptive mechanisms to detect timeouts and provide quality of service.