An Access Scheduling Tree to Achieve Optimal Freshness in Local Repositories
To achieve improved availability and performance, often, local copies of remote data from autonomous sources are maintained. Examples of such local copies include data warehouses and repositories managed by web search engines. As the size of the local data grows, it is not always feasible to maintain the freshness (up-to-dateness) of the entire data due to resource limitations. Previous contributions to maintaining freshness of local data use a freshness metric as the proportion of fresh documents within the total repository (we denote this as average freshness). As a result, even though updates to more frequently changing data are not captured, the average freshness measure may still be high. In this paper, we argue that, in addition to average freshness, it is important that the freshness metric should also include the proportion of changes captured for each document, which we call object freshness. The latter is particularly important when both the current and historical versions of information sources are queried or mined. We propose an approach by building an access scheduling tree (AST) to precisely schedule access to remote sources that achieves optimal freshness of the local data under limited availability of resources. We show, via experiments, the performance of our approach is significantly higher than a linear priority queue.
Unable to display preview. Download preview PDF.
- 1.Brewington, B.E., Cybenko, G.: How Dynamic is the Web? In: 9th World Wide Web Conference, WWW9 (2000)Google Scholar
- 2.Carrano, F.M., Prichard, J.J.: Data Abstraction and Problem Solving with C++, 3rd edn. Addison-Wesley, Reading (2001)Google Scholar
- 3.Cho, J., Garcia-Molina, H.: The Evolution of the Web and Implications for an Incremental Crawler. In: 26th International Conference on Very Large Databases (VLDB), pp. 200–209 (2000)Google Scholar
- 4.Cho, J., Garcia-Molina, H.: Synchronizing a Database to Improve Freshness. In: ACM SIGMOD International Conference on Management of Data, pp. 117–128 (2000)Google Scholar
- 5.Cho, J., Garcia-Molina, H., Page, L.: Efficient Crawling Through URL Ordering. In: 7th World Wide Web Conference, WWW7 (1998)Google Scholar
- 6.Douglis, F., Feldmann, A., Krishnamurthy, B., Mogul, J.: Rate of Change and Other Metrics: A Live Study of the World Wide Web. In: USENIX Symposium on Internetworking Technologies and Systems (December 1997)Google Scholar
- 8.Qin, L., Atluri, V.: An Access Scheduling Tree to Achieve Optimal Freshness in Local Repositories. Technical report (2002)Google Scholar
- 9.Wang, Y., DeWitt, D.J., Cai, J.-Y.: X-Diff: An Effective Change Detection Algorithm for XML Documents. In: 19th International Conference on Data Engineering, ICDE (2003)Google Scholar