Advertisement

An Access Scheduling Tree to Achieve Optimal Freshness in Local Repositories

  • Li Qin
  • Vijayalakshmi Atluri
Conference paper
Part of the Lecture Notes in Computer Science book series (LNCS, volume 2738)

Abstract

To achieve improved availability and performance, often, local copies of remote data from autonomous sources are maintained. Examples of such local copies include data warehouses and repositories managed by web search engines. As the size of the local data grows, it is not always feasible to maintain the freshness (up-to-dateness) of the entire data due to resource limitations. Previous contributions to maintaining freshness of local data use a freshness metric as the proportion of fresh documents within the total repository (we denote this as average freshness). As a result, even though updates to more frequently changing data are not captured, the average freshness measure may still be high. In this paper, we argue that, in addition to average freshness, it is important that the freshness metric should also include the proportion of changes captured for each document, which we call object freshness. The latter is particularly important when both the current and historical versions of information sources are queried or mined. We propose an approach by building an access scheduling tree (AST) to precisely schedule access to remote sources that achieves optimal freshness of the local data under limited availability of resources. We show, via experiments, the performance of our approach is significantly higher than a linear priority queue.

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

References

  1. 1.
    Brewington, B.E., Cybenko, G.: How Dynamic is the Web? In: 9th World Wide Web Conference, WWW9 (2000)Google Scholar
  2. 2.
    Carrano, F.M., Prichard, J.J.: Data Abstraction and Problem Solving with C++, 3rd edn. Addison-Wesley, Reading (2001)Google Scholar
  3. 3.
    Cho, J., Garcia-Molina, H.: The Evolution of the Web and Implications for an Incremental Crawler. In: 26th International Conference on Very Large Databases (VLDB), pp. 200–209 (2000)Google Scholar
  4. 4.
    Cho, J., Garcia-Molina, H.: Synchronizing a Database to Improve Freshness. In: ACM SIGMOD International Conference on Management of Data, pp. 117–128 (2000)Google Scholar
  5. 5.
    Cho, J., Garcia-Molina, H., Page, L.: Efficient Crawling Through URL Ordering. In: 7th World Wide Web Conference, WWW7 (1998)Google Scholar
  6. 6.
    Douglis, F., Feldmann, A., Krishnamurthy, B., Mogul, J.: Rate of Change and Other Metrics: A Live Study of the World Wide Web. In: USENIX Symposium on Internetworking Technologies and Systems (December 1997)Google Scholar
  7. 7.
    Heydon, A., Najork, M.: Mercator: A Scalable, Extensible Web Crawler. World Wide Web 2(4), 219–229 (1999)CrossRefGoogle Scholar
  8. 8.
    Qin, L., Atluri, V.: An Access Scheduling Tree to Achieve Optimal Freshness in Local Repositories. Technical report (2002)Google Scholar
  9. 9.
    Wang, Y., DeWitt, D.J., Cai, J.-Y.: X-Diff: An Effective Change Detection Algorithm for XML Documents. In: 19th International Conference on Data Engineering, ICDE (2003)Google Scholar

Copyright information

© Springer-Verlag Berlin Heidelberg 2003

Authors and Affiliations

  • Li Qin
    • 1
  • Vijayalakshmi Atluri
    • 1
  1. 1.MSIS Department and, Center for Information Management, Integration and Connectivity (CIMIC)Rutgers UniversityNewarkUSA

Personalised recommendations