Chapter

Digital Libraries: For Cultural Heritage, Knowledge Dissemination, and Future Creation

Volume 7008 of the series Lecture Notes in Computer Science pp 331-340

Towards Very Large Scale Digital Library Building in Greenstone Using Parallel Processing

  • John ThompsonAffiliated withDepartment of Computer Science, University of Waikato
  • , David BainbridgeAffiliated withDepartment of Computer Science, University of Waikato
  • , Hussein SulemanAffiliated withDepartment of Computer Science, University of Cape Town

Abstract

As very large digital library collections become more commonplace, software tools must adapt appropriately. This paper reports on an evolution of the Greenstone Digital Library software to support parallel processing during the collection building phase. A series of experiments were conducted to first establish a basic speed-up factor, and then deconstruct the parallelisation process to understand the execution profile of the application. Several bottlenecks were identified and resolved to further improve the performance. The adaptation of Greenstone confirms that the build phase is indeed a suitable candidate for parallelisation; and suggests that parallelisation of processing is a new avenue for exploration in emerging digital library architectures.

Keywords

Greenstone VLDL Parallel Processing Open MPI