Evaluating the Next Generation of Multimedia Software

Abstract

Developing new multimedia, software applications is a particularly fascinating but challenging task, depending, as it does, on the thorough evaluation of new concepts and prototypes. As complexity and choice increases exponentially, future development will be increasingly challenging for emerging, intelligent, multimedia and virtual reality applications, including Web 3D. In the past, we have been able to rely on generic evaluative heuristics. But are they generalisable to the newer generations of software? In this study, participants saw demonstrations of two interactive, virtual reality, multimedia applications, namely Second Life and 3d mailbox. They used them to generate questionnaires to capture those key aspects of such applications that were important to them. They did not find it difficult to generate questions grounded in their own experience. The resulting items turned out to be validated by significant levels of internal consistency across dependent variables. Surprisingly, however, the new heuristics bore little resemblance to traditional or current, cognitive items. The overwhelming influence was that of the immersive impact of such applications rather than standard design issues or cognitive user factors. Clearly, new software innovations require equally new innovations in evaluation techniques in general and context specific heuristics in particular, but we do not yet have a conceptual foundation upon which to base them.