“How Do I Evaluate THAT?” Experiences from a Systems-Level Evaluation Effort
In this paper we describe our experience deriving evaluation metrics for a systems-level framework called Ripple that connects software engineering and usability engineering life cycles. This evaluation was conducted with eight teams of graduate students (falling under four types of development models) competing in a joint software engineering and usability engineering course to create a software solution for a real world client. We describe the challenges of evaluating systems-level frameworks and the approach we used to derive metrics given our evaluation context. We conclude with the outcome of this evaluation and the effectiveness of the metrics we employed.
KeywordsSystems-level evaluation evaluation metrics goal-question metric
Unable to display preview. Download preview PDF.
- 3.Pyla, P.S., et al.: Towards a model-based framework for integrating usability and software engineering life cycles. In: Interact 2003 Workshop on Closing the Gaps: Software Engineering and Human Computer Interaction. 2003: Université catholique de Louvain, Institut d’ Administration et de Gestion (IAG) on behalf of the International Federation for Information Processing (IFIP), pp. 67–74 (2003)Google Scholar
- 4.Pyla, P.S., et al.: What we should teach, but don’t: Proposal for a cross pollinated HCI-SE curriculum. In: Frontiers in Education (FIE) Conference, Savannah, Georgia, pp. S1H17–22 (2004)Google Scholar
- 5.Pyla, P.S., et al.: Ripple: An event driven design representation framework for integrating usability and software engineering life cycles. In: Seffah, A., Gulliksen, J., Desmarais, M. (eds.) Human-centered software engineering: Integrating usability in the software development lifecycle, pp. 245–265. Springer, Heidelberg (2005)CrossRefGoogle Scholar
- 6.The Standish Group, The CHAOS Report (1994)Google Scholar
- 7.The Standish Group, Unfinished Voyages. A follow-up to The CHAOS Report (1995)Google Scholar
- 8.The Standish Group, Extreme CHAOS (2001)Google Scholar
- 9.Pyla, P.S., et al.: Evaluating Ripple: Experiences from a Cross Pollinated SE-UE Study. In: CHI 2007 Workshop on Increasing the Impact of Usability Work in Software Development, 4 pages (2007)Google Scholar
- 11.Fenton, N.E., Pfleeger, S.L.: Software metrics: A rigorous and practical approach, 2nd edn. International Thomson Computer Press (1997)Google Scholar