Empirical Software Engineering

, Volume 14, Issue 5, pp 579–601 | Cite as

Experiences in developing and applying a software engineering technology testbed

  • Alexander LamEmail author
  • Barry Boehm
Industry Experience Report


A major problem in empirical software engineering is to determine or ensure comparability across multiple sources of empirical data. This paper summarizes experiences in developing and applying a software engineering technology testbed. The testbed was designed to ensure comparability of empirical data used to evaluate alternative software engineering technologies, and to accelerate the technology maturation and transition into project use. The requirements for such software engineering technology testbeds include not only the specifications and code, but also the package of instrumentation, scenario drivers, seeded defects, experimentation guidelines, and comparative effort and defect data needed to facilitate technology evaluation experiments. The requirements and architecture to build a particular software engineering technology testbed to help NASA evaluate its investments in software dependability research and technology have been developed and applied to evaluate a wide range of technologies. The technologies evaluated came from the fields of architecture, testing, state-model checking, and operational envelopes. This paper will present for the first time the requirements and architecture of the software engineering technology testbed. The results of the technology evaluations will be analyzed from a point of view of how researchers benefitted from using the SETT. The researchers just reported how their technology performed in their original findings. The testbed evaluation showed (1) that certain technologies were complementary and cost-effective to apply; (2) that the testbed was cost-effective to use by researchers within a well-specified domain of applicability; (3) that collaboration in testbed use by researchers and the practitioners resulted comparable empirical data and in actions to accelerate technology maturity and transition into project use, as shown in the AcmeStudio evaluation; and (4) that the software engineering technology testbed’s requirements and architecture were suitable for evaluating technologies and accelerating their maturation and transition into project use.


Testbed Software maturity Software adoption Technology evaluation Technology transition 



This work was supported by NASA-HDCP contracts to CMU, JPL, and USC. It also benefited from support from the JPL-MDS team and the NSF HDC programs that includes Dr. Roshanak Roshandel, Dr. Steve Fickas and his graduate students, Dr. David Garlan and the AcmeStudio team, Dr. Gupta, Dr. Helmy, Ganesha Bhaskara, and Dr. Carolyn Talcott. In addition, I’d like to acknowledge the USC graduate students who helped in developing SCRover.


  1. Benzel T, Braden R, Kim D, Neuman C, Joseph A, Ostrenga R et al Design, Deployment, and Use of the Deter Testbed. Proceedings of the DETER Community Workshop on Cyber Security Experimentation and Test, August 2007.Google Scholar
  2. Boehm B (1996) Anchoring the software process. IEEE Softw 73–82, (July). doi: 10.1109/52.526834
  3. Boehm B, Port D 2001. Balancing discipline and flexibility with the Spiral Model and MBASE. Crosstalk, December 2001, pp. 23–28 (
  4. Boehm, B. and USC Center for Software Engineering (2003) Guidelines for Model-Based (System) Architecting and Software Engineering. (
  5. Boehm B, Bhuta J, Garlan D, Gradman E, Huang L, Lam A et al (2004) Using testbeds to accelerate technology maturity and transition: The SCRover Experience. ACM-IEEE International Symposium on Empirical Software Engineering, August, pp. 117–126Google Scholar
  6. Booch G, Rumbaugh J, Jacobson I (1999) The unified modeling language user guide. Addison Wesley, ReadingGoogle Scholar
  7. Chillarege R, Bhandari IS, Chaar JK, Halliday MJ, Moebus DS, Ray BK et al (1992) Orthogonal Defect Classification—A Concept for In-Process Measurements. IEEE Trans Softw Eng 18(11). doi: 10.1109/32.177364
  8. Dvorak D, Rasmussen R, Reeves G, Sacks A (2000) Software architecture themes in JPL’s Mission Data System. Proceedings of 2000 IEEE Aerospace Conference.Google Scholar
  9. Fickas S, Prideaux J, Fortier A 2004. ROPE: Reasoning about OPerational Envelopes.
  10. Garlan D, Monroe RT, Wile D (2000). Acme: architectural description of component-based systems. In: Leavens GT, Sitaraman M (eds) Foundations of component-based systems. Cambridge University PressGoogle Scholar
  11. Kruchten P (2001) The rational unified process (2nd edn). Addison Wesley, ReadingGoogle Scholar
  12. Lindvall M, Rus I, Donzelli P, Memon A, Zelkowitz M, Betin-Can A et al (2007) Experimenting with software testbeds for evaluating new technologies. Empir Softw Eng 12(4):417–444 doi: 10.1007/s10664-006-9034-0 CrossRefGoogle Scholar
  13. Mettala E, Graham M (1992) The domain-specific software architecture program. Technical Report CMU/SEI-92-SR-9, CMU Software Engineering InstituteGoogle Scholar
  14. Mills H (1972) On The Statistical Validation of Computer Programs. IBM Federal Systems Division Report 72-6015Google Scholar
  15. Redwine S, Riddle W (1985) Software technology maturation. Proceedings of the 8th International Conference on Software Engineering (ICSE1985), pp. 189–200Google Scholar
  16. Rinker G (2002) Mission Data Systems Architecture and Implementation Guidelines. Ground System Architectures Workshop (GSAW 2002). El Segundo, CaliforniaGoogle Scholar
  17. RoboCup 2007. <>
  18. Roshandel R, Schmerl B, Medvidovic N, Garlan D, Zhang D (2004a) Understanding tradeoffs among different architectural modeling approaches. Proceedings of the 4th Working IEEE/IFIP Conference on Software Architecture (WICSA 2004) Oslo, NorwayGoogle Scholar
  19. Roshandel R, van der Hoek A, Mikic-Rakic M, Medvidovic N (2004b) Mae—a system model and environment for managing architectural evolution. ACM Trans Softw Eng Methodol 11(2):240–276 doi: 10.1145/1018210.1018213 CrossRefGoogle Scholar
  20. Roshandel R, Banerjee S, Cheung L, Medvidovic N, Golubchik L (2006) Estimating software component reliability by leveraging architectural models. 28th International Conference on Software Engineering (ICSE 2006), Shanghai, China, May, pp. 853–856Google Scholar
  21. Stone P (2003) Multiagent competition and research: lessons from RoboCup and TAC. RoboCup-2002: Robot Soccer World Cup VI. Springer Verlag, Berlin, pp 224–237Google Scholar
  22. Tracz W (1995) DSSA (Domain-Specific Software Architecture) pedagogical example. ACM SIGSOFT Softw Eng Notes 20(3):49–62CrossRefGoogle Scholar

Copyright information

© Springer Science+Business Media, LLC 2008

Authors and Affiliations

  1. 1.University of Southern CaliforniaLos AngelesUSA

Personalised recommendations