Abstract
Third-party software certification should attest that the software product satisfies the required confidence level according to certification standards such as ISO/IEC 9126, ISO/IEC 14598 or ISO/IEC 25051. In many application areas, especially in mission-critical applications, certification is essential or even mandatory. However, the certification of software products using common off-the-shelf (COTS) components is difficult to attain, as detailed information about COTS is seldom available. Nevertheless, software products are increasingly being based on COTS components, which mean that traditional certification processes should be enhanced to take COTS into account in an effective way. This paper proposes a mean to help in the certification of component-based systems through an experimental risk assessment methodology based on fault injection and statistical analysis. Using the proposed methodology the certification authority or the system integrator can compare among components available the one that best fit for the system that is assembling a component that provides a specific functionality. Based on the results it is also possible to decide whether a software product may be considered certified or not in what concerns the risk of using a COTS into the system. The proposed approach is demonstrated and evaluated using a space application running on top of two alternative COTS real-time operating systems: RTEMS and RTLinux.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
References
Amland, S.: Risk-based Testing: Risk analysis fundamentals and metrics for software testing including a financial application case study. The Journal of Systems and Software 53, 287–295 (2000)
Arlat, J., et al.: Fault Injection and Dependability Evaluation of Fault Tolerant Systems. IEEE Transaction on Computers 42(8), 919–923 (1993)
Bach, J.: Heuristic Risk-Based Testing. In: Software Testing and Engineering Magazine (1999)
Basili, V., Briand, L., Melo, W.: Measuring the Impact of Reuse on Quality and Productivity in Object-Oriented Systems. Technical Report, University of Maryland, Dep. Of Computer Science, CS-TR-3395 (1995)
Chillarege, R., Orthogonal Defect Classification, Ch. 9 of Handbook of Software Reliability Engineering, M. Lyu Ed., IEEE Computer Society, McGraw-Hill (1995)
Christmansson, J., Chillarege, R.: Generation of an Error Set that Emulates Software Faults-Based on Fields Data. In: Proc. of 26th Int. Symp. on Fault-Tolerant Computing, Sendai, Japan, pp. 304–313 (1996)
Colombo, R., Guerra, A.: The Evaluation Method for Software Product. In: ICSSEA 2002. Proc. of Int. Conf. on Software & Systems Engineering & Applications, Paris, France (2002)
Councill, B.: Third-Party Certification and Its Required Elements. In: Proc. of The 4th Workshop on Component-Based Software Engineering (CBSE), Springer, Heidelberg, Canada. Lecture Notes in Computer Science (LNCS) (2001)
Durães, J.: Madeira, H. Definition of Software Fault Emulation Operators: A Field Data Study. In: Proc. of The International Conference on Dependable Systems and Networks - DSN2003, pp. 105-114, San Francisco, USA (2003) (William Carter Award for the best student paper)
Durães, J., Madeira, H.: Emulation of Software Faults: A Field Data Study and a Practical Approach. IEEE Transactions on Software Engineering 32(11) (November 2006), ISSN: 0098-558
El Emam, K., Benlarbi, S., Goel, N., Rai, S.: Comparing Case-based Reasoning Classifiers for Predicting High Risk Software Components. Journal of Systems and Software 55(3), 301–320 (2001)
EN 45020 General Terms and Definitions Concerning Standardization and Related Activities. CEN, Brussels (1993)
Fenton, N., Ohlsson, N.: Software Metrics and Risk. In: FESMA 1999. Proc. of The 2nd European Software Measurement Conference (1999)
Halstead, M.: Elements of Software Science. Elsevier Science Inc., New York (1977)
Hosmer, D., Lemeshow, S.: Applied Logistic Regression. John Wiley & Sons, Chichester (1989)
Health & Safety Commission The use of computers in Safety-critical Applications. Technical Report, UK (1998)
Hudepohl, et al.: EMERALD: A Case Study in Enhancing Software Reliability. Proc. of IEEE Eight Int. Symposium on Software Reliability Engineering - ISSRE98 98, 85–91 (1998)
ISO/IEC 12119. International Organization For Standardization ISO/IEC 12119, Information Technology - Software packages - Quality requirements and testing, p. 16, Geneve (1994)
ISO/IEC 14598-1. International Organization For Standardization ISO/IEC 14598-1 Information Technology - Software product evaluation - Part 1: General Overview; Geneve ISO (1999)
ISO/IEC 9126-1. International Organization For Standardization ISO/IEC 9126-1, Software Engineering - Software product quality - Part 1: Quality Model; Geneve ISO (2001)
ISO/IEC 25051 Software Engineering - Requirements for quality of Commercial Off-The-Shelf (COTS) software product and instructions for testing, Final Draft International Standard (2006)
Iyer, R.: Experimental Evaluation. In: Special Issue FTCS-25 Silver Jubilee, 25th IEEE Symposium on Fault Tolerant Computing, pp. 115–132 (1995)
Jacobson, I., Griss, M., Jonsson, P.: Software Reuse: Architecture, Process and Organization for Business Success. Addison-Wesley, Longman (1997)
Khoshgoftaar, et al.: Process Measures for Predicting Software Quality. In: Proc of High Assurance System Engineering Workshop - HASE 1997 (1997)
Leveson, N.: Safeware, System Safety and Computers. Addison-Wesley Publishing Company, Reading (1995)
The linux kernel. Accessed on Feb/06 (2006), http://www.kernel.org
Lyu, M., Chen, J., Avizienis, A.: Experience in Metrics and Measurements for N-Version Programming. Int. Journal of Reliability, Quality and Safety Engineering 1(1), 41–62 (1994)
Lyu, M.: Handbook of Software Reliability Engineering. IEEE omputer Society Press, McGraw-Hill, Los Alamitos (1996)
Madeira, H., Vieira, M., Costa, D.: On the Emulation of Software Faults by Software Fault Injection. In: Proc. of The Int. Conf. on Dependable Systems and Networks, NY, USA (2000)
Menzies, T., Greenwald, J., Frank, A.: Learning Defect Predictors. Journal (submitted 2006) (accessed, February 2006), http://menzies.us/
Moraes, R., Durães, J., Martins, E., Madeira, H.: A field data study on the use of software metrics to define representative fault distribution. In: DSN 2006. Proc. of The International Conference on Dependable Systems & Networks, IEEE Computer Society Pres, Los Alamitos (2006)
Moraes, R., Durães, J., Barbosa, R., Martins, E., Madeira, H.: Experimental Risk Assessment and Comparison using Software Fault Injection. In: The International Conference on Dependable Systems and Networks - DSN 07, Edimburgo (2007)
Morris, J., Lee, G., Parker, K., Bundell, G., Lam, C.: Software Component Certification. IEEE Computer 34(9), 30–36 (2001)
Musa, J.: Software Reliability Engineering. McGraw-Hill, New York (1996)
Munson, J., Khoshgoftaar, T.: Software Metrics for Reliability Assessment. In: Michael, R. (ed.) Handbook of Software Reliability Engineering, IEEE Comp. Society Press, Los Alamitos (1995)
Kitchenham, B., Pfleeger, S., Fenton, N.: Towards a framework for software measurement validation. IEEE Transactions on Software Engineering 21(12), 929–944 (1995)
Rodríguez-Dapena, P.: Software Safety Certification: A Multidomain Problem. IEEE Software 16(4), 31–38 (1999)
Rome Laboratory (RL). Methodology for Software Reliability Prediction and Assessment. Technical Report RL-TR-92-52, vol. 1 and 2 (1992)
Rosenberg, L., Stapko, R., Gallo, A.: Risk-based Object Oriented Testing. In: Proc of. 13th International Software / Internet Quality Week-QW, San Francisco, California, USA 2 (2000)
Resource Standard Metrics, Version 6.1 (2005), Last access: http://msquaredtechnologies.com/m2rsm/rsm.htm
Real-Time Operating System for Multiprocessor Systems. (February 2006) (accessed), http://www.rtems.com
Rushby, John Modular Certification. Langley Research Center. Report Number: NAS 1.26212130, NASA CR-2002-212130, SRI-11003
Sherer, S.: A Cost-Effective Approach to Testing. IEEE Software 8(2), 34–40 (1991)
Singpurwalla, N.: Statistical Methods in Software Engineering: Reliability and Risk, 1st edn. Springer, Heidelberg (1999)
Stafford, J., Wallnau, K.: Is Third-Party Certification Necessary? In: Proceedings of the 4th ICSE Workshop on Component-Based Software Engineering, Toronto, Canada, May, Toronto, Canada, pp. 13–17 (2001)
Systems Integration Requirements Task Group Certification Considerations for Highly-Integrated or Complex Aircraft Systems, Technical Report AS-1C, ASD, SAE (1996)
Tang, M., Kao, M., Chen, M.: An Empirical Study on Object-Oriented Metrics. In: Proceedings of the Sixth International Software Metrics Symposium, pp. 242–249 (1999)
Nuclear Safety Directorate Computer Based Safety Systems. Technical Assessment Guide T/AST/046, UK (2000)
Testwell Oy Ltd. (2006) (accessed on March 2006), http://www.testwell.fi
Voas, J.: Certifying Off-the-Shelf Software Components. IEEE Computer 31(6), 53–59 (1998)
Voas, J.: Certifying Software for High-Assurance Environments. IEEE Software 16(4), 48–54 (1999)
Weyuker, E.: Testing Component-Based Software: A Cautionary Tale. IEEE Software (1998)
Yang, Y., Boehm, B., Clark, B.: Assessing COTS Integration Risk Using Cost Estimation Inputs. In: Proc. of 28th International Conference on Software Engineering, Shangai, China (2006)
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 2007 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Moraes, R., Durães, J., Martins, E., Madeira, H. (2007). Component-Based Software Certification Based on Experimental Risk Assessment. In: Bondavalli, A., Brasileiro, F., Rajsbaum, S. (eds) Dependable Computing. LADC 2007. Lecture Notes in Computer Science, vol 4746. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-75294-3_14
Download citation
DOI: https://doi.org/10.1007/978-3-540-75294-3_14
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-75293-6
Online ISBN: 978-3-540-75294-3
eBook Packages: Computer ScienceComputer Science (R0)