Do semantic standards lack quality? A survey among 34 semantic standards
- First Online:
- Cite this article as:
- Folmer, E., Oude Luttighuis, P. & van Hillegersberg, J. Electron Markets (2011) 21: 99. doi:10.1007/s12525-011-0058-y
The adoption of standards to improve interoperability in the automotive, aerospace, shipbuilding and other sectors could save billions. While interoperability standards have been created for a number of industries, problems persist, suggesting a lack of quality of the standards themselves. The issue of semantic standard quality is not often addressed. In this research we take a closer look at the quality of semantics standards, development processes, and survey the current state of the quality of semantic standards by means of a questionnaire that was sent to standards developers. This survey looked at 34 semantic standards, and it shows that the quality of semantic standards for inter-organizational interoperability can be improved. Improved standards may advance interoperability in networked business. Improvement of semantic standards requires transparency of their quality. Although many semantic standard development organisations already have quality assurance in place, this research shows that they could benefit from a quality measuring instrument.
KeywordsNetworked businessSemantic standardsQualityInteroperabilityAdoption
As early as 1993, a number of businesses and governments alike were aware of the importance of standards for ensuring interoperability (Rada 1993). Today, in an increasingly interconnected world, interoperability is more important than ever, and interoperability problems are very costly. Studies of the US automobile sector, for example, estimate that insufficient interoperability in the supply chain adds at least $1 billion in additional operating costs, of which 86% is attributable to data exchange problems (Brunnermeier and Martin 2002). The adoption of standards to improve interoperability in the automotive, aerospace, shipbuilding and other sectors could save billions (Gallaher et al. 2002).
Although interoperability standards have been created for a range of industries (Zhao et al. 2005), problems persist, suggesting a lack of quality of the standards themselves, and the processes by which they are developed. In 2009, the European Commission recognized the importance of quality of standards and set a policy to “increase the quality, coherence and consistency of ICT standards” (European Commission 2009). Sherif et al. (2005) state that their paper on standards’ quality was the first to address this topic, albeit only for technical standards. But what about semantic standards? Even though they are important in the creation of inter-organizational interoperability and solving data exchange problems, is there a need to measure the quality of semantic standards?
The ability to measure the quality of semantic standards can be useful to compare the quality of different standards, but also to (im)prove the quality of individual standards. Regarding semantic standards, Markus et al. (2006) asserts “the success of vertical information systems standards diffusion is affected by the technical content of the developed standard, …”. In other words, the quality of a standard is directly correlated to its adoption. However, adoption involves acceptance and implementation, and does not necessarily mean that interoperability will be achieved. In other words not all successful standards (high adoption) are high quality standards that lead to interoperability.
Despite the importance of standards in the evolution of information and communication technology (Lyytinen and King 2006), the issue of semantic standard quality is not often addressed (Folmer et al. 2009). In this research we take a closer look at the quality of semantics standards and their development processes, and survey the current state of quality and adoption of semantic standards through a survey of standards developers.
The remainder of this paper will present the definitions used, research structure, the survey population and the results of this survey, before answering the main research question in the concluding section.
Quality of standards defined
Development approach: Most semantic standards are developed outside traditional Standard Development Organizations (e.g. ISO) (Rada and Ketchell 2000), resulting in a wide range of standard development organisations per industry (RosettaNet for the electrotechnical industry, HL7 for health care, etc.), each having their own development and maintenance approach.
Context dependencies: Both the content and development approaches of semantic standards are highly dependent on the context. Examples of context factors are, among others, regulatory, governance structure, government participation, IT maturity of an industry and the market situation.
We define the quality of a semantic standard as: its ability to achieve its intended purpose—semantic interoperability—effectively and efficiently. A high quality standard is, or has a high chance to become, an effective and efficient solution for an interoperability problem; a low quality standard does not solve the problem for which it is designed, cannot be implemented efficiently, or has little chance to become adopted. All phases of the lifecycle of a standard (Söderström 2004) may influence quality. Moreover, quality deals with both intrinsic aspects (the document) and situational aspects (environment) of the standard. This definition applies Juran’s definition of quality—fitness for use (Juran and Gryna 1988)—to the semantic standards domain, and is in line with the ISO 9126 software quality definition: the totality of characteristics of an entity that bear on its ability to satisfy stated and implied needs (ISO/IEC 2001). In the end, high quality semantic standards involve network externalities, avoid lock-ins, increase variety of systems products, trade facilitation and reduce transaction costs (Blind 2004). More importantly, they solve or lower economic and social problems, such as the $1 billion imperfect interoperability costs of US automotive industry (Brunnermeier and Martin 2002) or the calculated 98,000 losses of life caused by lack of interoperability in care IT systems (Venkatraman et al. 2008).
As the starting point for this research, a research framework was developed. The starting point was the main research question defined in generic terms: Is there an urgent need, based on the current situation and experienced problems, for a solution? In relation to the subject of semantic standards quality, this can be formulated as: Is there, based on the current standards development processes and experienced interoperability and adoption problems, a need to elicit the quality of semantic standards? In this research, such elicitation involves a quality measurement instrument for semantic standards.
Proposition for the current standards development process
One of the issues in this research is whether the quality of semantic standards can be improved. Therefore it is interesting to know whether the current standards development processes include steps focused on quality aspects of standards, and whether there is room for further improvement.
The quality of standards can be increased by improving the standards development process.
Proposition for the interoperability problem
If there is room for quality improvement, it would only make sense if it leads to better semantic interoperability.
Improved quality of standards leads to improved interoperability.
Proposition for the adoption problem
Even if high-quality semantic standards have a great semantic interoperability potential, such potential only materializes when the standard is actually being adopted in processes and systems.
High-quality standards will have a better chance of being adopted.
Proposition for the desired assessment and visibility of quality
If the interoperability and adoption problems (addressed in propositions 2 and 3) were proven to be related to the quality of standards, then we need to verify whether the proposed solution could contribute to solving these problems. Is transparency of the quality of the standard valuable for standards developers? And, if so, do standards developers value an instrument for this?
There is a need to make the quality of the standard visible by assessment.
We conducted a survey to test these propositions. Surveys are often appropriate for problem clarification and verification of problem relevance. They offer a way of getting more stakeholders involved and getting structured and comparable results in a time-efficient manner (Creswell 2009). Other methods, such as interviews or focus groups, would limit us too much in the scope of standards and respondents.
Our research addresses semantic standards in general, so that the survey had to include a broad range of semantic standards. The intended respondents were standards developers from Standard Setting Organisations (SSO). Unfortunately, to the best of our knowledge, no up-to-date list of semantic standards exists. Nevertheless the authors’ professional networks were activated and a list of possible respondents was set up, mainly from the Netherlands, Belgium, and Germany. In order to get additional and more international respondents, literature on semantic standards was assessed. Following other research (Zhao et al. 2005), xml.org was used. This list was enhanced with other semantic standards that are mentioned in literature (Hasselbring 2000; Markus et al. 2006; Nelson et al. 2005; Steinfield et al. 2007). The Internet was searched for standards developers involved in those standards. In cases where the standards developers could not be identified, we decided to send the invitation to the general e-mail address of the development organisation.
Structure of survey questions
Is there, based on the current standards development processes and experienced inter-operability and adoption problems, a reason to develop more knowledge about quality of standards?
Current standards development process
Quality as part of the current process
Quality assurance is an explicit part of our current development process of the standard
Quality end-check implemented
There is not a minimum quality check in place before the standard is released
Usage of tools for quality
An instrument/tool is used to measure the quality of our standard
Perceived need for quality improvement
The quality of the current standard can be improved
New or updated releases cover avoidable corrections to the previous versions of our standard
The achieved interoperability is worse than expected
Influence of standard on interoperability
Currently the achieved interoperability is affected by the limitations of our standard
Improvements to the quality of our standard will lead to improved implementations and ultimo lead to improved interoperability
The current adoption is better than expected
Influence of standard on adoption
Design choices of the standard have influenced the adoption process
The adoption will be more successful when the quality of the standard is explicitly known to the users, and proven sufficient or improved
Desired assessment and visibility of quality
Minimum quality for interoperability
A minimum quality level of our standard is needed to achieve interoperability
Minimum quality for adoption
A minimum quality level of our standard is needed for high adoption rates
Potential usage of instrument
I will not use an instrument/tool to measure the quality of the standard when it will be available
Assessment of quality
It would be helpful to have an instrument/tool/knowledge to gain insight in the quality of the standard
Visibility of quality
If the quality of the standard is not known then it is hard to improve the standard
Subsequently, the survey results for each statement were analyzed and related to the propositions. A correlation analysis was done to find additional insights. The results section of this paper will relate the survey results to the propositions, and possible explanations for the results are given.
Our research approach implies several limitations. First, this research does not address the factors that influence quality, although the characteristics of the development process of standards is expected to influence the quality of the standard, just as the management process of the standard and other context dependencies. This aspect is out of scope for this research, but addressed by other research like the research on best practices for standardization resulting in a model for development and management of semantic IS standards called BOMOS (Folmer and Punter 2011). Another limitation is the broad definition and invitation of semantic standards, resulting in a heterogeneous group of semantic standards, making it hard to generalize our results to all individual or sub-groups of semantic standards. Finally, although we broadly defined quality as “fitness for use”, implying the relation with achieving interoperability; we are aware that in practice different views on quality exist. Some relate quality solely to the specification document, while others see quality as the adoption success of the standard. To align the respondents, the survey started with presenting a definition on quality, but still we are aware that some respondents will use a different view on quality.
The chosen approach is aimed to find out the practical relevance and issues related to quality of semantic standards, and therefore deliberately contains broad definitions. The limitations of our research can be addressed in follow-up in-depth research, but will only become relevant when the propositions of our research are supported.
The survey was held from August 25th to September 25th 2009. In total, 111 persons were invited, of which 48 responded, yielding a response rate of 43.6%. These 48 respondents represent 34 different semantic standards. These 34 standards constitute a wide collection of semantic standards, both international (e.g. HL7) and national (e.g. SETU), both governmental (e.g. StUF) and industry (e.g. Chem eStandards), and across different industry domains (e.g. healthcare, education, tourism, agriculture, finance, etc.).
List of participating semantic standards
Name of standard
Exchange of insurance data
Information exchange about water management
Product modelling with integrated parametric geometry
CEN/ISO EN 13606
Exchange between Electronic Health Records
Chemical Industry B2B data exchange
Interoperability between digital learning related systems
Standardization of the description of, mostly Dutch, learning objects
Data exchange between bulb growers, intermediaries and bulbtraders
Data exchange standard for the energy domain
Exchange of student information
Data for use in assessment procedures
Interoperability in tourism (accommodation, events, activities, attractions, food & beverage)
Interoperability in the supply chain from grower to retail
Trade processing between firms for Over The Counter (OTC) Derivatives
Simplified and distributed data exchange for complex building resource analysis simulation engines
Identification of products, locations, documents, etc.
Meaningful exchange of health information across different healthcare information systems
Quality Approaches for Learning, Education and Training
Information exchange between jewelers and suppliers
Financial services trading and market data
Standard for legal sources
Exchange of geo-information in different communities (spatial planning, cultural heritage, etc.)
Business information and transaction standards for the travel industry
Metadata standard for Dutch government organisations
Different ways of electronic message exchange in the forest and paper industry
Exchange of information in the staffing industry
Interoperability between systems in short sea shipping
Sharing of recruiting and staffing data between suppliers, customers and third parties.
Administrative and Instructional applications within a school, school district and State
Connecting clinical domains and cross country borders
Information exchange between Dutch municipal systems
Structured information exchange in social security
The sending of declarations by health practitioners to several health insurers and also the receiving of declarations by a health insurer from several health practitioners
Taxonomies developed locally/regionally in use globally
The detailed survey results, including correlation analysis of the question results, are presented in appendix B. Significant correlations and other survey results that contribute to the propositions are mentioned when we review the propositions in this section.
The quality of standards can be increased by improving the current standards development process.
This proposition was agreed upon by 64.6% of the respondents, while 8.3% disagreed. Other results show that quality is embedded within development organisations as 77.1% of the respondents have quality assurance as an explicit part of their standards development process. And 81.3% already have some kind of minimum quality check in place before a standard is released. Although the survey did not ask for specifics, this quality check could be in the form of a final review before a new (version of the) standard is released.
It seems that current standards contain avoidable errors since 45.8% of the respondents stated that new or updated releases of their standard include corrections to avoidable mistakes in previous versions. This correlates to the statement about whether the quality of the standard can be improved (P = 0.29; p = 0.05). This result suggests there is considerable room for improvement within the current standards. But, is there any value in additional quality? This question is covered by the second proposition.
Improved quality of standards leads to improved interoperability.
A substantial 66.7% of the respondents agreed with this proposition; only 8.4% disagreed. An even higher percentage (89.6%) viewed that a minimum quality level is a necessary requirement for interoperability. These figures lead us to conclude that the respondents correlate the quality of a standard with the achieved interoperability. At the same time, 64.6% disagreed with the statement that the achieved interoperability is affected by the limitations of the standard. Respondents seemed to anticipate and accept the interoperability level achieved. Only 10.4% said that interoperability is worse than expected. This satisfaction, or acceptance, of achieved interoperability may seem surprising in relation to interoperability problems in practice. However it might be explained by the population of the survey, consisting of standards developers. Responses from standards implementers or users might lead to different results, because they might have a different opinion of interoperability in practice.
We also see a positive correlation between a standard’s achieved interoperability and whether there is quality assurance as part of the development process of the standard: Standards that have quality assurance as part of their current development process also have a minimum quality check in place (P = 0.57; p = 0.00), and rank high on the achieved interoperability (P = 0.32; p = 0.03). The data also show that where the quality of the standard could be improved the achieved interoperability is actually worse than expected (P = 0.32; p = 0.03). These results all confirm the positive correlation between quality and the achieved interoperability. This is in line with the literature suggesting that the need for interoperability is one of the key drivers for the development of standards (Nelson et al. 2005). Will the same hold for adoption, that is, will quality improvement increase adoption rates? This is addressed by the third proposition.
High-quality standards will have a better chance of being adopted.
Although 60.4% saw a relation between design choices of the standard and the adoption of the standard, this proposition is not completely supported by this survey. The results show more diverse opinions on this topic; 37.5% both agreed and disagreed with the question whether adoption will be more successful when the quality of the standard is known, proven sufficient, or improved.
“Don’t forget the important role of communities and the community owner(s) (dominant players). Often these companies have a strong influence on the adoption and quality of standards.”
“The degree of adoption depends on many other things than just the quality of the standard.”
“So, although improving the standard itself is always a worthy goal, if the action to be transacted cannot be agreed upon, then adoption will always be limited.”
Even so, 83.3% thought a minimum quality level of the standard is needed for high adoption rates. A significant correlation is present for the following: In the cases where the adoption is rated better than expected, design choices of the standard have influenced adoption (P = 0.48; p = 0.00), and the adoption of the standard will be more successful if the quality is known, proven sufficient or improved (P = 0.42; p = 0.00). This suggests dependence between adoption and the quality of the standard which is also supported by case studies. Both the MISMO case within the mortgage industry (Markus et al. 2006) and the RosettaNet case within the electro technical industry (Boh et al. 2007) reported a similar relation between adoption and the content of the standard. Also in a recent innovation-centric adoption and diffusion framework for standards, standards characteristics are included as area that impacts adoption (Wapakabulo Thomas 2010). Our conclusion is that a standard’s quality is seen as necessary but is not really an adequate condition for adoption. But, will knowledge about a standard’s quality contribute to quality improvement? This is addressed by the fourth proposition.
There is a need to make the quality of the standard visible by assessment.
The majority (54.2%) agreed with the statement that it is difficult to improve the standard if the quality is not known (14.6% disagreed). Another 85.5% considered it helpful to have some kind of instrument to make quality transparent, thereby supporting this proposition.
A notable 81.3% would use a quality measurement instrument, when available. Interesting to see is that standards which focus on quality are potential tools for users of such an instrument: Standards that have quality assurance as part of their current development process also have a minimum quality check in place (P = 0.57; p = 0.00), and score high on the achieved interoperability (P = 0.32; p = 0.03), and will use an instrument to measure the quality when available (P = 0.40; p = 0.00). And also those respondents who agreed to the statement that improved quality might lead to improved interoperability, thought that it is helpful to have some kind of instrument to gain insight into the quality of the standard (P = 0.30; p = 0.04). This goes even further than the proposition; not only is there a need to make the quality visible by assessment, but also some kind of instrument would be welcomed to assess quality.
Remarkably, the majority of respondents who already deploy a quality check before a standard’s release also use some kind of instrument to measure the quality of the standard (P = 0.39; p = 0.01), and they would welcome a newly developed instrument and would use it when available (P = 0.38; p = 0.01).
Finally, there is no negative correlation between the current use of an instrument to measure the quality of the standards, and whether a new instrument would be welcomed. This suggests that respondents see room for enhancement or improvement of their quality assurance.
The results of our study indicate that quality of standards is not properly addressed in current standardization practice, and this reduces standards’ quality, and therefore interoperability. What are the possible reasons for this? A thorough analysis of such causes requires further research. At this point, however, we can present three possible explanations.
First, developing standards, much like enterprise interoperability, is not considered to be a profession yet (Oude Luttighuis and Folmer 2010). Even though considerable standardization experience and professionalism in formal standardization bodies is present, the semantic standards realm is characterized by a wide range of development processes. Most semantic standards are developed by a specific domain organization, and are outside of the traditional standard development organisations. This disperses standards development knowledge and experience and limits the proliferation of such experience and re-use of process and product components. This effect is increased by the fact that most standards developers are domain experts. Domain knowledge is crucial to standards development, but differs from general standardization expertise. Moreover, education and certification, so common for other professions, are hardly available or required in the standardization field. On the positive side, standards developers are intrinsically motivated (Teichmann 2010) and eager to improve the quality of their standards when appropriate knowledge and tools are available, as shown by the survey results. Adequate assessment of the impact of this factor would require further research and might yield opportunities for a more mature standardization profession.
Second, notwithstanding the role of standards in acquiring interoperability, they do not yet rank high on a CEO’s or a CIO’s priority list. Related topics to standardization, though, such as enterprise integration, business process improvement and reducing enterprise costs have been priorities among many CIOs’ (McDonald 2010; Park and Ram 2004). However, this does not seem to result in standardization, as a topic on its own, being given high priority.
A third possible explanation of the survey results is that, even though standards developers may think interoperability could be improved, current interoperability levels satisfy current business needs. Thus there could be a discrepancy between the supply side (standards developers) and the demand side (end users) regarding standards quality and the importance of interoperability. This research focuses on the supply side, and therefore cannot reflect the viewpoint of the demand side. An imperfect standard (from the viewpoint of standards developers) might be quite acceptable to the end user. Lower interoperability levels might satisfy current needs. This argument nevertheless deserves some nuancing. The respondents surveyed are standards developers, who have diverse backgrounds but include standards users (Zhao et al. 2005). Some are employed by software vendors. Others work for user organizations. So, at least some user perspective may be expected to have been included in the survey. Needless to say, to assess the impact of imperfect standards and interoperability on the demand side, we need to extend our study to end users.
Conclusions & further research
Is there, based on the current standards development processes and experienced interoperability and adoption problems, a need to elicit the quality of semantic standards?
Achieving inter-organizational interoperability may lead to significant cost savings, performance improvements and efficiency gains. A wealth of semantic standards have been developed for various industry sectors. We can conclude that there is a need for wider use of measures and tools to assess the quality of semantic standards. Currently there is no instrument available in the literature that enables the measurement of semantic standards quality (Folmer et al. 2009). A parallel can be drawn with the early days of Software Engineering when software development processes were informally defined and software product quality was not measured according to standard frameworks. Our current research is aimed at developing a framework for semantic interoperability standards, expressed in SMML as the domain specific language (Garcia et al. 2009). Such a framework should deal with both intrinsic aspects (the specification) as well as the situational aspects (external environment) of the standard. The quality of a standard is related to the adoption of the standard and achieved interoperability.
A widely adopted framework can make the quality of semantic standards assessable by standards developers and end users, i.e. the businesses that adopt and depend on semantic standards for interoperability. This is a much needed tool, as more and more new and competing semantic standards are being introduced. Without quality enhancement, standardization may become a failing paradigm, as argued by Cargill and Bolin (2007).
This article is distributed under the terms of the Creative Commons Attribution Noncommercial License which permits any noncommercial use, distribution, and reproduction in any medium, provided the original author(s) and source are credited.