Abstract
The meaning and properties of a commonly used index of reliability, S/L,were examined critically. It was found that the index does not reflect any conventional concept of reliability. When used for an identical behavioral observation session, it is not statistically correlated with other reliability indices. Within an observation session, the standardizing measure of Lis beyond the control of the investigator. Furthermore, the reason for the choice of Las the standard is unclear. The role of chance agreement in S/Lis not known. The exact interpretation of the index depends on which observer reports L.Overall the conceptual and mathematical meaning of S/Lis dubious. It is suggested that the S/Lindex should not be used until its nature is shown to be a measure of reliability. Other approaches such as the intraclass correlations and generalizability coefficients should be used instead.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Armitage, P., Blendis, L. M., & Smyllie, H. C. (1966). The measurement of observer agreement in the recording of signs.Journal of Royal Statistical Society, Series A 129, 98–109.
Beck, R. A. (1979). Generalizability of behavioral observations: A clarification of interobserver agreement and interobserver reliability.American Journal of Mental Deficiency, 20, 37–46.
Fleiss, J. L. (1975). Measuring agreement between two judges on the presence and absence of a trait.Biometrics, 37, 651–659.
Gaylord-Ross, R. T., Haring, T. G., Breen, C., & Pitts-Conway, V. (1984). The training and generalization of social interaction skills with autistic youth.Journal of Applied Behavior Analysis, 17, 229–247.
Gill, P. E., Murray, W., & Wright, M. H. (1981).Practical Optimization. New York: Academic Press.
Goodman, L. A., & Kruskal, W. H. (1954). Measures of association for cross classifications.Journal of the American Statistical Association, 49, 732–764.
Hart, B. (1983). Assessing spontaneous speech.Behavioral Assessment, 5, 71–82.
Hartmann, D. P. (1977). Considerations in the choice of interobserver reliability estimates.Journal of Applied Behavior Analysis, 10, 103–116.
Hartmann, D. P. (1982). Assessing the dependability of observational data. In D. P. Hartmann (Ed.),Using observer to study behavior. San Francisco, CA: Jossey-Bass.
Hartmann, D. P., & Wood, D. D. (1982). Observational methods. In A. Bellack, M. Hersen, & A. E. Kazdin (eds.),International handbook of behavior modification and therapy. New York: Plenum.
Hawkins, R. P., & Dotson, V. A. (1975). Reliability scores that delude: An Alice in Wonderland trip through the misleading characteristics of interobserver agreement scores in interval recording. In E. Bamp & C. Semb (Eds.),Behavior Analysis: Areas of Research and Application. Englewood Cliffs, N.J.: Prentice-Hall.
Hays, W. L. (1973).Statistics for the Social Sciences. New York: Holt, Rinehart and Winston.
Jason, L. A., & Liotta, R. F. (1982). Reduction of cigarette smoking in a university cafeteria.Journal of Applied Behavior Analysis, 15, 573–577.
Kazdin, A. E. (1982).Single-case research designs: Methods for clinical and applied setting. New York: Oxford University Press.
Kelly, M. B. (1977). A review of observational data collection and reliability procedures reported in the Journal of Applied Behavior Analysis.Journal of Applied Behavior Analysis, 10, 97–101.
Kelly, M. L., & Stokes, T. F. (1982). Contingency contracting with disadvantaged youth: Improving classroom performance.Journal of Applied Behavior Analysis, 15, 447–454.
Murphy, H. A., Hutchinson, J. M., & Bailey, J. S. (1983). Behavioral school psychology goes outdoors: The effect of organized games and playground aggression.Journal of Applied Behavior Analysis, 16, 29–35.
Nunnally, J. C. (1978).Psychometric theory. New York: McGraw-Hill.
Prochnow-LaGrow, J. E. (1983).A meta-analysis of single subject research utilizing the differential reinforcement reinforcement of behavior omission. Doctoral dissertation. DeKalb: Northern Illinois University.
Suen, H. K., & Ary, D. (1984). Variables influencing one-zero and instantaneous time sampling outcome.Primates, 25, 89–94.
Suen, H. K., & Lee, P. S. C. (1984). Effects of the use of percentage agreement on behavioral observation reliabilities: A re-assessment. Paper presented at the Annual meeting of the American Educational Research Association, New Orleans, LA.
Yelton, A. R. (1979). Reliability in the context of the experiment: A comment on two articles by Birkimer and Brown.Journal of Applied Behavior Analysis, 12, 565–569.
Author information
Authors and Affiliations
Additional information
The authors are indebted to Johnny Matson for his critique of an earlier version of this paper.
Rights and permissions
About this article
Cite this article
Suen, H.K., Lee, P.S.C. & Prochnow-LaGrow, J.E. A critical review of theS/L reliability index. J Psychopathol Behav Assess 7, 277–287 (1985). https://doi.org/10.1007/BF00960758
Accepted:
Issue Date:
DOI: https://doi.org/10.1007/BF00960758