In discussions of teaching barriers in the social sciences, it is not uncommon to hear professors complain that it is difficult for students to understand the complex relationship between theory and research methodology and the way in which both are used to develop an empirical understanding of substantive topics. There are several possible explanations for this complaint. For instance, when students are first introduced to social science topics, for many it seems like common sense. Consequently, students may come to believe that the knowledge base in these fields is not really based on science. Added to this is the fact that students are generally not often introduced to the complex scientific methodologies utilized by social scientists until they take the research courses required of majors. Unfortunately, this means that non-majors (and sometimes minors) are never actually taught about these complexities; and majors are often not exposed to them until late in their academic training. One possible way to address this problem is to introduce students to the process of social science research early in their academic experience.

The practice of teaching the scientific method to sociology students is not a new one. In sociology, faculty members have recommended teaching students research skills early in their academic training (Clifton, 1976; Conklin & Robinson, 1985; Davis, 1983; Ziner, 1994). Such additions to the curriculum arguably could improve the sociology major by helping to cultivate the sociological perspective, an idea outlined in the American Sociological Association's recommendations found in Liberal Learning and the Sociology Major (American Sociological Association, 1991). This same strategy could be used to improve the curricula of other social sciences. This study offers an evaluation of the effect of implementing data analysis in a large introductory-level sociology course.

Background and Significance

Curricular Changes in Sociology

In 1980, Goldsmid and Wilson voiced concern about the way that undergraduate sociology students learn sociology. In particular, they were concerned with how research is isolated from teaching in the undergraduate curriculum, leading to a disjunction between theory, research, and substantive coursework. Faculty members continued to discuss the problem and more than 10 years ago, the Association of American Colleges and Universities (formerly the Association of American Colleges) sponsored a 3-year project to examine the undergraduate curricula of programs within 13 disciplines, including sociology (see Howery, 2001). As part of that effort, the Sociology Task Force developed recommendations for improving the undergraduate experience for sociology students. One of the recommendations from that task force focused upon the research training that emerged as a common denominator in most undergraduate sociology programs. Students in undergraduate sociology programs typically take a course or sequence of courses that focus upon research methodology. These courses, however, are often taken late in the program; and students fail to grasp the intricate relationship between theory, research, and substantive coursework. The report and subsequent discussion within the discipline, such as commentary made by McKinney, Saxe, and Cobb (1998) in the Hans O. Mauksch Award presentation, have brought the need for more rigorous and extensive research training to the forefront of education in the discipline of sociology (Howery, 2001).

Although sociology faculty members nation-wide recognize the importance of offering students an integrated curriculum, many programs have found doing so to be difficult. Sociology as a discipline tends to be confronted with hurdles to the development of a cohesive curriculum. In particular, students are rarely exposed to the discipline prior to entering the university or college setting, and therefore, they lack a sociological foundation. This is exacerbated by the lack of extensive sequencing in the sociology curriculum that allows students to pick and chose their courses and the order in which they take them quite freely, thereby making it more difficult to build a hierarchy of skills and knowledge. To complicate matters, sociology majors frequently do not declare their major until they are in the last years of their program, and many of the students enrolled even in upper-division courses are non-majors (Howery, 2001). The result of these barriers is that students often fail to develop a sense of connection between their specialty area courses and the methods and theory courses.

Perhaps in response to the issues raised by the report from the Association of American Colleges and Universities as well as issues discussed by sociologists across the country, the American Sociological Association (ASA) has increased its efforts to encourage departments to incorporate formalized efforts to include “scientific literacy” as an integral part of undergraduate sociology curricula across the country. These efforts have been supported through the organization's development of resources and programs designed to assist faculty and departments to do this. Part of these efforts have included the Minority Opportunities through School Transformation (MOST) program, a project funded by the Ford Foundation to assist 18 departments in transforming their programs to become more effective for minority students, and ultimately for all students (Howery, 2001).

Recently, using the information gained from the MOST program, the ASA teamed with colleagues at the Social Science Data Analysis Network (SSDAN) at the University of Michigan. Through this collaboration a proposal was developed and funded by the National Science Foundation to provide training at the departmental level to 12 departments over 2 years. The training and support were designed to encourage and support departments in incorporating the ASA's objective of “scientific literacy as a key curricular goal … early and often” (Howery, 2001; p. 2). The program that was developed was named Integrating Data Analysis into the Curriculum (IDA). During the summers of 2002 and 2003, it provided training to a majority of the faculty in the 12 participating departments on ways to include data analytic components in undergraduate courses beginning with introductory courses. Ultimately, the investigators at SSDAN and ASA hope to disseminate these strategies to other departments and social science disciplines. They have designed these methods to improve the quantitative literacy of the undergraduate sociology student. However, these measures may also benefit students in other ways as well.

Changes to the Traditional Classroom Idea of Teaching

Over the past 25 years, higher education in the United States has been reforming itself to accommodate a more non-traditional student population. This population is placing increasing demands upon their academic institutions to provide more flexibility while at the same time providing skills that are marketable to employers. Concurrently, there has been increased pressure on post-secondary institutions to provide this education with budgets that have not kept up with the rising costs of providing such an education. These trends, combined with increased student diversity, leave “educators searching for new and effective teaching modalities that are time- and cost-efficient” (Dietz, 2002; p. 80). The task is daunting—to provide an increasingly diverse and larger student body with marketable skills while at the same time reducing costs and demonstrating effectiveness.

Astin (1984, 1996) espoused activities that link the students to the academic community, faculty, and their peers as powerful forces that increase student investment in their education and enhance learning. Universities and colleges, programs, and educators are turning to technology to accomplish the task. Through the use of virtual learning and interactive activities, educators are finding that they can actually increase the quantity and quality of student interactions (Jaffee, 1997). While virtual learning exercises may be particularly suitable for the social sciences, Brooks (1997) has argued that the social sciences have lagged behind many other disciplines in adopting technology in the classroom. In a joint effort, the ASA and the SSDAN are attempting to increase the use of technology while simultaneously providing students with skills that they will be able to use in their continued studies and the workplace by encouraging the use of data analytic strategies throughout the undergraduate curriculum. Thus, the goal is to spread this type of teaching pedagogy to social sciences as a whole.

The Training

In June 2002, eight members of the sociology faculty at the University of Central Florida attended the first IDA training workshop held at the University of Michigan. The workshop was conducted over 5 days by members of the SSDAN faculty and representatives and affiliates of the ASA. Subsequently, the first IDA integration efforts at the University took place in courses offered during the Summer of 2002. In addition, the department held a day-long meeting of all sociology faculty members to discuss formalizing the effort. They remain supportive of and committed to these efforts and have seen some positive outcomes (as well as some negative outcomes) for the students, the program, and ultimately the department's graduate program. Specifically, faculty members recognize that by introducing students to research and analysis earlier in their training, they are able to provide more advanced training in the research sequence for the majors. Moreover, they recognize that this curricular change has enabled the students to develop a better understanding of the ways in which theory and methodology are connected. At the same time, the faculty members recognize that courses that have a heavy research component may not be attractive to less motivated students and especially to non-majors. There has been a small decline in student enrollment in some courses. However, the number of undergraduate sociology majors has increased, and the instructors who teach the upper-division courses that include a heavy research component do not report that their teaching evaluations have suffered as a result. In some instances, incorporating the modules may increase the work load of the faculty member, especially in the early stages of creating the modules for the courses.

Data analysis modules taught by faculty in the program include analyzing census data, crime data, aging data, and General Social Survey data, as well as other types and sources of data. Because faculty members have the autonomy to develop their own modules in their own courses, there is no mandated consistency. Consequently, this article offers an examination of the effects of integrating data analysis modules into a single introductory sociology course during the fall of 2004. This evaluation was conducted by the instructor of record for the course, who was one of the original individuals trained at the ASA workshop. Other faculty members in the department as well as others in departments around the country are conducting similar assessments of their own courses and departmental-wide efforts, and the ASA is conducting an evaluation of the program itself. Although the funding for formal training is no longer available, smaller workshops are being held at the national meetings and by those who were trained in the workshops at regional meetings. Furthermore, the author of this paper has conducted workshops at the University of Central Florida for faculty members in other disciplines during workshops sponsored by our Faculty Center for Teaching and Learning.

Methods

Approximately, 250 students enrolled in the author's fall 2004 Introduction to Sociology course. The course was designed such that students were allowed to choose from a list of exams and assignments those components that would comprise their final grades. Students could choose to complete three exams and two on-line data analysis assignments, two exams and six such assignments, or one exam and 10 assignments. The on-line data analysis assignments required students to go to the General Social Survey data website (http://webapp.icpsr.umich.edu/GSS/) and to complete assignments outlined in a custom published textbook developed by the course instructor/author. They conducted a number of different types of analyses ranging from simple frequency distributions with recoding to ANOVAs to multiple regressions and were then given instructions on how to produce the output and how to interpret the tables and statistical output and what the output meant. They were required to complete on-line exercises, to apply theories to the output, and to submit the assignments on-line.

Using data gathered through an on-line administration of a pre-test and post-test questionnaire, this study provides the results of an evaluation of changes in knowledge about basic research methodologies and student perceptions of what they learned, as well as the value of the data analysis modules. It also measured the student's overall satisfaction with the experience and their intentions and desires about future experiences with sociology and research. The pre-test was administered in the first week of classes prior to the introduction of the chapter on research methodology. The post-test was administered during the last week of classes. The purpose of the study was to examine the change from pre-test to post-test in knowledge on research and to evaluate the predictors of that change. These procedures had been submitted to and approved by the University's Human Subjects Review Board.

My hypotheses were as follows:

  • There will be a statistically significant difference between the pre-test and post-test score.

  • Those students who completed 10 assignments will show a greater improvement on post-test than those who completed only six assignments, and those who completed six assignments will show greater improvement than those who completed only two assignments.

  • There will be a positive relationship between the number of assignments completed and interest in participating in sociological research projects in the future.

  • There will be a positive relationship between the number of assignments completed and interest in participating in research projects in general.

  • There will be a positive relationship between the number of assignments completed and interest in taking additional sociology and IDA courses.

  • Those students who completed more assignments will be more likely to report a greater interest in majoring in sociology.

  • There will be a positive relationship between the number of assignments completed and interest in pursuing a career in sociology.

  • There will be a positive relationship between the number of assignments completed and level of agreement with statements demonstrating knowledge of the empirical nature of sociology.

Measures

The researcher developed a series of measures to examine student perceptions of what they learned from the data analysis modules as well as how well they liked the modules and whether or not they planned to take more courses that utilized these types of teaching tools. Most measures were in a Likert-format, with four possible categories ranging from strongly disagree (1) to strongly agree (4). The pre-test and post-test were administered on-line through WebCT. Specific items are detailed below.

Demographic Variables

Demographic variables included sex/gender, classification (freshman, sophomore, junior, senior, or other), whether or not the student had taken a sociology course prior to the current course (such as in high school, junior college, or perhaps retaking the introductory courses after failing or doing poorly in it, or even having taken one of a few sociology courses that do not require the introductory course as a prerequisite), the number of sociology courses that students had taken before the current course, whether or not the student had taken a research methods course before, the student's major, and the student's minor.

Research Knowledge Variables

A series of 13 multiple–choice questions designed to test student knowledge of sociology and research methodology was given during the pre-test as well as the post-test and included the following questions:

  • How do sociologists know what they know?

  • Which of the following is the best definition for the term “variable”?

  • Assume that Professor Smith wants to learn about the study habits of the university's students. Which of the following illustrates a variable that Professor Smith might study in this research project?

  • What is a hypothesis?

  • In the following hypothesis, identify the independent variable. “The rate at which an individual can run one mile is associated with his/her age.”

  • What is a correlation?

  • Using the table below, what percentage of the sample is female?

  • Using the following list of numbers, identify the mode.

  • Using the following list of numbers, identify the median.

  • Using the following list of numbers, identify the mean.

  • Using the table below, is there a statistically significant difference between the two groups (males and females)?

  • What is meant by the term “statistically significant”?

  • What is meant by the term “sample” as it relates to research?

Satisfaction and Perception Variables

During the post-test, students were asked, “Would you consider taking a class that uses data analysis modules in the future?” Students were asked to indicate their answer by marking either “yes” or “no”. In addition, there was a series of Likert-format items designed to measure their perceptions about the data analysis modules. Students were asked to indicate the degree to which they agree or disagree with the following items:

Student Understanding Items

  • I hated the data analysis modules.

  • The data analysis modules were not helpful.

  • The data analysis modules helped me to better understand the scientific nature of sociology.

  • The data analysis modules helped me to better understand the relationship between data and theory.

  • The data analysis modules helped me to better understand where sociologists get the information that they use to draw conclusions about the social world.

  • I do not see that the data analysis modules were helpful at all in understanding sociology.

  • Because of the data analysis modules, I have begun to question some of the statistics that I hear from the media, etc.

  • Because of the data analysis modules, I have begun to reconsider what I thought was true about society.

Effort items

  • The data analysis modules were too difficult.

  • The data analysis modules were not challenging enough.

Student Interest Items

  • I would take other courses that include data analysis modules in the future (may include other disciplines).

  • The data analysis modules have led me to consider a career in sociology.

  • The data analysis modules have led me to consider majoring or minoring in sociology.

  • I would like to participate in research projects with a faculty member (not necessarily sociology) in the future because of the data analysis modules.

  • I would like to participate in a sociological research project in the future because of the data analysis modules.

  • I will probably sign up to take another sociology course because of the data analysis modules.

  • I will probably sign up to take another course that uses data analysis modules because of the data analysis modules.

Results

Although a total of 245 students initially enrolled in the course, only those students who completed the course and both the pre-test and post-test were included in this analysis (N=164). Nine students withdrew from the course and an additional 72 chose to not complete either the pre-test or the post-test or both. Of the students who participated in the pre- and post-tests 64% were females, and 74% were first-year students. Most (84%) had never before taken a sociology course. Of those who had, they may have taken a course in high school or another college-level course. Interestingly, 14% of the respondents reported that they had taken some type of research course prior to this course. Details about the types of research methods courses or sociology courses that these students may have taken were unavailable. The mean pre-test score was 68. The mean post-test score at the end of the course was 79, representing a mean change of 11 points. The median number of days present during the semester was 27. More specific details about the sample characteristics are provided in Table I.

Table I Sample Characteristics
Full size table

Table II provides detailed frequency distributions for these variables by the assignments completed groupings. Each student was given the option of choosing to take three exams and complete two on-line data analysis assignments, two exams and six assignments, or one exam and 10 assignments. The majority of students, (64%) chose to complete 10 assignments while 26% chose to complete six assignments, and the remaining chose to complete two assignments. More than half (62%) of the students indicated that they would consider taking a

Table II Student Perception and Intention Measures
Full size table

sociology class that used data analysis modules in the future. Few (17%) of the students reported thinking that the modules were not challenging enough while slightly less than one-third (29%) reported that the modules were too difficult. The majority of the students reported that the modules enabled them to understand the relationship between research methodology and theory (85%), to re-evaluate their common sense understandings of the social world, and to utilize a more scientific-based inquiry to understanding of the world (88%). Perhaps most surprisingly, more than one-third of the students reported that they would like to participate in a sociology research project (38%), and more than one-half reported that they would like to participate in a research project of some type (52%).

A series of paired samples t-tests was conducted to evaluate change over time in the scores on the pre- and post-tests. The results indicated that the mean score on the post-test for the class (M=78.5, SD=9.6) was significantly higher than the mean score on the pre-test (M=68.2, SD=12.4). Similarly, mean scores on the post-tests were all significantly higher than mean scores on the pre-tests for each of the three sub-groups of students, although the largest improvement (11.6 points) emerged for the group that completed only two assignments and the smallest improvement (8.1 points) emerged for the group that completed six assignments. A series of one-way ANOVAs was conducted to evaluate the relationship between the number of assignments completed and post-test score and change in post-test score as well as satisfaction and future intentions. The ANOVAs to measure differences in post-test score and change in post-test score by number of assignments completed showed no statistically significant differences. Likewise, no statistically significant difference emerged for the questions measuring intentions or knowledge.

Conclusions and Discussion

As hypothesized, and hoped, there was a mean increase in the mean score on post-test score. While the increases in the scores were not substantial, this should not be interpreted to mean that such activities should not be included as part of a sociology curriculum, or another other social science curriculum. One should note that there was no incentive given to students to perform well on the post-test, so they may not have done as well as they could have. In addition, one should also recall that the mean score of pre-test was 68, suggesting that many of the students already knew much of the information tested on the pre- and post-tests. Future research should incorporate more difficult questions testing knowledge of research methodology and sociology to be able to better test improvement.

The second hypothesis was not supported. Students who completed the most exercises did not show the greatest improvement. There are several possible explanations for this finding. There is an obvious self-selection bias in who elects to complete 10 assignments versus those who elected to complete two. Thus, one might actually expect to see less change among those who elected to complete more assignments because those who elected to complete more assignments may have been those students who were more familiar with statistics or research or who felt more confident in completing these assignments from the very beginning. Consequently, they may not have had as much room to improve. Furthermore, it is also possible that the students learned much of the material covered on the post-test from the classroom activities at the same time that they learned it from the data analysis assignments. Thus, it is impossible to completely disentangle the effects of the modules from the effects of the class itself. It would be unethical to attempt to measure the effects of the modules themselves while completely ignoring such important topics as research methodology in a sociology course.

Similarly, the lack of statistically significant difference in the knowledge and intention variables should not be interpreted necessarily to mean that the modules have no impact or that having several modules versus few modules has no impact. One should also keep in mind that the vast majority of the students chose to complete 10 modules resulting in small numbers of students in the two assignment and six assignment categories. Had the numbers of students in these categories been more evenly distributed the answers may not have been skewed as they were.

It is interesting, too, to note the patterns that emerged in Table II. I had hypothesized that those students who completed more assignments would express greater levels of agreement on those items measuring student understanding and interest than those who completed fewer. This was not the case, however. Indeed, no statistically significant differences were found and the opposite pattern emerged. There are several plausible explanations for this. Very nearly every student indicated agreement on these items, so the message that sociology is a science seems to have been learned. Nevertheless, it is interesting that those who completed more exercises seemed to be somewhat, although not significantly, less likely to report that they view sociology as a science. It is possible that any exposure may have the desired effect of introducing students to social science so multiple exercises are not necessary for teaching the basic notions of empiricism. I am left wondering, however, about the effect of one component of the class not measured in the data that deserves mentioning here. The students were allowed to complete these assignments very nearly up until the last day of class. For those students who completed only two assignments, the issue of procrastination likely proved to be a much less daunting task than it was for those students who had chosen to complete 10 assignments. Students who waited to the last minute to complete their assignments may have been frustrated when they took their post-test, and it may have reflected in their post-test responses. Furthermore, I must also note that these students were also confronted with the inconvenience of cleaning up after one hurricane and the irritation of two additional hurricanes during the semester in which they took this course. Thus, the problem of procrastination may have been even worse due to the problems associated with that, especially for those who chose to complete 10 assignments. Unfortunately, however, this is beyond the scope of this study. The issue of procrastination can be potentially addressed in future research projects.

While attention has been devoted to the not-so-noteworthy results, perhaps the most interesting findings of all can be found in Table II. These results provide ample support for the incorporation of data analysis exercises into undergraduate sociology courses. Students’ answers on questions about their experiences with these modules were enlightening, and, even hopeful. As mentioned previously, many suggest that integrating research into the undergraduate classroom is challenging because it presents so many problems. Reports of student frustration, expense, and inefficiency may have prevented such efforts in the past. These results suggest otherwise. These students reported that not only were the modules helpful, but many reported that they would consider taking more classes like them. More exciting, they reported interest in participating in actual research projects. These results do not suggest that students were overly frustrated with the modules. Moreover, because the modules were implemented in a fashion which made them free to the department and to the student using them and were accessible whenever it was convenient for the student. Plus, because there were no human subjects involved, there was no need to deal with issues of institutional review boards and other time-consuming and inefficient activities. Finally, through the use of Web-CT, the module assignments were constructed to be submitted on-line and automatically graded. Thus, it was possible to manage assignments, even in a large enrollment course.

While students should become familiar with other aspects of research such as qualitative research and independent research projects, the use of publicly available datasets that can be analyzed on-line does represent one possible option for faculty and programs to introduce the empirical nature of quantitative literacy to students. Because the data used are in the public domain, it is possible to encourage even undergraduate students to tackle empirical research. Moreover, the General Social Survey data covers topics that are of interest to several disciplines and could be used by such programs as political science, health science programs, criminal justice, policy programs, education programs and others.

Because the data had often been gathered via large-scale, federally funded, nationally representative projects, the instructor can demonstrate attitudes from a national perspective. For a handful of extremely motivated students who become familiar with the data and who want to use them to complete independent, publishable research projects to make themselves more competitive for graduate school, they may do so. While on the surface this may seem unrealistic, the IDA program has become a stepping stone for doing just that at our University. Students who have first learned data analysis in introductory sociology courses and who have continued to participate in these types of analyses through additional courses have progressed to presenting papers at professional meetings and to publishing in journals designed specifically for student authors.

In sum, while the investment of time and energy to develop learning activities using these publicly available data may seem daunting initially, these data demonstrate that the benefits can outweigh the costs for the student and ultimately for the instructor and program in the long run, even in a large-enrollment course.