Introduction

The laboratory has always been given a central and distinctive role in scientific education and educators agree about the importance of practical skills and the rich benefits from using laboratory activities [1,2,3]. Some hands-on laboratory skills are fundamental (e.g., weighing, pipetting, preparation of solutions, measuring pH, laboratory safety, and filtration) and proper instruction of these is essential. Many of these basic operations are routinely performed in the laboratories and some will be critical to the confidence that can be placed in the final result [4]. For these reasons, different studies and organizations have found that mastery of basic laboratory skills is an important goal in the undergraduate curriculum [3, 5,6,7,8,9,10,11,12,13,14,15,16]. As the American Chemical Society (ACS) stated, lacking basic practical skills reduces opportunities “to develop the plethora of important scientific and transferable skills that a practical chemistry course should provide” [16].

Posner and Keele [17] argued that the learning of skills proceeds through three phases: the cognitive phase requiring instruction, the associative phase requiring practice and feedback, and the autonomy phase when the performance of the skill becomes almost automatic. Therefore, laboratory skills cannot be simply taught within a lesson; they need to be practiced and reinforced since the more practice you get, the more your skills will improve [18, 19]. As stated by di Trapani and Clarke [19], “development of laboratory skills and competencies (as with golf swings and putting) do not come naturally, they must be taught, practiced and consolidated before they can be performed with confidence and reproducibility.” The ACS also reported [16] that “students should be challenged to use appropriate laboratory skills and instrumentation to solve problems.”

The assessment of students’ hands-on techniques is complicated because it requires time and personnel resources during the laboratory period. Different authors have suggested that practical laboratory skills should be assessed in the laboratory by observing what the students are actually performing [20,21,22]. However, the most common situation is that students are usually assessed only on written lab reports or answers to examination questions [18, 21, 22], which makes it impossible to get a real picture of the laboratory skills acquired by students as the doing elements may not be translated into a text. As reported by Chabalengula et al. [21], when an experimental result written in the final report is correct, it may be assumed that the technique used to produce the result was performed correctly; however, when an experimental result is incorrect, the problem may lie either in the conceptual understanding or in the technique that produces the result. In this situation, students will place greater emphasis on the results of an exercise rather than the processes employed in achieving the goals and usually avoid learning hands-on skills [6, 18]. Students tend to value grades rather than learning and have the notion that the skills they are using in the laboratory are of lesser importance than the final written report for their final grades [6]. As a consequence, they go through the experiment without adequate stimulation and focus on more affective goals such as achieving satisfaction by finishing the lab quickly, with little regard for the skills they might be acquiring [3, 11, 12]. Kirton and Al-Ahmad [6] reported that when students fail to effectively learn practical skills at an early stage in training, it can hinder performance in subsequent modules.

Another problem is that it seems that some university programs have reduced the amount of laboratory work carried out by students during their training in the last decades (especially at the end of the last century) [3, 9], significantly affecting the practice of basic skills as once they are shown to students early on in the curriculum, they are taken for granted and little attention is usually paid to them later. As a result, undergraduate students typically only pass through the first of the three essential phases in skill acquisition that Posner and Keele [17] described. Focusing on this point, for Chemistry graduates, the ACS has fixed that “the certified graduate must have 400 h of laboratory experience beyond the introductory chemistry laboratory” [16], whereas the Royal Society of Chemistry (RSC) has stated that “Bachelor’s programmes students should typically complete at least 300 timetabled hours (exclusive of project work)” [5].

Over the last decade, in our faculty, we have noted two facts related to basic laboratory skills that are of concern. Firstly, some employers receiving final year undergraduate students for practical training in their companies have pointed out that students have a deficit in hands-on laboratory skills. Secondly, members of the Students Council have pointed out that some science students are signing up to further specialized hands-on training courses after graduation because they have seen that their laboratory skills are insufficient. This phenomenon has also been observed in different studies, which have reported that a lack of practice of basic skills is leading to many graduates having limitations in performing basic operations in the laboratory, and employers are becoming critical of graduates emerging from the university system without these basic skills [2, 6, 7, 9, 18, 23,24,25,26]. This view was expressed, for example, by the director of the National Technical Development Centre of the UK during the Lab Innovations 2019 conference in Birmingham: “the skills shortage within the UK and Europe is very serious.” An internship program in the UK developed for recent bioscience graduates focused on skill deficits found that 80% of the 1035 applicants sought laboratory-based projects, which was explained by a decline in the practical components of undergraduate degrees [20].

Despite employers usually being highly satisfied with the level of knowledge of new graduates, their degree of satisfaction with regards to basic skills is lower and many conclude that new graduate recruits require further training in order to make them effective employees [20, 27,28,29]. It should be mentioned that while the concerns of employers are mainly focused on soft skills, such as teamwork and communication skills, those studies that have included assessments of hands-on laboratory skills have reported deficits in these skills. These findings may lead us to conclude that science subjects should not only be based on core knowledge but also on experimental and hands-on laboratory skills.

Unfortunately, teachers and school administrators are often not well informed about what are considered best professional practices and they do not understand the rationale behind such consideration [30]. It has been reported that a philosophical and pedagogical impasse exists between what employers expect and educators believe or do [28], and that there has not been a coordinated mechanism to discuss skill needs between universities and industry in a collaborative way [20]. A survey of the Catalan Agency for the University System of Catalonia (AQU) in Spain found that 86% of employers had never been involved in the design and/or discussion of curricula and study programs [29]. A study from the BioHealth Capital Region (BHCR: Maryland, Virginia, and Washington, DC, USA) found that 91.7% of department chairs and program directors surveyed stated that their graduates were well trained and highly prepared for the workforce; however, employers did not think graduates were job-ready [28]. It was found that despite 75% of academic staff being able to identify the technical skills required by employers, the level of proficiency achieved by students was not known and academic staff members were unaware of whether skills were merely discussed in lectures or were evaluated through hands-on performance. Moreover, it was also found that graduates from many scientific majors did not have training in quality control, a fundamental topic for the industry in general and any laboratory performing quantitative analyses.

Most of these studies have based their conclusions on surveys answered by students, employers, and academic staff. However, no study has been based on the critical evaluation of students’ real experimental results to find the source of their experimental errors and assess whether these mistakes can be associated with basic laboratory skills, so determining whether or not these skills have been adequately acquired by students. In the present study, control quality methodologies have been applied to assess laboratory results obtained by students. The results obtained have been carefully evaluated in order to determine whether the mistakes found can really be attributed to a deficit in basic skills. For this purpose, quality controls were analyzed by students to assess whether systematic errors appeared in their quantitative results. When a bias was found, the source of mistakes was discussed with students in an attempt to find their origin and determine whether the lack of basic laboratory skills in graduates is a reality or a misguided assumption.

Methodology

In this study, the experimental results obtained by third-year Biotechnology and Chemistry students in two laboratory subjects have been assessed. In the case of the Biotechnology students, the study covers a period of eight academic years. Students had to take a laboratory subject on quantitative instrumental analysis based on the HPLC determination of the caffeine content in commercial soda drinks in which they had to analyze quality control samples mixed with their own samples. For the Chemistry students, they participated in an edition of an inter-comparison exercise organized by the Department of Analytical Chemistry at the University of Barcelona [31], in which they used a GC-FID method to determine the ethanol content of a beer certified material. In both cases, the corresponding method had previously been validated by the instructors before using it in the laboratory subjects.

Before starting the two laboratory subjects, the students had taken different practical subjects in earlier academic years. All the students had taken a 60-h laboratory course devoted to learning basic laboratory techniques at the beginning of their training, and either 240 h (Biotechnology) or 330 h (Chemistry) of other laboratory subjects. Given this prior preparation, it is reasonable to have expected that they would have a good level of basic practical skills.

Surprisingly, in the case of the Biotechnology students, it was found that this was the first time that they had been taught about quality control in the laboratory, despite having performed many other quantitative measurements in earlier courses.

The laboratory work done by students was relatively simple in both cases and only required the preparation of a stock solution, the preparation of calibration standards from the stock solution, the degassing of the samples, the measuring of a volume of samples, the dilution of the samples, and the instrumental chromatographic measurement of the standards and samples. This means that the basic laboratory skills required were just weighing, the preparation of solutions, and the manipulation of volumetric material: all routine skills that students should be fully competent in before finishing their studies.

In the case of the Biotechnology students, a seminar was introduced at the end of the laboratory sessions after the third year of this study to assess together with the students the quality of the results reported and discuss the possible sources of their non-conforming results [32]. A detailed assessment of the experimental procedure that the students followed was made using their quality control results, through the revision of their laboratory notebooks and by interviewing them. This methodology made it possible to identify the source of the systematic errors for most of the non-conforming results.

For the Chemistry students, once the final report was presented by the organizer of the exercise, the non-conforming results obtained by our students were evaluated just by reviewing their laboratory reports. It was not possible to discuss with students their specific laboratory work because the final report arrived at the end of the term after the subject had finished. The results obtained with the Chemistry students were only evaluated to find out whether the possible source of errors was specific to the Biotechnology students or if a more general trend seemed likely.

Results and discussion

The analysis of control samples during the laboratory experiments performed by the Biotechnology students and the use of control charts have helped to detect a percentage of incorrect results in preliminary students’ reports that ranged from 33 to 67% (Table 1). In the case of our Chemistry students, 24 certified samples were analyzed by the students and it was decided not to send four results (16.7%) for evaluation as they were considered to be outliers. Of the 20 results sent for evaluation in the inter-comparison exercise, two (10.0%) were classified as non-conforming results (z-score > 3) and a further three (15.0%) were classified as questionable results (2 < z-score < 3). These results were in accordance with the overall results obtained by the different universities participating in this exercise (mean values: 17.3% non-conforming and 9.2% questionable results). After receiving the final inter-comparison report, the four outliers were evaluated using the reference data reported in the report and it was confirmed that these results had indeed been outliers (z-score > 3). This means that of the total of 24 results obtained by our students, six were incorrect (25.0%) and three were questionable (12.5%). All the non-conforming percentages obtained must be considered inadequate taking into account the type of laboratory work being undertaken and the fact that students were close to the end of their university training and had already clocked up experiencing amounting to more than 300 laboratory hours.

Table 1 Percentage of non-conforming results reported by students every academic year in the quantitative analysis of quality controls
Full size table

It is reasonable to assume that the smaller percentage of non-conforming results obtained by the Chemistry students (25% vs. > 33%) can be attributed to their having had more previous experience in the laboratory. The Chemistry students not only had at least 390-h laboratory experience, as compared to 300 h in the case of the Biotechnology students, but they had also taken an Analytical Chemistry Laboratory subject in their second year in which they had specifically practiced some basic laboratory skills. Di Trapani et al. [19] reported that different students achieve experimental skills at different rates, and that while it was not difficult for students to achieve basic laboratory skills, usually not requiring more than 3–4 attempts to reach a minimum level of proficiency, repetition and practice were necessary.

During the first 3 years of the study for the Biotecnology students, the amount of non-conforming results (> 47%) suggested that there were significant problems with the level of laboratory skills that they had. Despite laboratory instructors having a strong suspicion of a skills deficit, which was observed in some cases during the laboratory sessions, it was not possible to assess the source of the errors as the students’ reports were handed in at the end of the term and no feedback was possible. Therefore, it was decided to introduce a seminar about control charts at the end of the laboratory sessions to discuss with students their results and to try to find the sources of their non-conforming results [32].

During these seminars, the most common failure was found to be related to the regression equations applied for the quantitative calculations, which was due to experimental mistakes in the preparation of the calibration standards. In quantitative analysis, a major objective is to evaluate the merits and limitations of a given method, including the chosen preparation technique [24]. Therefore, the preparation of stocks and standards is one of the most important laboratory skills as the quality of the results depends on the correctness of the calibration equation obtained. Of fifty-eight calibrations that the students prepared during these years, incorrect calibrations were only reported in two. The remaining 56 (96.6%) were considered to be correct by students, basically because the determination coefficients (R2) of these calibrations were > 0.99.

In the case of the Chemistry students, it was only possible to review the laboratory reports handed in by students at the end of the term. It was found that for the six non-conforming results, the calibration equations applied in the calculations were significantly different from those used by the students that had obtained correct results. The re-evaluation of these experimental results with a correct calibration equation solved the error in four cases as the recalculated concentrations yielded z-scores < 1.

A basic mistake made by the students of both studies at this point was that they had only used the R2 value to assess the validity of their calibrations. This misconception can be associated to the fact that in earlier courses it had been shown that the determination coefficient is the parameter to assess the goodness-of-fit of a linear calibration equation. When interviewing the Biotechnology students, they gave explanations such as “I was shown that when performing regression calculations, I should delete points that did not allow a correct value of R2 to be obtained” or “if you have an R2 < 0.99 you must delete those points that are separated from the model, until you reach a good value of R2.” In fact, in 13 (23.2%) of the reported calibrations, students had previously deleted at least one calibration point before doing their calculations without any more justification than their desire to improve the determination coefficient value. However, it was found in the seminars that only in four of these cases (30.8%) did the removal of the chosen standards in fact lead to a non-biased calibration equation.

Despite the importance of aspects such as quality control and method validation when dealing with quantitative analysis, this was the first time that Biotechnology students were introduced to these topics and were shown how to confirm the validity of a calibration by comparison with a previously verified calibration. In both studies, this was the first time that students had to measure quality control samples or a reference material to assess the correctness of the experimental results obtained in the laboratory. This finding tends to confirm the observation by Thompson et al. [28] that in many degree courses there is a lack of training given to students in quality control, which is an issue of fundamental importance in laboratories.

Systematic errors associated with calibration mistakes

During the seminars, the calibration equations obtained by the different groups were statistically evaluated and compared. Before starting the laboratory sessions each year, instructors performed a verification of the method and the calibration results obtained (mean of five calibrations, with rsd < 3% for slope values) were taken as the gold standard. As an example, Fig. 1a shows the calibration curves handed in by three groups of students at one of the seminars. Each group had prepared its own stock and standard solutions. All calibrations were considered correct by students in their preliminary reports because R2 > 0.998. However, when these calibrations were compared with the gold standard (mean slope(n=5) = 135419, sd = 1540, rsd = 1.1%), it was found that only the steeper calibration curve (A) can be considered as non-biased and that the other two curves (B and C) presented some bias. Linear regression and Bland–Altman plots were drawn to compare each pair of calibration curves [33]. In the case of linear regression analysis (Fig. 1b), significant deviations of the slopes from the line of equality (x = y) were found for curves B and C when these calibrations were compared to the non-biased curve A (A vs. B: slope = 0.92, confidence interval CI = 0.89–0.95; A vs. C: slope = 0.82, CI = 0.76–0.89), whereas the intercepts always included the zero value. These results confirmed that curves B and C provide proportional errors in the results obtained applying these regression equations. This can also be observed in the Bland–Altman plot (also called Tukey’s mean difference plot), where all the experimental results obtained by these three groups were evaluated together using calibrations A and B (Fig. 2). A clearly discernible pattern is seen, which reveals a proportional dependency between the differences obtained with calibration curves A and B and their averages.

Fig. 1
figure 1

 (a) Calibration curves handed in by three groups of students. These calibrations were obtained the same academic year and each group had prepared their own stock and standard solutions, which were measured with the same instrument following the same procedure. (b) Linear regression analysis for the comparison of calibrations B and C versus calibration A, which was found to be a non-biased calibration. The dotted line corresponds to the line of equality (x=y), which corresponds to equivalent (non-biased) calibrations. Specific data of the calibrations can be obtained from Supplementary Materials

Full size image
Fig. 2
figure 2

Plot of differences between results obtained applying calibrations A and B vs the mean of the two calibrations, with the representation of the limits of agreement from -1.96 sd to +1.96 sd (Bland-Altman plot or Tukey mean-difference plot). Specific numerical results of the calculation results for the averages and mean-differences can be obtained from Supplementary Materials

Full size image

To demonstrate the effect of using a biased calibration, students were asked to recalculate the results of their own control samples (n = 6 in this case, two for each group of students) with each one of the three regression equations, and to introduce the results in the same control chart (Fig. 3). It can be seen that only those results obtained applying the non-biased regression equation A were randomly distributed around the center line (expected value = 100 mg·l−1) and inside control limits. The relatively small difference obtained between curves A and B (Bland–Altman plot: mean difference − 0.57, sd = 0.88) may suggest that this variability could be acceptable and that calibration B could be considered as non-biased. However, although the majority (83%) of the results calculated applying regression equation B were inside control limits, all of them were located on the same side of the control chart, which confirmed a systematic bias in these results. In the case of curve C, all results were clearly out of control.

Fig. 3
figure 3

Control chart for the six quality control samples measured by three students’ groups. The first six results correspond to the results calculated after applying regression curve A shown in Figure 1, whereas the others were calculated using calibration curves B and C. (UCL: upper control limit; LCL: lower control limit; UWL: upper warning limit; LWL: lower warning limit)

Full size image

The same procedure was applied with all regression equations handed in by students every year. After the evaluation of all the calibrations reported by the Biotechnology students, it was found that only 31 of the 56 calibrations could be considered as being adequate, whereas the remaining 25 (44.6%) were biased calibrations that resulted in incorrect calibration equations being taken into account.

Over the 5 years in which this methodology was applied (from 2016/17 until 2020/21), a total of 102 control results were evaluated and 43 (42.2%) were found to be non-conforming. It was found that the majority of these non-conforming results had been obtained after performing calculations with biased calibration equations. Twenty-eight (65%) of the non-conforming results were simply solved after recalculation using a non-biased calibration equation and only 15 controls still gave non-conforming results (Table 1).

When the source of the bias detected in the calibrations was assessed, it was found that an incorrect application of basic laboratory skills during the preparation of the stock solution and standards was responsible in all cases. For example, in the situation described in Fig. 1, students of groups B and C had weighed an amount of the solid that differed significantly from the theoretically calculated value for the preparation of the stock solution (calculated = 0.1 g, amount weighed by students preparing calibration B was 0.0951 g, 4.9% variation). However, they had not taken into account this bias and had applied the theoretical value when preparing their standards without recalculating the concentration of the stock and standards. This means that they had prepared their standards with > 5% bias in their contents, which is excessive for quantitative analysis. In this specific situation, it was found that the bias was due to a mistake in the calculations, and that this could be solved simply by the recalculation of the concentration of their stock solution and standards.

In general, two types of mistakes related to basic skills were found in all of the biased calibrations. Firstly, the use of incorrect weighing procedures (e.g., using a top-loading balance instead of an analytical balance in the preparation of the stock solution, weighing the solid with conventional filter paper instead of using a weighing paper or an anti-static weighing boat, and weighing without checking the level indicator only to find out later that the balance was not leveled), and, secondly, the incorrect use of volumetric material (e.g., preparing the standards by mixing a volume of the stock solution measured with a graduated cylinder with a volume of the solvent also measured with a graduated cylinder, instead of using pipettes and volumetric flasks; using a beaker to measure the final volume of the solutions; and not shaking the solutions after mixing to ensure homogeneity).

The interviews with the Biotechnology students confirmed that they had only prepared stock solutions and standards in their first laboratory subject at the university. Thereafter, they found that stocks and standards had usually already been prepared for them at the beginning of laboratory sessions. This detail was confirmed by the teaching laboratory technicians, who explained that stock solutions were pre-prepared due to schedule limitations and so as to have time to perform other demonstrations during the laboratory sessions. These answers confirm that basic skills are in many situations only shown to students in preliminary stages of the curricula and are later simply taken for granted.

Students’ comments about the use of the determination coefficient led me to question whether this misconception might not also be present among laboratory instructors. Prichard [4] has already suggested that the little attention that is sometimes paid to basic laboratory skills may be associated with the fact that these skills are shown to students by people who themselves may have only been shown how to perform these skills on single occasions many years earlier. Other studies have also reported that sometimes instructors themselves may not have the necessary practical skills to teach them adequately [26]. For this reason, in the last year of the study, I contacted several instructors who had taught the same students in previous laboratory courses, showed them the calibration results obtained by the students, and asked for their opinion about the calibration equations. Most of the lecturers answered in the same way as the students, making the same mistake in the interpretation and evaluation of the calibration curves. Only those lecturers with some experience in industry or who had knowledge in quality control and method validation were able to answer correctly, but these amounted to < 10% of the laboratory instructors who had participated in the training of the students who were assessed. It was found that, unfortunately, in many scientific degrees, analytical calibrations using regression equations are simply considered a tool that is needed to calculate quantitative results, and no effort is made to explain and show students how to use and interpret calibration data correctly. Usually, students only receive explanations as to how to use a particular type of software to obtain the calibration parameters from experimental results.

Other sources of systematic errors

Finding the source of the errors in the case of the 15 non-conforming results that were not related to the use of biased calibration equations was much more complex because students did not usually note down all of the materials used in their lab notebooks. For this reason, from course 2017/18 onwards, students were reminded at the beginning of laboratory sessions that they had to record this information. Having this information proved helpful and made it possible to discuss with them how the material was manipulated. Despite it not being possible to find the sources of all the systematic errors, I was able to confirm that the most common mistake was in the selection and manipulation of the volumetric material. It was, for example, common to manipulate incorrectly class A volumetric glass bulb pipettes, which have two marks, as students often only took the upper mark into account.

For the Chemistry students, the two cases in which the use of a correct calibration did not solve the error, it was found that the ratio obtained after the correction by the internal standard (IS) used in the method gave values that were significantly different than those of the other students, when they should have been very similar as all students had theoretically prepared the IS at the same concentration and had added the same volume of IS to the same volume of the reference material. Despite it not being possible to verify this assumption directly with the students, it seems that the error must be attributed to an incorrect volume of IS being added during the preparation of the sample or to an incorrect preparation of the IS solution. Unfortunately, these two steps were not detailed in their laboratory reports.

Conclusions

It has been shown that using quality control methodologies in laboratory lessons helps to find whether students have worked properly in the laboratory. Moreover, when a bias is detected, it is possible to find the source of the systematic error in most situations. In this study, quality control methodologies have been applied not only to find the source of biases but also to try to explain why these errors took place. The results obtained in this study clearly show that many of our undergraduate students still have serious limitations with basic laboratory skills when they are very close to graduating. However, the most significant conclusion of this study is that the application of these methodologies can reveal whether or not science students have a deficit in their basic laboratory skills.

Unfortunately, the finding of a lack of basic laboratory skills in our students confirms the perception of many employers about the deficit of these skills. It is clear that instead of continuing with current procedures unamended, our university administrators and teaching staff should address this problem head-on.

The type of basic laboratory mistakes that are reported in this study is something that might be expected from first-year students during their earliest laboratory sessions when being introduced to basic laboratory skills. However, such mistakes should be considered unacceptable in third-year students.

Although the methodology described here can be considered as equivalent to student assessment based on written reports, allowing instructors to gauge errors in data collection [18], as was the case with the calibration equation errors, it had the additional advantage of revealing the source of most of the errors detected. However, this methodology only helps to find out the problem and does not in itself correct the lack of basic laboratory skills that have been found. The most productive way to solve the problem that has been detected is to increase the amount of work that is done in the laboratory. This would require a modification to current laboratory procedures so that students can practice and reinforce basic lab skills in all of their laboratory subjects, not only in a preliminary subject during their first year of university. Such a simple change would help to reinforce these skills, so resolving the problem and meeting some of the needs of employers [20].

There are two ways in which this change can be made. Firstly, by reducing the amount of time spent on more complex demonstrations performed in each laboratory session to allow time to work on basic skills. However, this is a point that many university lecturers will disagree with as it means a reduction in the practice of core scientific knowledge, which they usually consider to be the main objective of their teaching. Secondly, by increasing the time scheduled for the laboratory lessons. In this case, the university administrators can be expected to pose objections as this will require extra funding and more staff for these subjects. Nevertheless, one or other of these solutions, or a combination of the two, must be adopted if we are to tackle the deficit in basic skills that have been observed in our students and the perception by employers that university training is currently inadequate.

A second problem observed in this study is that once students have consolidated the acquisition of a misconception and applied it routinely over a long period of time, it is very difficult to overcome and correct it. A typical question from students when trying to solve mistakes in the use of the determination coefficient for analytical calibrations was “why should I believe you when all the previous lecturers have taught me this topic differently?” Such protests reinforce the importance of good training in basic laboratory skills right from the very outset of students’ university educations.