The fundamental issue behind the Receiver Operator Characteristic (ROC) analysis is simple: the brain contains no photoreceptors, and thus has no direct vision of the outside world—instead, it contains neurons that fire a certain number of times. The brain then has to analyze what happens in the outside world based on the firing of its neurons. The ROC analysis calculates how well an imaginary observer of the firing activity of a given neuron could be at detecting a particular target stimulus. In the case at hand, this ideal observer has to decide on each trial if a particular action (his target action or best action) was performed. He has to take this perceptual decision based on the number of spikes produced by an audiovisual mirror neuron on each individual trial.
The decision rule used by the imaginary observer of the neural activity is simple: he decides on a threshold spike-count value theta, and compares the count x i on a given trial i with this threshold. If x i ≥theta, the observer responds that his target action occurred. If x i <theta, he reports that another action must have been performed. The decision of the observer is then compared with what action was really performed on that trial.
The working of the ROC analysis can be best explained by applying the method to two different neurons: one, which firing has nothing to do with what action the experimenter performed in front of the monkey, and one—an audiovisual mirror neuron—that fires more when its best action was performed.
Figure 2A shows the spike-count distribution for the audiovisual mirror neuron. Note how the histogram when the experimenter performed the best action (top of Fig. 2A) is shifted rightwards compared with the lower histogram when the less effective action was performed (bottom). This shift means that the neuron is more likely to produce large spike-counts when the best action is performed. In the example of Fig. 2A, the observer places his theta at an intermediate value (e.g., 20, dotted vertical line in Fig. 2A). Not knowing what action was really performed by the experimenter, the observer applies his decision rule blindly to both conditions. He reports the best action when the spike count is larger than or equal to theta (i.e., at the right of his threshold) in both cases. In our example, he reports the best action in nine out of the ten best action trials (i.e., when the experimenter really performed the best action), and in two of the ten less effective action trials. The former 9/10 are correct decisions, called 'hits' (black bars in Fig. 2A), and his hit rate is thus 90%. The latter 2/10 are errors, called false alarms (gray bars), and his false alarm rate is thus 20%. The hit and false alarm rate depend not only on the response of the neuron, but also on the theta used by the observer: Placing theta very low (i.e., moving the dotted line leftwards), the observer would always report the best action, and both the hit and false alarm rate would approach 1. Choosing very large theta, he would never report the best action, and both the rates will approach 0. Figure 2B illustrates the relationship between the hit rate and the false alarm rate for this audiovisual mirror neuron as a function of theta. This curve, called the receiver operator characteristic curve, is obtained by testing all the possible theta values, plotting the hit and false alarm rates for each theta, and connecting all the points. For this neuron, this curve is very far away from the dotted diagonal, and the surface under the curve is close to 1 (0.96). What does this large surface mean? Given that the two histograms overlap very little, if the observer starts at a theta=42 (bottom left of Fig. 2B), and reduces theta until 23, the observer only responds with best action for best action trials, i.e., the criterion does not include trials of the lower histogram in Fig. 2A, and thus the hit rate increases without increasing the false alarm rate. Only if the observer reduces theta below 23 will he respond with best action also in trials where the worst action was really performed. The curve thus rises vertically until the 80% hit rate, and then moves almost horizontally towards a false alarm of 100%, covering almost all the surface in the box. This is symptomatic for cases where the neuron accurately discriminates between the two types of actions.
In contrast to the audiovisual mirror neuron, let us now consider an imaginary neuron that does not discriminate between the two actions. Figure 2C illustrates the spike counts for this neuron. Note how much overlap there is between the two histograms. Under such conditions, placing theta at 20 means that the observer will respond with best action 6/10 times when the best action and 5/10 times when the worst action was really performed. The result is that the hit and false alarm rate (60 and 50% respectively) are almost equal. Figure 1D illustrates the ROC curve in this case. The curve remains very close to the diagonal, meaning that a decrease of theta increases the hit rate, but at the cost of equally increasing the false alarm rate. The observer is essentially randomly guessing: the spike count gives him no information about what action was performed. The surface under the curve will be close to 0.5 (here 0.54), which is symptomatic for the cases when the activity of the neuron is unrelated to the action performed by the experimenter.
The surface under the ROC curve is thus an indication of the proportion of correct decisions that the observer makes, taking all the thetas into account. From the two examples, it is intuitive that a surface close to 0.5 represents random performance while a surface close to 1 indicates that the observer is very good at telling what action was performed. This surface then gives us valuable information about what function the neuron might have in the brain. If the imaginary observer is very good at telling what action was performed (large surface under the ROC curve), then the brain too could use this neuron to discriminate between the two actions. The neuron could thus participate in the perception of the actions. If the observer is very poor at telling the difference between the two actions using the spike count of this neuron, so would the brain be, and the neuron therefore is unlikely to be involved in the perception of the actions.