An abnormality or anomalyFootnote 1 can be defined as the event or phenomenon that deviates from a common rule, type, arrangement and/or form. Without making any qualitative judgement and/or taking any posture on the validity of the sense of abnormality,Footnote 2 it is under no doubt that abnormalities represent entities of interest in several health, manufacturing, leisure and security scenarios: whether to apply corrective actions (e.g., on industrial applications, surveillance) and preventive treatments (e.g., in health, security), to perform in-depth exploration studies (e.g., sciences, like genomics, astrophysics, bio medicine, mines research), to invest resources (e.g., oil research, advertising campaigns) or to simply identify abnormalities and track their progress (e.g., patient monitoring) [2, 4, 5, 7]. This applies for computer vision and its applications, where abnormalities have been the implicit motivation and subject of research in many studies and research fields.

The subject of anomalies is of particular interest to the looking at people (LaP) field. LaP focuses on the visual analysis of human behaviour, action/gesture/activity recognition, facial analysis and pose estimation. Since humans are involved in LaP tasks, it is clear that the study of abnormalities is critical in this context. This is particularly true in LaP applications like health and security, where uncommon patterns/behaviours are the most interesting ones. Despite its relevance, it is remarkable that so far the study of abnormalities in LaP, and more generally in computer vision, has not been characterised. Questions such as what makes an event/situation/behaviour abnormal, or how abnormal behaviour in images or video has been analysed, cannot be answered without reading a dozen of articles.

We edited a special issue that aims to compile the latest research human abnormal behavioural analysis. The focus of this issue is on all forms of behaviours and all kinds of atypical activities, hence comprising a broad scope across several fields of study. Including computer vision and machine learning, and benefiting from others are partially related to the issue in the aforementioned multi-disciplinary context. We received a total of twenty-two submissions that were subject to a strict reviewing process. Despite the high quality of all of the submissions, we only accepted fourteen outstanding contributions. In the following, we briefly summarise these contributions, grouped in thematic subjects.

Action and gesture recognition Abnormalities have been widely studied in the context of action and gesture recognition. Several contributions on this topic are part of the special issue. Li et al. describe a spatiotemporal attention-based ResC3D model for large-scale gesture recognition with an emphasis on abnormal gesture recognition from RGBD videos [9]. They conducted an experimental evaluation using the Chalearn IsoGD Dataset [15] showing the effectiveness of the attention model. In the same line, Lin et al., in their Abnormal Gesture Recognition Based on Multi-Model Fusion Strategy article [10], proposed a novel refined fused model that combines a masked (RGB and Depth) ResC3D networks with a skeleton LSTM for abnormal gesture recognition in RGBD videos. Based on their experimental results, their proposed method could distinguish the abnormal gesture samples effectively achieving state-of-the-art performance in the IsoGD dataset. The IsoGD dataset is becoming a useful resource for studying abnormal behaviour [14, 15].

On the other hand, Raman et al. focused on predicting the walking directions of pedestrians, a topic that can have a direct impact on abnormal behavioural analysis. In their work entitled Beyond Estimating Discrete Directions of Walk: A Fuzzy Approach [11], the authors proposed a type 1 fuzzy approach over apposite inter- and intra-frame locomotion features of pedestrian to predict eight different directions of walk. Their exhaustive experimental results showed that their proposed methodology overcome the existing state-of-the-art techniques in direction of walk prediction.

Instead of working with videos, Hu et al. focused on still images to study the problem of abnormal behavioural detection in the driving context [6]. In their Driving behavior Recognition from Still Images by Using Multi-stream Fusion CNN article, Hu et al. used multi-stream CNNs to extract multi-scale features by filtering images with receptive fields of different kernel sizes and then used fusion of the multi-scale information to generate their final prediction. The conducted experiments indicated that their method achieves the significant performance improvements compared to the state of the art. Likewise, Ding et al. dealt with the problem of posture assessment, a topic that also can have a direct impact on abnormal behavioural analysis [3]. In their A Real-Time Webcam-Based Method for Assessing Upper-Body Postures paper, Ding et al. presented a vision-based method for real-time assessment of upper-body postures of a subject whose lower body is not visible to the camera. They introduced a database of upper-body postures that cover the various scenarios when a subject is sitting in front of a desk.

Applications with potential societal impact As previously mentioned, abnormalities can be found in any domain and their detection could be used develop systems that can have an impact on society. A couple of submissions in this line are part of the special issue.

On the one hand, Sun et al. report a study for automatically monitoring infants in the context of health clinics [13]. In their paper entitled Video-Based Discomfort Detection for Infants, Sun et al. proposed a video-based method for automated detection of discomfort/pain of newborn infants based on a two-phase classification workflow, where Phase 1 was subject-independent (considering generic visual descriptors) and Phase 2 was subject-dependent (including subject templates). Experimental results in a database with 12 infants revealed almost perfect performance for the subject-dependent setting with lower performance for the subject-independent phase. On the other hand, Lashkar et al. introduced, in their paper A motion-based waveform for the detection of breathing difficulties during sleep [8], a method based on motion to detect breathing movements of sleeping patients from infrared videos. The detected movements are mapped into a waveform where different sleeping–breathing patterns can be detected. Experimental results showed the effectiveness of their proposed technique, motivating further research on automatic recognition of breathing events.

In general terms, both previous papers addressed quite novel scenarios in applications that could have a direct impact on society. We foresee these works will motivate further research on abnormal behavioural detection in health.

Surveillance applications Perhaps the most common application for abnormality detection methodologies from visual information is surveillance/security. Because of this, it is of no surprise that the special issue is formed by several articles in such topic.

Zhang et al. introduced a method for reducing the effect of perspective distortion present in real surveillance scenarios [16]. In their article Detection of Abnormal behavior in Narrow Scene with Perspective Distortion, the authors introduced a mechanism for making up the distorting effect in the region of interest extraction followed by improved pyramid L-K optical flow method with perspective weight and disorder coefficient. A set of rules are defined based on the optical flow monitored across frames. Their experimental results indicated that their algorithm can effectively compensate for the distortion effect and improve the accuracy of abnormal behavioural detection.

Instead of focusing on subjects’ behaviours, Shean et al. focused on crowd behaviour in their paper entitled Statistical detection of a panic behavior in crowded scenes [12]. They proposed a sparse representation that accounts for temporal variation of motion magnitudes and orientations, the crowd density and interactions. Their goal was characterising and detecting mass abnormal behaviour; they achieved this goal by characterising the distribution of normal crowd behaviour and identifying outliers of such distribution as panic behaviour. The experimental evaluation showed the superiority of their proposed method over the existing state-of-the-art techniques. The same problem was studied by Xu et al. in their work Dual-Channel CNN for Efficient Abnormal Behavior Identification through Crowd Feature Engineering [16]. The authors proposed a deep learning-based solution that is feed with raw video and with a motion-based spatiotemporal descriptor, resulting in a two-channel convolutional network that learns to detect abnormal behaviour in crowds. Their proposed method showed substantial improvements in terms of detection accuracy and efficiency for online crowd abnormal behavioural detection.

The special issue also includes methods that study abnormal behaviour for surveillance from other perspectives. Camina et al. describe in their paper Bagging Random-Miner: A one-class classifier for file accesses-based masquerade detection an ensemble classifier for file access-based masquerade detection by using a one-class classification approach. Based on their work, it has been shown that one-class classification attains better classification results than many state-of-the-art multi-class classifiers for such important problem.

In terms of security applications, the special issue includes relevant articles covering subject-level and crowd-level solutions. In addition, solutions that go beyond video surveillance are also considered.

Unconscious behavioural analysis Last but not least, the special issue received several submissions on the detection of abnormal unconscious behaviours. This sort of behaviours refers to aspects that go beyond the conscious/standard ones, for example personality and emotion analysis, in which the sole visual information is not enough for an effective analysis.

Avots et al. focused on recognising emotions from audio–visual information by considering a cross-corpus evaluation. In their paper entitled Audio-Visual Emotion Recognition in Wild, the authors perform an extensive evaluation of multimodal features when recognising emotions across datasets. Interestingly, they provide a comparison between automatic emotion recognition using automatic techniques and human decision-makers. At a broader scale, Favaretto et al. introduced a methodology to detect personality and basic emotion characteristics of crowds in video sequences. In their work Detecting Personality and Emotion Traits in Crowds from Video Sequences, Favaretto et al. explore the feasibility of inferring personality (OCEAN model) and emotion from groups of people. In their experimental results, they showed that mapping emotion characteristics to OCEAN dimensions generates coherent information when compared to data provided in the available literature.

From a different perspective, Bablani et al. studied the problem of deception detection in their work An Efficient Concealed Information Test: EEG Feature Extraction and Ensemble Classification for Lie Identification [1]. They approached the problem of detecting lies from subjects by using EEG signal data. Ten subjects who participated in the study were shown images that should cause a stimuli. Signals were represented with traditional features, and an ensemble model was adopted for classification.

Although it is clear that characterising unconscious behaviours is a complicated task that is far from being solved. Efforts like those described above will motivate further research on this relevant topic.

From the above survey, it is clear that the special issue covers a wide diversity in terms of methodologies, applications and even information sources. This evidences the broadness of the field of abnormal behavioural analysis. All of the articles in this issue address important research questions and propose novel solutions that resulted quite competitive. More importantly, these articles pose additional questions that will surely motivate further research on the subject in the forthcoming years.

As guest editors of this special issue, we carefully examined the feedback and discussions provided by the authors and reviewers throughout the revision cycle. This allowed us to filter out a number of high-level research contributions on human behavioural analysis which are focusing on addressing some abnormality detection. We were delighted to see that many researchers have started working on this particular challenge, which this SI was designed for.

The guest editors express their appreciation to the authors for their high-quality work and their contribution to the state of the art of this emerging domain. We would like to thank all reviewers for their helpful feedback and comments. We thank Professor Rahul Sukthankar for his support of this special issue and the Springer team for their support.

We hope you find this special issue both interesting and inspiring.