Visual detection of knives in security applications using Active Appearance Models
- 2k Downloads
In this paper, a novel application of Active Appearance Models to detecting knives in images is presented. In contrast to its popular applications in face segmentation and medical image analysis, we not only use this computer vision algorithm to locate an object that is known to exist in an analysed image, but–using an interest point typical of knives–also try to identify whether or not a knife exists in the image in question. We propose an entire detection scheme and examine its performance on a sample test set. The work presented in this paper aims to create a robust visual knife-detector to be used in security applications.
KeywordsActive Appearance Models Knife-detection Computerised video surveillance Harris interest-point detector
One application of object detection in images is computer-aided video surveillance. It can be used both in security applications  and as legal evidence . A CCTV operator usually monitors multiple video feeds at the same time; this is a complex and challenging task in terms of allocating attention effectively. One study suggests that detection rates for operators monitoring four, nine and 16 screens oscillate around 83 %, 84 % and 64 % respectively, and will drop significantly after an hour . Therefore, the need to automate the process is obvious. There have been attempts at detecting suspicious events in video material  and recognising human activity in videos .
This study focuses on automatic detection of knives in images. Carrying knives in public is either forbidden or restricted in many countries; due to the fact that knives are both widely available and can be used as weapons, their detection is of high importance for security personnel. For instance, the idea of software-based knife-detection has a practical application in surveillance of the public using CCTV. Should a knife be detected, an alarm is raised and the human operator can immediately focus their attention on that very scene, and either confirm or reject that detection. Although people will almost always outperform software algorithms for object detection in images , in the long run the computer could be of significant assistance to the human CCTV operator when it comes to dealing with tens of simultaneous video feeds for many hours a day. Another application of automatic knife-detection is computer-aided analysis of luggage x-ray scans. Visual detection in security applications approach is a new research area. Visual detectors designed to work in video surveillance or x-ray scanners are not widely available.
Knives are a very wide class of objects of immense diversity. Moreover, they easily reflect light, which reduces their visibility in video sequences; automatic knife-detection in images therefore represents a challenging task. In this paper, a novel application of the well-established Active Appearance Models (AAMs) is presented. So far, these have been extensively used for medical image interpretation   , and for the matching and tracking of faces  . Among many existing shape-modelling algorithms, such as the Active Contour Models (Snakes), we have focused on AAMs because they model not only the shape but also the appearance (that is, pixel intensities within the image region bounded by the shape). As the knife-blade typically possesses quite a uniform texture, modelling its appearance should contribute to the general resistance of the model so that it does not converge to objects that have a shape similar to that of the knife-blade.
The novelty of this work is twofold. Not only has there been (to the best knowledge of the authors) no other research on knife-detection, but also AAMs have so far not been used to detect objects belonging to a general class. They have been used in what is referred to as ‘detection’–as in –but that is not what is meant by ‘detection’ in computer vision, in the strict sense of the word. By detecting, for example, a face in an image, we mean answering the question of whether there is or there is not a face in the given image . This process can be characterised by two parameters: the positive and the negative detection rates. As in the case of , before what is referred to as ‘detection’ is performed, an assumption exists that the object is somewhere in the image, and the task is to precisely locate it. For instance, given an image of a face, finding the nose is not a task of detection since we can assume that all faces have noses. It is, rather, the task of location, and can be characterised by the level of localisation accuracy, but not by positive and negative detection rates. In this case, the assumption is that there always is a face in the analysed image. Should the AAM be performed on a non-face image, it would converge to the parts of the image whose appearance is closest to its model. This is still theoretically correct, but makes no sense from a practical point of view. Moreover, AAMs are sensitive to the initial location of their landmark points in the analysed image. Even if there is a face somewhere in a large image, for the algorithm to correctly segment it into elements, the initial location of its landmark points needs to be roughly around the face region. A common technique for face segmentation with AAMs is the use of Viola and Jones’s face detector  to initialise the AAM as in .
In this paper, a method for detection of objects, in this case knives, using AAMs is introduced. It aims to answer the question of whether or not there a knife exists in the given image.
2 Active Appearance Models
AAMs were introduced  in 1998 as a generalisation of the popular Active Contour Model (Snake) and Active Shape Model algorithms. They are a learning-based method, which was originally applied to interpret face images. Due to their generic nature and high effectiveness in locating objects, numerous applications in medical image interpretation followed. The typical medical application is finding an object, usually an organ in a medical image of a specific body part, such as locating the bone in a magnetic resonance image (MRI) of the knee , the left and right ventricles in a cardiac MRI , or the heart in a chest radiograph .
Choose an initial reference shape (e.g. the first object’s shape).
Align all other shapes to the reference shape.
Re-calculate the mean shape of the aligned shapes.
If the distance of the mean shape to the reference one is larger than a threshold, set mean shape as the reference shape and go back to point 2. Otherwise, return the mean shape.
Calculate the centres of gravity (COGs) both of the mean and the shape being superimposed to the mean.
Rescale the shape being superimposed so that its size equals that of the mean.
Align the position of the two COGs.
Rotate the shape being superimposed so that the Procrustes distance is minimal.
3 Corner detection
4 Overcoming the AAM’s variance to rotation and scale change for detection purposes
AAMs are not invariant to rotation. An AAM will converge only to objects that are oriented at the same angle as the objects in the images it was trained for. If we want it to converge to an object of the class it was trained for, but placed at an arbitrary angle in an image, one obvious workaround is to rotate that image until the object considered is placed at a right angle. This is a plausible solution, but a drawback to this approach is the computational effort needed to rotate the input image. For example, rotating a 640 × 480 pixel image by 15° takes some 10 ms on a modern PC (Intel(R) Core(TM) i5-3450 CPU @ 3.10GHz, 8GB RAM), so a whole rotation by 360° with a 15-° angle step will take some 240 ms.
Instead of rotating the input image by 360° at a fixed angle step, different AAMs could be trained for different knife orientations. From a computational point of view, this approach has a significant computational advantage over the former, as it requires just as many AAM single searches, while it does not require rotation of the input image. Hence, this approach of using multiple AAM trained to locate objects in different orientations, i.e. at different angles, has been chosen.
5 Detection scheme and results
As stated in the introduction, AAMs have been used to locate objects in images and–in our application–to detect objects. In this section, the procedure of object detection utilising AAM is described. It is based on the assumption that if a knife exists in the analysed image, its tip will be designated as a corner by the Harris corner detector. All the designated points will be used to initialise AAMs trained to locate knives. The results of running the Harris corner detector on knife images have been presented in Fig. 2. We can see that the tip of the knife is highly likely to be designated as a corner.
The general principle of multiscale image search is that at each corner, AAMs trained for all knife orientations are run, and if at least one of them converges to the knife orientation it was trained for, from the initial location designated by the corner and from slightly varying locations, the detection result is positive. We have trained 24 AAMs for objects rotated by a 15-degree angle step to cover a full 360-degree rotation. Each AAM is composed of 25 landmark points defining the shape polygon.
Each AAM was trained using the same six images of knives rotated by a respective angle. Adding additional images to the training set brought little (if any) improvement to the AAM performance. We believe that this is due to the rather simple shape we are dealing with, in contrast to, for example, the shape of the human face.
Put the points designated by the Harris corner detector in the knife-tip candidate set. Choose the first point and remove it from the set. Choose the first AAM of the 24 AAMs trained for different object orientations.
Set the AAM’s landmark corresponding to the knife-tip at the chosen point.
Run the chosen AAM, calculate the minimum-area bounding box of the landmark points that have now converged to a new location. The box’s longer symmetry axis is considered as the knife-candidate’s pseudo symmetry axis.
Calculate the percentage of landmark points that lie on the edges found in the image undergoing detection by the Canny edge detector.
Set the AAM’s landmark point corresponding to the knife-tip to the following three locations: three pixels to the left of, above and to the right of the initial knife-tip, and perform steps 3 and 4 on them.
- 6.The result of detection is positive if the following conditions have been met:
In all four cases (the original corner and the three points in its neighbourhood), landmark points of the model converge to the same location.
In all four cases, the knife-candidate’s pseudo symmetry axes have a similar skew-angle.
Most of the landmark points lie on edges detected by the Canny edge detector.
If the result is positive, stop the detection procedure. If at least one of the above conditions is not met, the result of detection is negative. In this case, continue to the next point.
Choose the next AAM, i.e. the AAM trained for the orientation increased by 15° in relation to the current AAM, and go to point 3. If all orientations have been covered, go to the next point. Theoretically multiple AAM searches could be performed in parallel in order to speed up the detection process.
If the knife-tip candidate set is not empty, choose the next point, remove it from the set and go back to point 2.
The idea behind this detection scheme is that if the appearance (and so the shape) that the AAM has been trained to locate exists in the image, the model will converge to it if reasonably initialised, even from slightly varying locations (see point 5). In other words, the AAM will converge to the same location, which is evaluated with the minimum-area bounding box and skew-angle. If the appearance that the AAM is searching for does not exist, the model will converge to some random locations. Moreover, to make the positive-detection condition even stronger, we assume that most of the landmark points lie on an edge, to eliminate cases where there are no edges visible (and therefore no knife exists). The exact number of landmarks that actually lie on an edge can be chosen only by a heuristic rule. We have chosen this threshold to be 70 % of the points on each of the two edges of the knife-blade.
Characteristics of the proposed AAM-based detection scheme
Number of images in positive test set
Number of correctly classified positive images
Number of images in negative test set
Number of wrongly classified negative images
6 Application in baggage-scanning systems
The proposed knife-detection scheme can be applied to interpreting images produced by baggage-scanning x-ray systems. Modern baggage-scanners produce images of excellent quality and are deployed at numerous locations, including airports, courts of law, government offices and other venues where the danger of bringing in a dangerous tool or explosive exists.
An undoubted advantage of utilising the AAM in knife-detection is its observed zero false-positive detection rate. If there is no knife in an image, AAMs initialised from slightly varying locations will either diverge or converge to different locations. Theoretically it could be the case that an object of a similar shape to the knife-blade will be falsely recognised as a knife. When it comes to application in x-ray baggage-scan analysis, a low non-zero false-positive rate should present little problem in a practical setting, since such images need always to be assessed by a human operator. On the other hand, the condition that the knife-tip is clearly visible in order to properly initialise AAMs may limit the number of possible applications, but most often should be met in baggage x-ray scans. Moreover, due to the technical capabilities of x-ray scanners, objects made from different materials are clearly distinct in images, which in addition to the fact that the size of a knife is relatively large in relation to the size of the suitcase makes our approach of utilising AAMs especially well suited to application in baggage x-ray scan analysis.
7 Summary and conclusions
Applying AAMs to the problem of object detection is a novel approach. So far, they have been used to locate an object in images that were known to contain it, such as medical images containing a body organ–or images of faces, where the task was to locate particular facial elements. In our work, we have used the fact that the knife-blade has a very specific interest point, which can be easily detected as a corner, i.e. its tip. This point can be used to initialise the AAM. The rule that an AAM trained for the orientation close to that of the knife in the image initialised from similar locations should converge to the same location allows us to decide whether the object in question is in fact a knife. The presented approach is well suited to applications where the knife-tip is clearly visible, such as in baggage-scanning systems. If it is not the case, then combining the AAM with a detector of a different kind, due to the AAM’s theoretical zero false-positive detection rate, will surely create a robust knife-detector; this is the subject of ongoing research.
This work has been co-financed by the European Regional Development Fund under the Innovative Economy Operational Programme, INSIGMA project no. POIG.01.01.02-00-062/09.
- 2.Bradski G (2000) The OpenCV Library. Dr. Dobb’s Journal of Software ToolsGoogle Scholar
- 3.Bradski, Rost G, Kaehler A (2008) Learning opencv, 1st edition, O’Reilly Media, Inc.Google Scholar
- 4.Cootes TF, Edwards GJ, Taylor CJ (1998) Active appearance models. In Proceedings: European Conference on Computer Vision 2:484–498Google Scholar
- 5.Cootes TF, Taylor CJ (2001) Statistical models of appearance for computer vision. Tech. report. University of ManchesterGoogle Scholar
- 6.Cristinacce D, Cootes TF (2004) A comparison of shape constrained facial feature detectors. Automatic face and gesture recognition. In Proceedings:. Sixth IEEE International Conference, pp 375–380Google Scholar
- 7.Daijin Kim, Jaewon Sung (2006) A real-time face tracking using the stereo active appearance model. In Proceedings: Image Processing, 2006 IEEE International Conference, pp 2833–2836Google Scholar
- 8.Dalal N, Triggs B (2005) Histograms of oriented gradients for human detection. In Proceedings: Computer Vision and Pattern Recognition (CVPR): pp. 886–893Google Scholar
- 9.Derpanis KG (2004) The Harris corner detector. Accessed 10 April 2013 http://www.cse.yorku.ca/~kosta/CompVis_Notes/harris_detector.pdf
- 10.Edwards GJ, Taylor CJ, Cootes TF (1998) Interpreting face images using active appearance models. Automatic face and gesture recognition. In: Proceedings: Third IEEE International Conference, pp 300–305Google Scholar
- 12.Leung KYE, van Stralen M, van Burken G, van der Steen AFW, de Jong N, Bosch JG Automatic (2010) 3D left ventricular border detection using active appearance models. In Proceedings: Ultrasonics Symposium (IUS) pp 197–200Google Scholar
- 13.Liang Y-M, Shih S-W, Shih AC-C (2011) Human action segmentation and classification based on the Isomap algorithm. Multimedia Tools and Applications: pp 1–20Google Scholar
- 14.Maalouf A, Larabi M-C, Nicholson D (2012) Offline quality monitoring for legal evidence images in video-surveillance applications. Multimedia Tools and Applications: pp 1–30Google Scholar
- 16.Mitchell S, Lelieveldt B, Geest R, Schaap J, Reiber J, Sonka M (2000) Segmentation of cardiac MR images: An active appearance model approach. In Proceedings: Image Processing, San Diego CA, SPIE 1:224–234Google Scholar
- 18.Paul Viola and Michael Jones (2001) Rapid Object Detection Using a Boosted Cascade of Simple Features. In Proceedings: IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), Kauai, USAGoogle Scholar
- 20.Stegmann MB, Gomez DD (2002) A brief introduction to statistical shape analysis. Technical report. In: Informatics and mathematical modelling, Technical University of Denmark, DTUGoogle Scholar
- 21.Tadeusiewicz R (2011) How intelligent should Be system for image analysis? In: Kwasnicka H, Jain LC (eds) Innovations in intelligent image analysis. Studies in computational intelligence, vol. 339. Springer Verlag, BerlinGoogle Scholar
- 22.Tadeusiewicz R, Introduction to Intelligent Systems (2011) In: Wilamowski BM, Irvin JD (Eds.) The industrial electronics handbook—intelligent systems. CRC Press, Boca Raton. pp 1–1 – 1–12.Google Scholar
- 24.van Ginneken B, Stegmann MB, Loog M (2004). Segmentation of anatomical structures in chest radiographs using supervised methods: A comparative study on a public database. Medical Image AnalysisGoogle Scholar
- 25.Yifeng J, Zhijun Z, Feng C, Hung Tat T, Tze Kin L (2004) An enhanced appearance model for ultrasound image segmentation. Pattern Recognition 3:802–805Google Scholar
Open Access This article is distributed under the terms of the Creative Commons Attribution License which permits any use, distribution, and reproduction in any medium, provided the original author(s) and the source are credited.