Using OpenCV

Building and Transferring the Video Capture Program

With the toolchain properly installed on your computer, open a terminal shell and type the following:

mcramon@ubuntu:∼/ $ cd <YOUR BASE TOOLCHAIN PATH>

mcramon@ubuntu:∼/xcompiler$ source environment-setup-*

This command will set all the variables of the environment in your system to the current installation of your toolchain.

To test if it is working, considering the programs created in this book to understand how V4L is written in C, run the following command in your computer shell:

mcramon@ubuntu:∼/xcompiler$ ${CC} --version

i586-poky-linux-gcc (GCC) 4.7.2

Copyright (C) 2012 Free Software Foundation, Inc.

This is free software; see the source for copying conditions.  There is NO

warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

The GCC compiler represented by ${CC} was properly set and you are ready to build the program.

To build the program, you need to run the following:

${CC} -O2 -Wall 'pkg-config --cflags --libs libv4l2' galileo_video_capture.c -o galileo_video_capture

Note that pkg-config is being used to inform the libv4l2 what will be used in the compilation.

Now you can transfer the galileo_video_capture program using your favorite program, as explained in  Chapter 5. For example, if you are using an Ethernet cable or a WiFi card in Intel Galileo and your operation system is Linux/MacOSX, you can use scp. If your computer is Windows, you can use WinSCP. Here’s an example using scp.

On the Intel Galileo terminal shell, create a direct connection like the following:

root@clanton:∼# ifconfig eth0 192.254.1.1 netmask 255.255.255.0 up

Then on your Linux/MacOSX, transfer the file:

mcramon@ubuntu :∼/scp galileo_capture root@192.254.1.1:/home/root/

.

If you are using a customized BSP image with the development tools present on your SD card image, it is easier because you can compile directly through Intel Galileo’s terminal shell. There is no needed to transfer files once the executable is already in the file system.

Running the Program and Capturing Videos

The program is executed in the command line and must run using the Intel Galileo terminal shell. The program accepts some arguments that are used to configure the capture type and redirect the content capture.

Another important argument is - f. If this argument is not set, the current camera settings are used to capture the video. This means you can change them using the v4l2-ctl tool, as explained.

If -f is used, the capture is forced to use a width of 1280, a height of 720, and the pixel format that supports the Motion JPEG stream (MJPEG). This is the only change from the original code, which supports a different format.

You also can define the number of frames you want to be part of your video with the argument - c. You need to use -c <NUMBER OF FRAMES>.

Finally, the-o argument redirects the content captured to an output that can be redirected to a file.

With all these arguments in mind, suppose you want to capture a video with 100 frames, force the resolution to 1280X720 using Motion JPEG encode, and create an output file named video.mjpeg. For this, you can execute the program with the following arguments using the Intel Galileo terminal shell:

root@clanton:∼# ./galileo_video_capture -m -f -c 100 -o > video.mjpeg

....................................................................................................

Each dot “. ” represents a frame captured in the output.

Now use a different setting using the v4l2-ctl command tool and run the same command line as before, but omit the -f. Let’s reduce the video resolution.

root@clanton:∼# v4l2-ctl --set-fmt-video width=320,height=176,pixelformat=1

Then run the command to accept this configuration, by omitting the -f option.

root@clanton:∼# ./galileo_video_capture -m  -c 100 -o > video2.mpjpeg

....................................................................................................

You will have a new video but with a different resolution.

If your device is not enumerated as /dev/video0, it is necessary to use the -d </dev/video*> option. For example, suppose your device is enumerated as /dev/video1. Your command line must then be:

root@clanton:∼# ./galileo_video_capture -d /dev/video1 -m  -c 100 -o > video2.mpjpeg

Converting and Playing Videos

If you read and worked through the previous section, you have two videos collected with the Motion JPEG encode and with different resolutions. If you transfer these files to your computer and try to play them you, will have a very sad surprise. They cannot be played because they are created for streaming, which means some specific headers in the files are missing.

In case you prefer to run it on your computer, the instructions for downloading and installing ffmpeg in different operational systems are found at http://www.ffmpeg.org/download.html . If your computer runs Linux, the easier way is to install it from static releases present at http://ffmpeg.gusari.org/static/ and then run the following commands:

mcramon@ubuntu∼$:∼/$ mkdir ffmpeg;cd ffmpeg

mcramon@ubuntu∼$:∼/$ wget http://ffmpeg.gusari.org/static/64bit/ffmpeg.static.64bit.2014-03-02.tar.gz

mcramon@ubuntu:∼/$ tar -zxvf ffmpeg.static.64bit.2014-03-02.tar.gz

ffmpeg will be available in the same directory you extracted the files.

On your personal computer or in Intel Galileo, you need to execute ffmpeg to convert the videos to a “playable” format in most systems. For example, to convert the first and second videos captured, you can run the following:

mcramon@ubuntu:∼/video_samples$ ffmpeg -f mjpeg -i video.mjpeg -c:v copy video.mp4

ffmpeg version N-63717-g4e3fe65 Copyright (c) 2000-2014 the FFmpeg developers

  built on Jun 3 2014 01:10:16 with gcc 4.4.7 (Ubuntu/Linaro 4.4.7-1ubuntu2)

  configuration: --disable-yasm --enable-cross-compile --arch=x86 --target-os=linux

  libavutil      52. 89.100 / 52. 89.100

  libavcodec     55. 66.100 / 55. 66.100

  libavformat    55. 42.100 / 55. 42.100

  libavdevice    55. 13.101 / 55. 13.101

  libavfilter     4.  5.100 /  4.  5.100

  libswscale      2.  6.100 /  2.  6.100

  libswresample   0. 19.100 /  0. 19.100

Input #0, mjpeg, from 'video.mjpeg':

  Duration: N/A, bitrate: N/A

    Stream #0:0: Video: mjpeg, yuvj422p(pc), 1280x720, 25 fps , 25 tbr, 1200k tbn, 25 tbc

Output #0, mp4, to 'video.mp4':

  Metadata:

    encoder         : Lavf55.42.100

    Stream #0:0: Video: mjpeg (l[0][0][0] / 0x006C), yuvj422p, 1280x720, q=2-31, 25 fps, 1200k tbn, 1200k tbc

Stream mapping:

  Stream #0:0 -> #0:0 (copy)

Press [q] to stop, [?] for help

frame=  100 fps=0.0 q=-1.0 Lsize=    2871kB time=00:00:03.96 bitrate=5939.9kbits/s

video:2870kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.041544%

mcramon@ubuntu:∼/video_samples$ ffmpeg -f mjpeg -i video2.mjpeg -vcodec copy video2.mp4

ffmpeg version N-63717-g4e3fe65 Copyright (c) 2000-2014 the FFmpeg developers

  built on Jun 3 2014 01:10:16 with gcc 4.4.7 (Ubuntu/Linaro 4.4.7-1ubuntu2)

  configuration: --disable-yasm --enable-cross-compile --arch=x86 --target-os=linux

  libavutil      52. 89.100 / 52. 89.100

  libavcodec     55. 66.100 / 55. 66.100

  libavformat    55. 42.100 / 55. 42.100

  libavdevice    55. 13.101 / 55. 13.101

  libavfilter     4.  5.100 /  4.  5.100

  libswscale      2.  6.100 /  2.  6.100

  libswresample   0. 19.100 /  0. 19.100

Input #0, mjpeg, from 'video2.mjpeg':

  Duration: N/A, bitrate: N/A

    Stream #0:0: Video: mjpeg, yuvj422p(pc), 320x176, 25 fps , 25 tbr, 1200k tbn, 25 tbc

Output #0, mp4, to 'video2.mp4':

  Metadata:

    encoder         : Lavf55.42.100

    Stream #0:0: Video: mjpeg (l[0][0][0] / 0x006C), yuvj422p, 320x176, q=2-31, 25 fps, 1200k tbn, 1200k tbc

Stream mapping:

  Stream #0:0 -> #0:0 (copy)

Press [q] to stop, [?] for help

frame=  100 fps=0.0 q=-1.0 Lsize=     843kB time=00:00:03.96 bitrate=1743.1kbits/s

video:841kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.137993%

Basically, -f indicates that the file is encoded as the Motion JPEG pixel format; the -i indicates an input file, and -vcodec copy maintains the same encode and quality, but adds the frames to a MP4 container.

All videos with the extensions of MJPEG and MP4 used in this section are present in the code/video_samples folder of this chapter. Thus you can exercise this conversion independently, with or without a webcam.

Videos with MP4 extensions can be played on your computer. If you are using Ubuntu, you can play them directly using the Movie Player. If you are using Windows or MacOSX, you can play them with the VLC player or using QuickTime (see Figure 7-5).

Open image in new window

By comparing the files before and after the inclusion of headers by ffmpeg, you can see that the frames in general were maintained in the same content and only the headers were added to the end of MP4 file and a few bytes in the beginning. Figure 7-6 shows the header added to the end of the original MJPEG file.

Open image in new window

Building and Transferring the Picture Grabber

The procedure for building and transferring the file is the same as the one used in the section “Building and Transferring the Video Capture Program” of this chapter, except for the command line used to compile the picture grabber program.

Type the following into the command line with the toolchain properly set.

${CC} -O2 -Wall 'pkg-config --cflags --libs libv4l2' picture_grabber.c -o picture_grabber

Then transfer the file to your Intel Galileo board.

Running the Program and Capturing Images

The program is executed in the command line and can be used with Intel Galileo terminal shell. The program accepts some arguments explained in the previous section.

The first step is to capture five images in RGB24 format with a resolution of 352x288.

root@clanton:∼# ./picture_grabber -W 352 -H 288 -c 5

Encode RGB24

Creating image: out000.ppm

Creating image: out001.ppm

Creating image: out002.ppm

Creating image: out003.ppm

Creating image: out004.ppm

As a result, five images with the prefix out and extension ppm are created. Copy these images to your computer and open them using an image viewer.

If you try to use a resolution not supported by the webcam, the V4L2 library will adjust to the closest one supported by the camera and a warning message will appear informing you of the real resolution used. For example, 300x200 is not supported:

root@clanton:∼# ./picture_grabber -W 300 -H 200 -c 5

Encode RGB24

Warning: driver is sending image at 176x144

Creating image: out000.ppm

Creating image: out001.ppm

Creating image: out002.ppm

Creating image: out003.ppm

Creating image: out004.ppm

If you try to run the problem using YUYV encode, just add the -y or --yuyvargument to it:

root@clanton:∼# ./picture_grabber -W 352 -H 288 -c 5 -y

Encode YUYV

Creating image: out000.ppm

Creating image: out001.ppm

Creating image: out002.ppm

Creating image: out003.ppm

Creating image: out004.ppm

Reviewing opencv_capimage.cpp

This first example uses a few objects based in the following classes. VideoCapture is used to create the capture objects that open and configure the devices, capture images and videos, and release the devices when they are not in use any more. Mat receives frames read and works with some algorithms to process the images. It can apply filters, change colors, and transform the images according to mathematical and statistical algorithms. In the next example, Mat is used only to read the image. However, in the next couple of examples, Mat will be used to process images as well.

To understand the code in Listing 7-1, you need a quick overview of each class used.

VideoCapture:: VideoCapture

The first thing to do is to use the VideoCapture class to create a video capture object and open the device or some video stored in the files system.

For more information regarding theVideoCapture class, see http://docs.opencv.org/modules/highgui/doc/reading_and_writing_images_and_video.html .

In the case of the webcam, you will create the object with the parameter -1 in the constructor, as follows:

VideoCapture cap(-1);

The value -1 means, “open the current device enumerated in the system,” so if you have the camera enumerated as /dev/video0 or /dev/video1, the webcam will be opened anyway. Otherwise, if you want to be specific regarding which device to open, you have to pass to the constructor the index of the enumerated device. For example, to open the device /dev/video0, you must pass the number 0 to the constructor like this:

VideoCapture cap(0);

If you’re using Intel Galileo and one camera, I recommend you use -1 to avoid problems with camera enumeration indexes versus the hardcoded number you use in the constructor.

VideoCapture::isOpened()

You can check if the webcam was opened and initiated with success by invoking theisOpened() method. It returns a Boolean as true if the webcam was opened and false if not.

VideoCapture::set(const int prop, int value)

This method sets a property (prop) to a specific value (value). You can set the image’s width, height, frames per second, and several other properties. In the code example, the video width and height are set to 960x544:

int w = 960 ;

int h = 544 ;

cap.set(CV_CAP_PROP_FRAME_WIDTH, w);

cap.set(CV_CAP_PROP_FRAME_HEIGHT, h);

For more information about the properties supported, visit http://nullege.com/codes/search/opencv.highgui.CV_CAP_PROP_FPS .

VideoCapture::read(Mat & image) or operator >> (Mat & image)

This method reads the image from the device. It grabs the image in one single call. The return is aMat object that is explained shortly.

This example uses the operator >>:

Mat frame;

cap >>frame;

VideoCapture::release( )

Once the video is captured, if the destructor of the object is not called, you must release the camera by invoking therelease() method.

cap.release();

At a glance, you can see how simple this is, compared to the software used when we were focusing on Video4Linux.

cv::Mat::Mat

Mat is an awesome class used for matrix operations and it is constantly used in OpenCV applications. Mat is used to organize images in the format of matrixes responsible for saving details of each pixel, including color intensity, position in the image, image dimension, and so on.

The Mat class is organized into two parts—one part contains the image headers with generic information about the image and the second part contains the sequence of bytes representing the image.

In the code example, Mat is called only as Mat instead of as cv::Mat because the namespace was defined in the beginning of the code:

using namespace cv;

Also, in the code example, there is a Mat object created with the simple constructors available in the class:

Mat frame;

In the next examples, other methods will be used and properly discussed. For now, keep in mind what the Mat class is for and this simple constructor.

For more details regarding the Mat class, visit http://docs.opencv.org/modules/core/doc/basic_structures.html#mat-mat . The tutorial maintained by docs.opencv.org is also recommended at http://docs.opencv.org/doc/tutorials/core/mat_the_basic_image_container/mat_the_basic_image_container.html .

cv::imwrite( const string& filename, InputArray img, const vector<int>& params=vector<int>())

This method saves an image to the file system. In the code example, the file is opencv.jpg, the input array is intrinsically casted as my Mat class with the object frame, and the optional vector of theparams argument is omitted.

Mat frame;

cap >>frame;

imwrite("opencv.jpg", frame);

In this case, with the omission of the vector of the params argument, the encode used to capture the image is based on the file extension .jpg. Remember that the camera does not support capturing images in the JPEG format. It captures streaming in Motion JPEG but JPEG is not extracted from Motion JPEG because there is segment called DHT that’s not present in this stream (check out http://www.digitalpreservation.gov/formats/fdd/fdd000063.shtml ). You can extract a series of JPEG images using ffmpeg from a Motion JPEG streaming file, but they will not be viewable in any image software due to the DHT segment missing.

In other words, when the file extension is specified and not supported by the webcam, the OpenCV framework converts the file.

The extensions supported besides JPEG are PNG, PPM, PGM, and PBM.

The docs.opencv.org site maintains a nice tutorial about how to load, modify, and save an image at http://docs.opencv.org/doc/tutorials/introduction/load_save_image/load_save_image.html .

Running opencv_capimage.cpp

Compile the code and transfer the file to Intel Galileo. Make sure the uvcvideo driver is loaded and the webcam is connected to the USB port (read the section called “Connecting the Webcam” in this chapter). Finally, smile at your webcam and run the software:

root@clanton:∼# ./opencv_capimage

VIDIOC_QUERYMENU: Invalid argument

VIDIOC_QUERYMENU: Invalid argument

VIDIOC_QUERYMENU: Invalid argument

VIDIOC_QUERYMENU: Invalid argument

VIDIOC_QUERYMENU: Invalid argument

VIDIOC_QUERYMENU: Invalid argument

VIDIOC_QUERYMENU: Invalid argument

Webcam is OK! I found it!

VIDIOC_QUERYMENU: Invalid argument

VIDIOC_QUERYMENU: Invalid argument

VIDIOC_QUERYMENU: Invalid argument

VIDIOC_QUERYMENU: Invalid argument

VIDIOC_QUERYMENU: Invalid argument

VIDIOC_QUERYMENU: Invalid argument

VIDIOC_QUERYMENU: Invalid argument

You should have a file named opencv.jpg in the same folder. Now, you might asking what the VIDIOC_QUERYMENU: Invalid argument message mean. Such messages are not related to OpenCV and there is nothing wrong with the code. It is simply OpenCV using the Video4Linux framework to understand the capabilities and controls offered by the webcam. When some control or capability is not offered, V4L informs you with these warning messages.

If you do not want to see these messages, you can redirect them using a stderr stream to a null device. For example:

root@clanton:∼# ./opencv_capimage 2> /dev/null

Webcam is OK! I found it!

Performance of OpenCV C++ versus OpenCV Python

To check for performance issues, suppose you have the Python program shown in Listing 7-2 and the C++ program shown in Listing 7-1 properly installed on Intel Galileo. You can measure performance using the bash terminal with the command date +%s, which returns the number of seconds passed since 00:00:00 1970-01-01 UTC. Execute the program and evaluate the time difference.

First, run the Python program with the following command:

root@clanton:∼# s=$(date +%s);python opencv_capimage.py; echo $(expr 'date +%s' - $s)

Webcam is OK! I found it!

8

Python took eight seconds to take the picture. Do the same thing with the C++ program:

root@clanton:∼# s=$(date +%s);./opencv_capimage 2> /dev/null; echo $(expr 'date +%s' - $s)

Webcam is OK! I found it!

4

The same program written in C++ took only four seconds. Even the programs running in the userspace context suffer some time execution variation because it’s not a real-time system. The OpenCV applications created in C++ are much faster than the same applications running in Python.

Reviewing opencv_capimage_canny.cpp

The image originally captured by the camera and the image processed with Canny algorithm are both stored in the file system as captured.jpg and captured_with_edges.jpg using the imwrite() function explained previously.

void cv::cvtColor(InputArray src, OutputArray dst, int code, int dstCn=0)

Converts the image space color to another one. In the following code example:

Mat frame_in_gray;

cvtColor(frame, frame_in_gray, CV_BGR2GRAY);

The input image was the one captured by the webcam and stored in the Mat object frame. The frame_in_gray object was created to receive the image converted in gray space color as requested by code CV_BGR2GRAY.

For more detail about thecvtColor() function and color in general, visit http://docs.opencv.org/modules/imgproc/doc/miscellaneous_transformations.html#cvtcolor .

void cv::Canny(InputArray image, OutputArray edges, double threshold1, double threshold2, int apertureSize=3, bool L2gradient=false)

The Canny function takes the image input array as a source image and transforms the edges into sharp ones. It stores the output array in edges. The input and output image in the example is the same object (frame_in_gray); for best effect, a grayscale image is used.

TheapertureSize argument is the size of the Sobel operator used in the algorithm (see http://en.wikipedia.org/wiki/Sobel_operator for more details) and the code keeps the default value of 3.

The L2gradient argument is a Boolean; when it’s true, the image gradient magnitude is used and when it’s false, only the normative equation is considered. This example used the default value of false.

Two hysteresis thresholds are represented by the arguments threshold1 and threshold2 and the values 0 and 28 were used, respectively. These values are based on my experiments with changing these values until I got results I considered good. You can change these values and check the effects you get.

int threshold1 = 0;

int threshold2 = 28;

Canny(frame_in_gray, frame_in_gray, threshold1, threshold1);

The official documentation about Canvas function is found on this link “ http://docs.opencv.org/modules/imgproc/doc/feature_detection.html?highlight=canny#canny .”

Running opencv_capimage_canny.cpp

Compile the code and transfer the file to Intel Galileo. Make sure the uvcvideo driver is loaded and the webcam is connected to the USB port (read the section entitled “Connecting the Webcam” in this chapter). Point your webcam to some object rich in edges, like the image shown in Figures 7-8 and 7-9.

root@clanton:∼# ./opencv_capimage_canny 2> /dev/null

Webcam is OK! I found it!

processing image with Canny...

Saving the images...

You should have two images stored in the file system as captured.jpg and captured_with_edges.jpg.

Open image in new window

Open image in new window

Reviewing opencv_face_and_eyes_detection.cpp

The following sections provide an explanation of each item used in the code.

cv::CascadeClassifier::CascadeClassifier( )

Creates theCascadeClassifier object. In the example code, two objects are created, one to detect the face and the other to detect the eyes.

CascadeClassifier face_cascade;

CascadeClassifier eyes_cascade;

cv::CascadeClassifier::load(const string & filename)

Loads the file with the classifier to the object. In the code, two classifiers were used, one to detect the face and the other to detect the eyes.

if( !face_cascade.load( face_cascade_name ) )

{

  cout << face_cascade_name << " not found!! aborting..." << endl;

  exit(-1);

};

if( !eyes_cascade.load( eye_cascade_name ) )

{

  cout << eye_cascade_name << " not found!! aborting..." << endl;

  exit(-1);

};

void cv::CascadeClassifier::detectMultiScale(const Mat& image, vector<Rect>& objects, double scaleFactor=1.1, int minNeighbors=3, int flags=0, SizeminSize=Size(), Size maxSize=Size())

//detecting faces

face_cascade.detectMultiScale( img, faces, 1.1, 2, 0|CV_HAAR_SCALE_IMAGE, Size(30, 30) );

...

...

...

//In each face, detect eyes

eyes_cascade.detectMultiScale( faceROI, eyes, 1.1, 2, 0 |CV_HAAR_SCALE_IMAGE, Size(30, 30)

In this code, the scaling factor used is 1.1, minNeighbors is 2 (a kind of hint), the flags are optimized for performance using CV_HAAR_SCALE_IMAGE, and the minimum size of the object to detect is 30x30 pixels. No maximum size is defined, so you can put your face very close to the webcam.

The code detected the faces in the image. For each face that’s detected, a rectangle is drawn delimiting the region.

// rectangle in the face

rectangle( img, faces[i], Scalar( 255, 100, 0 ), 4, 8, 0 );

The resulting regions containing the faces detected are stored in vector<Rect> faces . For example, faces[0] is the first face in the picture. If there is more than one person, you will have face[1], face[2], and so on. The object type Rect means rectangle so the faces vector is a group of rectangles without graphical objects. They are objects that store the initial coordinates (upper-left points) in (Rec.x,Rec.y) and the width (Rec.w) and height (Rec.h) of the rectangle in the object class.

For each region detected, a new image is created with the image content delimited by the rectangle, which forms a small area. This small area is called the ROI (Region of Interest). For best performance and to normalize the image in the eye, detection is converted to grayscale using the cvColor() function.

Mat frame_gray;

cvtColor( img, frame_gray, CV_BGR2GRAY );

// croping only the face in region defined by faces[i]

std::vector<Rect> eyes;

Mat faceROI = frame_gray( faces[i] );

In this small area that contains only the face, the cascade classifier tries to identify the eyes. For each eye detected, a circle is drawn. So the while the face uses the whole image to be detected, the eyes are detected only in the face regions. This optimizes the algorithm.

// In each face, detect eyes

eyes_cascade.detectMultiScale( faceROI, eyes, 1.1, 2, 0 |CV_HAAR_SCALE_IMAGE, Size(30, 30) );

The resultant regions containing the eyes is stored in vector<Rect> eyes.

This process is done with the for loops in this code:

  for( size_t i = 0; i < faces.size(); i++ )

  {

...

...

...

     for( size_t j = 0; j < eyes.size(); j++ )

      {

...

...

...     }

  }

To draw the circles around the eyes, the Point class was used. It extracts information from vector<Rect> eyes and stores the exact center of the eyes (the central coordinates):

Point center ( faces[i].x + eyes[j].x + eyes[j].width*0.5, faces[i].y + eyes[j].y + eyes[j].height*0.5 );

int radius = cvRound ( (eyes[j].width + eyes[j].height)*0.25 );

circle( img, center , radius , Scalar( 255, 0, 0 ), 4, 8, 0 );

Thus, thePoint center object is based on the rectangle’s dimension of the current face, identified by the center point of the eye and the variable radius. Using the function cvRound(), it determines the radius to be drawn around the eyes.

With those two information, it is possible to draw a circle using the function circle().

Figure 7-10 shows this code’s sequence.

Open image in new window

Running opencv_face_and_eyes_detection.cpp

Compile the code and transfer the file to Intel Galileo. Make sure the uvcvideo driver was loaded and the webcam is connected to the USB port (read the section called “Connecting the Webcam” in this chapter) and copy the haarcascade_frontalface_alt.xml and haarcascade_eye.xml files to the same location you transferred the executable program. Stay in front of camera and look in the direction of the lens. Then run the software:

root@clanton:∼# ./opencv_face_and_eyes_detection 2> /dev/null

camera is ok

processing the image....

done!

A image named face_and_eyes.jpg is created in the files system wilt all faces and eyes detected, as shown in Figure 7-11.

Open image in new window

Preparing the Desktop

You need to create a database with a few pictures of you. The process for generating this database is explained in detail in conjunction with some scripts that run in Python.

It’s necessary to have Python installed on your computer, with the pillow and setuptools modules installed.

Pillow is used to treat images using Python scripts and setuptools is a dependence that pillow requests. You should install the setuptools module first.

Pillow can be downloaded from https://pypi.python.org/pypi/Pillow and the setuptools module can be downloaded from https://pypi.python.org/pypi/setuptools . Both sites include information on how to install these modules on Linux, Windows, and MacOSX.

You will also need an image editor because it’s necessary to take some pictures of your face with different emotions and identify the coordinates of the center of each of your eyes. You can use Paint in Windows, Gimp on Linux/OSX and Windows, or any other software that allows you to move the mouse cursor in the image and obtain the coordinates.

You can download Gimp from http://www.gimp.org/ .

Creating the Database

Let’s look at each step in more detail.

Cropping the Images

The next step is to crop the images, removing the ears and hair, and try to generate 20x20 images with only the faces showing. The Python script that was initially created for gender classification was adapted to emotion classification, as shown in Listing 7-5.

Listing 7-5. align_faces.py

#!/usr/bin/env python

# Software License Agreement (BSD License)

#

# Copyright (c) 2012, Philipp Wagner

# All rights reserved.

#

# Redistribution and use in source and binary forms, with or without

# modification, are permitted provided that the following conditions

# are met:

#

#  * Redistributions of source code must retain the above copyright

#    notice, this list of conditions and the following disclaimer.

#  * Redistributions in binary form must reproduce the above

#    copyright notice, this list of conditions and the following

#    disclaimer in the documentation and/or other materials provided

#    with the distribution.

#  * Neither the name of the author nor the names of its

#    contributors may be used to endorse or promote products derived

#    from this software without specific prior written permission.

#

# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS

# "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT

# LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS

# FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE

# COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,

# INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING,

# BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;

# LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER

# CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT

# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN

# ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE

# POSSIBILITY OF SUCH DAMAGE.

#

# Manoel Ramon 06/11/2014- changed the code to support images used

#                          as example of emotion classification

#

import sys, math, Image

def Distance(p1,p2):

  dx = p2[0] - p1[0]

  dy = p2[1] - p1[1]

  return math.sqrt(dx*dx+dy*dy)

def ScaleRotateTranslate(image, angle, center = None, new_center = None, scale = None, resample=Image.BICUBIC):

  if (scale is None) and (center is None):

    return image.rotate(angle=angle, resample=resample)

  nx,ny = x,y = center

  sx=sy=1.0

  if new_center:

    (nx,ny) = new_center

  if scale:

    (sx,sy) = (scale, scale)

  cosine = math.cos(angle)

  sine = math.sin(angle)

  a = cosine/sx

  b = sine/sx

  c = x-nx*a-ny*b

  d = -sine/sy

  e = cosine/sy

  f = y-nx*d-ny*e

  return image.transform(image.size, Image.AFFINE, (a,b,c,d,e,f), resample=resample)

def CropFace(image, eye_left=(0,0), eye_right=(0,0), offset_pct=(0.2,0.2), dest_sz = (70,70)):

  # calculate offsets in original image

  offset_h = math.floor(float(offset_pct[0])*dest_sz[0])

  offset_v = math.floor(float(offset_pct[1])*dest_sz[1])

  # get the direction

  eye_direction = (eye_right[0] - eye_left[0], eye_right[1] - eye_left[1])

  # calc rotation angle in radians

  rotation = -math.atan2(float(eye_direction[1]),float(eye_direction[0]))

  # distance between them

  dist = Distance(eye_left, eye_right)

  # calculate the reference eye-width

  reference = dest_sz[0] - 2.0*offset_h

  # scale factor

  scale = float(dist)/float(reference)

  # rotate original around the left eye

  image = ScaleRotateTranslate(image, center=eye_left, angle=rotation)

  # crop the rotated image

  crop_xy = (eye_left[0] - scale*offset_h, eye_left[1] - scale*offset_v)

  crop_size = (dest_sz[0]*scale, dest_sz[1]*scale)

  image = image.crop((int(crop_xy[0]), int(crop_xy[1]), int(crop_xy[0]+crop_size[0]), int(crop_xy[1]+crop_size[1])))

  # resize it

  image = image.resize(dest_sz, Image.ANTIALIAS)

  return image

if __name__ == "__main__":

#Serious_01.jpg

#left  -> 528, 423

#right -> 770, 431

  image =  Image.open("serious_01.jpg")

  CropFace(image, eye_left=(528,423), eye_right=(770,431), offset_pct=(0.2,0.2)).save("serious01_20_20_70_70.jpg")

#Serious_02.jpg

#left  -> 522,412

#right -> 758, 415

  image =  Image.open("serious_02.jpg")

  CropFace(image, eye_left=(522,412), eye_right=(758,415), offset_pct=(0.2,0.2)).save("serious02_20_20_70_70.jpg")

#Serious_03.jpg

#left  -> 518, 423

#right -> 754, 425

  image =  Image.open("serious_03.jpg")

  CropFace(image, eye_left=(518,423), eye_right=(754,425), offset_pct=(0.2,0.2)).save("serious03_20_20_70_70.jpg")

#Smile_01.jpg

#left  -> 516, 377

#right -> 753, 379

  image =  Image.open("smile_01.jpg")

  CropFace(image, eye_left=(516,377), eye_right=(753,379), offset_pct=(0.2,0.2)).save("smile01_20_20_70_70.jpg")

#Smile_02.jpg

#left  -> 533, 374

#right -> 763, 380

  image =  Image.open("smile_02.jpg")

  CropFace(image, eye_left=(533,374), eye_right=(763,380), offset_pct=(0.2,0.2)).save("smile02_20_20_70_70.jpg")

#Smile_03.jpg

#left  -> 518, 379

#right -> 749, 381

  image =  Image.open("smile_03.jpg")

  CropFace(image, eye_left=(518,379), eye_right=(749,381), offset_pct=(0.2,0.2)).save("smile03_20_20_70_70.jpg")

#surprised_01.jpg

#left  -> 516,356

#right -> 754,355

  image =  Image.open("surprised_01.jpg")

  CropFace(image, eye_left=(516,356), eye_right=(754,355), offset_pct=(0.2,0.2)).save("surprised01_20_20_70_70.jpg")

#surprised_02.jpg

#left  -> 548, 364

#right -> 793, 364

  image =  Image.open("surprised_02.jpg")

  CropFace(image, eye_left=(548,364), eye_right=(793,364), offset_pct=(0.2,0.2)).save("surprised02_20_20_70_70.jpg")

#surprised_03.jpg

#left  -> 528, 377

#right -> 770, 378

  image =  Image.open("surprised_03.jpg")

  CropFace(image, eye_left=(528,377), eye_right=(770,378), offset_pct=(0.2,0.2)).save("surprised03_20_20_70_70.jpg")

If you use the same filenames in your pictures, the only thing that you must change are the coordinates of your eyes for each picture. Then you copy the script into the same folder your pictures are in and run this in the computer shell:

mcramon@ubuntu:∼/tmp/opencv/emotion/mypics$ python align_faces.py

A series of images with the suffix _20_20_70_70 is created:

mcramon@ubuntu:∼/tmp/opencv/emotion/mypics$ ls *20*

serious01_20_20_70_70.jpg  smile01_20_20_70_70.jpg  surprised01_20_20_70_70.jpg

serious02_20_20_70_70.jpg  smile02_20_20_70_70.jpg  surprised02_20_20_70_70.jpg

serious03_20_20_70_70.jpg  smile03_20_20_70_70.jpg  surprised03_20_20_70_70.jpg

If you use different filenames and a different number of pictures, you need to change the script accordingly.

Do not worry about the details of this code; only keep in mind that this script uses the pillow module to create an image object that, using the CropFace() function, crops the image according to the scale reduction. For example, to crop the image file surprised_02.jpg to a scale of 20% x 20%, the following line of code is necessary:

image =  Image.open("surprised_02.jpg")

CropFace(image, eye_left=(548,364), eye_right=(793,364), offset_pct=(0.2,0.2)).save("surprised02_20_20_70_70.jpg")

As a result, all the images will contain only your face, as shown in Figure 7-12.

Open image in new window

The next step is to transfer these cropped images to Intel Galileo. A quick way to do that if you are using Linux, MacOSX, or Windows Cygwin and have Intel Galileo with a valid IP address on your network is to use scp. Run the following in the command line in the directory containing your images:

mcramon@ubuntu:∼/tmp/opencv/emotion/mypics$ for i in $(ls *20*);do scp $i root@192.254.1.1:/home/root/. ;done

All the images are transferred to the /home/root directory.

The Code for Emotion Classification

The code for emotion classification uses a class called FaceRecognizer, which is responsible for reading your models. In other words, it reads the pictures and each state index in the database and, using a model called fisherface, feeds (or trains) the model in order to be able to predict emotions.

The code in this section is based on the code presented in Listing 7-7. Listing 7-8 shows the code with the new parts in bold.

Listing 7-8. opencv_emotion_classification.cpp

/*

* Copyright (c) 2011. Philipp Wagner <bytefish[at]gmx[dot]de>.

* Released to public domain under terms of the BSD Simplified license.

*

* Redistribution and use in source and binary forms, with or without

* modification, are permitted provided that the following conditions are met:

*   * Redistributions of source code must retain the above copyright

*     notice, this list of conditions and the following disclaimer.

*   * Redistributions in binary form must reproduce the above copyright

*     notice, this list of conditions and the following disclaimer in the

*     documentation and/or other materials provided with the distribution.

*   * Neither the name of the organization nor the names of its contributors

*     may be used to endorse or promote products derived from this software

*     without specific prior written permission.

*

*   See < http://www.opensource.org/licenses/bsd-license >

*

*  Manoel Ramon - 06/15/2014

*  manoel.ramon@gmail.com

*                 code changed from original facerec_fisherface.cpp

*                 added:

*                 - adaption to emotions detection instead gender

*                 - picture took from the default video device

*                 - added face and eyes recognition

*                 - crop images based in human anatomy

*                 - prediction based in face recognized

*

*/

#include <opencv2/opencv.hpp>

#include <stdio.h>

#include "opencv2/imgproc/imgproc.hpp"

#include "opencv2/core/core.hpp"

#include "opencv2/contrib/contrib.hpp"

#include "opencv2/highgui/highgui.hpp"

#include <iostream>

#include <fstream>

#include <sstream>

using namespace cv;

using namespace std;

String face_cascade_name = "haarcascade_frontalface_alt.xml";

String eye_cascade_name = "haarcascade_eye.xml";

Mat faceDetect(Mat img);

CascadeClassifier face_cascade;

CascadeClassifier eyes_cascade;

using namespace cv;

using namespace std;

enum EmotionState_t {

  SMILE     =0,   // 0

  SURPRISED,      // 1

  SERIOUS,        // 2

};

static void read_csv(const string& filename, vector<Mat>& images, vector<int>& labels, char separator = ';') {

    std::ifstream file(filename.c_str(), ifstream::in);

    if (!file) {

        string error_message = "No valid input file was given, please check the given filename.";

        CV_Error(CV_StsBadArg, error_message);

    }

    string line, path, classlabel;

    while (getline(file, line)) {

        stringstream liness(line);

        getline(liness, path, separator);

        getline(liness, classlabel);

        if(!path.empty() && !classlabel.empty()) {

            images.push_back(imread(path, 0));

            labels.push_back(atoi(classlabel.c_str()));

        }

    }

}

int main(int argc, const char *argv[])

{

  EmotionState_t emotion;

  // Check for valid command line arguments, print usage

  // if no arguments were given.

  if (argc < 2) {

    cout << "usage: " << argv[0] << " <csv.ext> <output_folder> " << endl;

    exit(1);

  }

  if( !face_cascade.load( face_cascade_name ) ){ printf("--(!)Error loading\n"); return -1; };

  if( !eyes_cascade.load( eye_cascade_name ) ){ printf("--(!)Error loading\n"); return -1; };

  // 0 is the ID of the built-in laptop camera, change if you want to use other camera

  VideoCapture cap(-1);

  //check if the file was opened properly

  if(!cap.isOpened())

  {

      cout << "Capture could not be opened succesfully" << endl;

      return -1;

  }

  else

  {

      cout << "camera is ok.. Stay 2 ft away from your camera\n" << endl;

  }

  int w = 432;

  int h = 240;

  cap.set(CV_CAP_PROP_FRAME_WIDTH, w);

  cap.set(CV_CAP_PROP_FRAME_HEIGHT, h);

  Mat frame;

  cap >>frame;

  cout << "processing the image...." << endl;

  Mat testSample = faceDetect(frame);

  // Get the path to your CSV.

  string fn_csv = string(argv[1]);

  // These vectors hold the images and corresponding labels.

  vector<Mat> images;

  vector<int> labels;

  // Read in the data. This can fail if no valid

  // input filename is given.

  try

  {

    read_csv(fn_csv, images, labels);

  } catch (cv::Exception& e) {

    cerr << "Error opening file \"" << fn_csv << "\". Reason: " << e.msg << endl;

    // nothing more we can do

    exit(1);

  }

  // Quit if there are not enough images for this demo.

  if(images.size() <= 1)

  {

    string error_message = "This demo needs at least 2 images to work. Please add more images to your data set!";

    CV_Error(CV_StsError, error_message);

  }

  // Get the height from the first image. We'll need this

  // later in code to reshape the images to their original

  // size:

  int height = images[0].rows;

  // The following lines create an Fisherfaces model for

  // face recognition and train it with the images and

  // labels read from the given CSV file.

  // If you just want to keep 10 Fisherfaces, then call

  // the factory method like this:

  //

  //      cv::createFisherFaceRecognizer(10);

  //

  // However it is not useful to discard Fisherfaces! Please

  // always try to use _all_ available Fisherfaces for

  // classification.

  //

  // If you want to create a FaceRecognizer with a

  // confidence threshold (e.g. 123.0) and use _all_

  // Fisherfaces, then call it with:

  //

  //      cv::createFisherFaceRecognizer(0, 123.0);

  //

  Ptr<FaceRecognizer> model = createFisherFaceRecognizer();

  model->train(images, labels);

  // The following line predicts the label of a given

  // test image:

  int predictedLabel = model->predict(testSample);

  // To get the confidence of a prediction call the model with:

  //

  //      int predictedLabel = -1;

  //      double confidence = 0.0;

  //      model->predict(testSample, predictedLabel, confidence);

  //

  string result_message = format("Predicted class = %d", predictedLabel);

  cout << result_message << endl;

  // giving the result

  switch (predictedLabel)

  {

    case SMILE:

      cout << "You are happy!" << endl;

      break;

    case SURPRISED:

      cout << "You are surprised!" << endl;

      break;

    case SERIOUS:

      cout << "You are serious!" << endl;

      break;

  }

  return 0;

  cap.release();

  return 0;

}

Mat faceDetect(Mat img)

{

  std::vector<Rect> faces;

  std::vector<Rect> eyes;

  bool two_eyes = false;

  bool any_eye_detected = false;

  //detecting faces

  face_cascade.detectMultiScale( img, faces, 1.1, 2, 0|CV_HAAR_SCALE_IMAGE, Size(30, 30) );

  if (faces.size() == 0)

  {

       cout << "Try again.. I did not dectected any faces..." << endl;

       exit(-1);  // abort everything

  }

  Point p1 = Point(0,0);

  for( size_t i = 0; i < faces.size(); i++ )

  {

    // we cannot draw in the image !!! otherwise will mess with the prediction

    // rectangle( img, faces[i], Scalar( 255, 100, 0 ), 4, 8, 0 );

     Mat frame_gray;

     cvtColor( img, frame_gray, CV_BGR2GRAY );

     // croping only the face in region defined by faces[i]

     std::vector<Rect> eyes;

     Mat faceROI = frame_gray( faces[i] );

     //In each face, detect eyes

     eyes_cascade.detectMultiScale( faceROI, eyes, 1.1, 3, 0 |CV_HAAR_SCALE_IMAGE, Size(30, 30) );

      for( size_t j = 0; j < eyes.size(); j++ )

      {

         Point center( faces[i].x + eyes[j].x + eyes[j].width*0.5, faces[i].y + eyes[j].y + eyes[j].height*0.5 );

         // we cannot draw in the image !!! otherwise will mess with the prediction

         // int radius = cvRound( (eyes[j].width + eyes[j].height)*0.25 );

         // circle( img, center, radius, Scalar( 255, 0, 0 ), 4, 8, 0 );

         if (j==0)

           {

              p1 = center;

              any_eye_detected = true;

           }

         else

         {

              two_eyes = true;

         }

      }

    }

  cout << "SOME DEBUG" << endl;

  cout << "-------------------------" << endl;

  cout << "faces detected:" << faces.size() << endl;

  cout << "x: " << faces[0].x << endl;

  cout << "y: " << faces[0].y << endl;

  cout << "w: " << faces[0].width << endl;

  cout << "h: " << faces[0].height << endl << endl;

  Mat imageInRectangle;

  imageInRectangle =  img(faces[0]);

  Size recFaceSize = imageInRectangle.size();

  cout << recFaceSize << endl;

  // for debug

  imwrite("imageInRectangle.jpg", imageInRectangle);

  int rec_w = 0;

  int rec_h = faces[0].height * 0.64;

  // checking the (x,y) for cropped rectangle

  // based in human anatomy

  int px = 0;

  int py = 2 * 0.125 * faces[0].height;

  Mat cropImage;

  cout << "faces[0].x:" << faces[0].x << endl;

  p1.x = p1.x - faces[0].x;

  cout << "p1.x:" << p1.x << endl;

  if (any_eye_detected)

  {

      if (two_eyes)

      {

          cout << "two eyes detected" << endl;

          // we have detected two eyes

          // we have p1 and p2

          // left eye

          px = p1.x /  1.35;

      }

      else

      {

          // only one eye was found.. need to check if the

          // left or right eye

          // we have only p1

          if (p1.x > recFaceSize.width/2)

          {

              // right eye

            cout << "only right eye detected" << endl;

            px = p1.x / 1.75;

          }

          else

          {

              // left eye

            cout << "only left eye detected" << endl;

            px = p1.x /  1.35;

          }

      }

  }

  else

  {

      // no eyes detected but we have a face

      px = 25;

      py = 25;

      rec_w = recFaceSize.width-50;

      rec_h = recFaceSize.height-30;

  }

  rec_w = (faces[0].width - px) * 0.75;

  cout << "px   :" << px << endl;

  cout << "py   :" << py << endl;

  cout << "rec_w:" << rec_w << endl;

  cout << "rec_h:" << rec_h << endl;

  cropImage = imageInRectangle(Rect(px, py, rec_w, rec_h));

  Size dstImgSize(70,70); // same image size of db

  Mat finalSizeImg;

  resize(cropImage, finalSizeImg, dstImgSize);

  // for debug

  imwrite("onlyface.jpg", finalSizeImg);

  cvtColor( finalSizeImg, finalSizeImg, CV_BGR2GRAY );

  return finalSizeImg;

}

Reviewing opencv_emotion_classification.cpp

In the beginning of the code, there is an enumerator created to define the emotional state. Note the value of each element on this enum matches the emotion index in the CSV file.

enum EmotionState_t {

  SMILE     =0,   // 0

  SURPRISED,      // 1

  SERIOUS,        // 2

};

In themain() function, a variable of type EmotionState_t is created and it is expected to receive the name of the CSV file as an argument.

int main(int argc, const char *argv[])

{

  EmotionState_t emotion;

  // Check for valid command line arguments, print usage

  // if no arguments were given.

  if (argc < 2) {

    cout << "usage: " << argv[0] << " <csv.ext> <output_folder> " << endl;

    exit(1);

  }

When the webcam is opened, the picture is collected as before. ThefaceDetect() method changes, compared to the faceDetect() method shown earlier:

Mat testSample = faceDetect(frame);

This new object stored in testSample contains the cropped face. This cropped image is the same size as the images in the database. This image returned is in grayscale and is cropped like the images shown in Figure 7-12.

The frame contains an 432x240 image and thetestSample image is 70x70. For now, let’s continue with the main() function. faceDetect() will be discussed in more detail later.

With the image prepared to be analyzed, new components are used to predict the emotional state:

Ptr<FaceRecognizer> model = createFisherFaceRecognizer();

model->train(images, labels);

// The following line predicts the label of a given

// test image:

int predictedLabel = model->predict(testSample);

class FaceRecognizer : public Algorithm

At a glance, FaceRecognizer looks very simple, but in fact it’s very powerful and complex. This class allows you to set different algorithms, including your own, to perform different kinds of image recognitions.

The model used in the code is fisherface and it’s created by the line:

Ptr<FaceRecognizer> model = createFisherFaceRecognizer();

void FaceRecognizer::train(InputArrayOfArrays src, InputArray labels)

This method trains the model based on your database. The code passes the images and index (or labels):

model->train(images, labels);

int FaceRecognizer::predict(InputArray src) const = 0

This method predicts the classification index (label) based on the image casted as the input array src.

For example, if the emotion “happy” is labeled as 0 in the CSV file and the FaceRecognizer was trained, the prediction will return 0 if the image src is a picture of you smiling.

This is represented by the following snippet:

int predictedLabel = model->predict(testSample);

...

...

...

  // giving the result

  switch (predictedLabel)

  {

    case SMILE:

      cout << "You are happy!" << endl;

      break;

    case SURPRISED:

      cout << "You are surprised!" << endl;

      break;

    case SERIOUS:

      cout << "You are serious!" << endl;

      break;

  }

If the image returned by faceDetect()is cropped properly and your expression is similar to the expression in the database, the algorithm will predict accurately.

The faceDetect() method basically does what was done before, as explained in the flowchart in Figure 7-10. In other words, it detects the face and eyes.

After this example detects the face and eyes, an algorithm containing a few simple concepts about human anatomy was introduced that crops the image to the face area.

It tries to crop the image captured by the webcam dynamically and do the same thing that was done manually by the script in Listing 7-7, but instead crop the image exclusively in the area the face is detected.

To understand how the logic works, see Figure 7-13.

Open image in new window

Take a look at Figure 7-13 and follow this logic. When a face is recognized the whole face is typically captured, including the ears, head or hat, part of the neck, and part of the background image (imageRectangle). However, these elements are not interesting to the emotion classifier and must be removed (the red arrows area) and only the portion containing eyes, nose, and mouth are cropped (cropImage).

The cropped image has the initial px and py coordinates with the extensions rec_w and rec_h, which forms a triangle with perfect dimensions for cropping the area. Such a rectangle corresponds to the ROI (Region of Interest) area.

To reach the ROI area, the eyes are detected and then the human proportions are found using the px, px, rec_w, and rec_h values in the image and crop the image.

When the eyes are detected, it is possible to define a point object p1 that corresponds to the center of the eye. The point object p1 has two members x and y that represent the distance in pixels to the original image. There are a couple of problems, however. Sometimes only one eye is detected and the algorithm must determine if it’s the right or the left. Other times, no eye is detected.

  //detecting faces

  face_cascade.detectMultiScale ( img, faces, 1.1, 2, 0|CV_HAAR_SCALE_IMAGE, Size(30, 30) );

  Point p1 = Point(0,0);

  for( size_t i = 0; i < faces.size(); i++ )

  {

...

...

...

     // In each face, detect eyes

     eyes_cascade.detectMultiScale ( faceROI, eyes, 1.1, 3, 0 |CV_HAAR_SCALE_IMAGE, Size(30, 30) );

      for( size_t j = 0; j < eyes.size(); j++ )

      {

         Point center( faces[i].x + eyes[j].x + eyes[j].width*0.5, faces[i].y + eyes[j].y + eyes[j].height*0.5 );

...

...

...

         if (j==0)

           {

              p1 = center;

              any_eye_detected = true;

           }

         else

         {

              two_eyes = true;

         }

      }

    }

At the moment, you might have the center of one of the eyes and it is known if one, two, or none eyes were detected. Now it is necessary to find the px and py coordinates, as well as the ROI dimensions, rec_w and rec_h.

In human anatomy, the eyes are located at the top of the horizontal red line that splits the human face in half. If you split the middle horizontal line equally in four parts, the eyes are separated by each other by half of the largest horizontal proportion and one-fourth from the laterals.

The nose and mouth are centralized in the middle of the face, with the inferior proportional divided in five equal parts. The eyebrows are 12.5% above the lines of the eyes because they are 50%/4 of the superior part of the face.

If no eye is detected then it is not possible to crop using a simple algorithm. With these proportions in mind, the following lines were created:

  int rec_w = 0;

  int rec_h = faces[0].height * 0.64;

  // checking the (x,y) for cropped rectangle

  // based in human anatomy

  int px = 0;

  int py = 2 * 0.125 * faces[0].height;

  Mat cropImage;

  cout << "faces[0].x:" << faces[0].x << endl;

  p1.x = p1.x - faces[0].x;

  cout << "p1.x:" << p1.x << endl;

  if (any_eye_detected)

  {

      if (two_eyes)

      {

          cout << "two eyes detected" << endl;

          // we have detected two eyes

          // we have p1 and p2

          // left eye

          px = p1.x /  1.35;

      }

      else

      {

          // only one eye was found.. need to check if the

          // left or right eye

          // we have only p1

          if (p1.x > recFaceSize.width/2)

          {

              // right eye

            cout << "only right eye detected" << endl;

            px = p1.x / 1.75;

          }

          else

          {

              // left eye

            cout << "only left eye detected" << endl;

            px = p1.x /  1.35;

          }

      }

  }

  else

  {

      // no eyes detected but we have a face

      px = 25;

      py = 25;

      rec_w = recFaceSize.width-50;

      rec_h = recFaceSize.height-30;

  }

  rec_w = (faces[0].width - px) * 0.75;

  cout << "px   :" << px << endl;

  cout << "py   :" << py << endl;

  cout << "rec_w:" << rec_w << endl;

  cout << "rec_h:" << rec_h << endl;

  cropImage = imageInRectangle(Rect(px, py, rec_w, rec_h));

For debugging purposes, the faceDetect() method saves two images in the file system every time the software runs. One is called onlyface.jpg and it contains the cropped image. The other is called imageInRectangle.jpg and it contains the detected image.

Mat imageInRectangle;

imageInRectangle =  img(faces[0]);

...

...

...

  // for debug

  imwrite("imageInRectangle.jpg", imageInRectangle);

  cropImage = imageInRectangle(Rect(px, py, rec_w, rec_h));

...

...

...

  Size dstImgSize(70,70); // same image size of db

  Mat finalSizeImg;

  resize(cropImage, finalSizeImg, dstImgSize);

Running opencv_emotion_classification.cpp

Compile the code and transfer the file to Intel Galileo. Make sure theuvcvideo driver is loaded and the webcam is connected to the USB port (read the “Connecting the Webcam” section in this chapter), and transfer the program to the same location of your CSV file. Stay in front your camera, preferably two feet away, make some emotional expressions, and then run the following command:

root@clanton:∼/emotion# ./opencv_emotion_classification my_csv.csv 2> /dev/null

camera is ok.. Stay 2 ft away from your camera

processing the image....

SOME DEBUG

-------------------------

faces detected:1

x: 172

y: 25

w: 132

h: 132

[132 x 132]

faces[0].x:172

p1.x:-172

px   :25

py   :25

rec_w:80

rec_h:102

Predicted class = 0

You are happy!

The software classifies the image as happy. Extracting the debug images onlyface.jpg and imageInRectangle.jpg from the file system, it is possible to observe my expression in the cropped image, shown in Figure 7-14.

Open image in new window

Note in Figure 7-14 the areas that are cropped out, including the background, the hair, and the ears.

root@clanton:∼/emotion# ./opencv_emotion_classification my_csv.csv 2> /dev/null

camera is ok.. Stay 2 ft away from your camera

processing the image....

SOME DEBUG

-------------------------

faces detected:1

x: 178

y: 3

w: 143

h: 143

[143 x 143]

faces[0].x:178

p1.x:43

two eyes detected

px   :31

py   :35

rec_w:84

rec_h:91

Predicted class = 1

You are surprised!

The software classifies this image as surprised. Extracting the debug images onlyface.jpg and imageInRectangle.jpg from the file system, it observes my expression and crops the image, as shown in Figure 7-15.

Open image in new window

Keep varying the expression and checking the efficiency of the captured images.