Camera Roll Face Filter - A no-trained model for filtering out photos containing the same faces

Imagine digging through a lot of photos everytime you decide to send a photo from your smart phone to a whatsapp contact or to your facebook/twitter account.

There has to be an easier way to filter out photos containing the required content(somebody's face) in the camera roll.
This blog post is about a naive approach to solving this issue of search for your photos.

Algorithm -

1. Take any photo (one input and one to compare with the input)

2. Find the face in the image such that the face has specific features used(no hair but eyes, nose and mouth present) and resize the face to a 100 x 100 image

Note that this step uses LBP cascade classifier to detect faces in the image. Find related information here -

OpenCV Cascade Classifier tutorial

static Mat cropFaceFromImg(InputArray _frame)

{

    Mat testFrame = _frame.getMat();

    std::vector<Rect> faces;

    Mat frame_gray;

    std::vector<Mat> croppedFaces;

    Mat resizedFace;//dst image

    

    cvtColor( testFrame, frame_gray, CV_BGR2GRAY );

    //equalizeHist( frame_gray, frame_gray );

    

    //-- Detect faces

    face_cascade.detectMultiScale( frame_gray, faces, 1.1, 2, 0|CV_HAAR_SCALE_IMAGE, Size(1, 1) );

    

    for( size_t i = 0; i < faces.size(); i++ )

    {

        Mat faceImg;

        Rect rect = Rect( faces[i].x + (faces[i].width/6), faces[i].y , faces[i].width*2/3, faces[i].height ); // ROI rect in srcImg

        frame_gray(rect).copyTo(faceImg);

        Size size(100,100);//the dst image size,e.g.100x100

        

        resize(faceImg,resizedFace,size);//resize image

        croppedFaces.push_back(resizedFace);

        

    }

    //imshow("ResizedFace", resizedFace);

    //waitKey(0);

    return resizedFace;

}

3. Calculate histogram for the image.

The essence of this step is calculating a pixel with its neighboring 8 pixels and assigning a 8 bit binary value for that pixel H(I) based on whether the neighboring pixel is darker or lighter than the pixel at the center.
Related code to calculate histogram bins [From OpenCV src code]

//------------------------------------------------------------------------------

// cv::elbp

//------------------------------------------------------------------------------

template <typename _Tp> static

inline void elbp_(InputArray _src, OutputArray _dst, int radius, int neighbors) {

    //get matrices

    Mat src = _src.getMat();

    // allocate memory for result

    _dst.create(src.rows-2*radius, src.cols-2*radius, CV_32SC1);

    Mat dst = _dst.getMat();

    // zero

    dst.setTo(0);

    for(int n=0; n<neighbors; n++) {

        // sample points

        float x = static_cast<float>(radius * cos(2.0*CV_PI*n/static_cast<float>(neighbors)));

        float y = static_cast<float>(-radius * sin(2.0*CV_PI*n/static_cast<float>(neighbors)));

        // relative indices

        int fx = static_cast<int>(floor(x));

        int fy = static_cast<int>(floor(y));

        int cx = static_cast<int>(ceil(x));

        int cy = static_cast<int>(ceil(y));

        // fractional part

        float ty = y - fy;

        float tx = x - fx;

        // set interpolation weights

        float w1 = (1 - tx) * (1 - ty);

        float w2 =      tx  * (1 - ty);

        float w3 = (1 - tx) *      ty;

        float w4 =      tx  *      ty;

        // iterate through your data

        for(int i=radius; i < src.rows-radius;i++) {

            for(int j=radius;j < src.cols-radius;j++) {

                // calculate interpolated value

                float t = static_cast<float>(w1*src.at<_Tp>(i+fy,j+fx) + w2*src.at<_Tp>(i+fy,j+cx) + w3*src.at<_Tp>(i+cy,j+fx) + w4*src.at<_Tp>(i+cy,j+cx));

                // floating point precision, so check some machine-dependent epsilon

                dst.at<int>(i-radius,j-radius) += ((t > src.at<_Tp>(i,j)) || (std::abs(t-src.at<_Tp>(i,j)) < std::numeric_limits<float>::epsilon())) << n;

            }

        }

    }

}

4. Compare the histogram with the histogram of the input face image.(this is done similar to a type of Face Recognizer in OpenCV called Linear Binary Patterns Histogram)

For this step, OpenCV has four different types of Histogram compare methods.
I use the CV_COMP_CORREL method as it allows for a basic correlation done and its value does not change even if we reverse the compared photo with the input. The formula for the correlation of histogram bins is -


where
  

and N is the total number of historgram bins.
[From Histogram Comparison OpenCV tutorials]
The above formula gives a value of 1 if the images are same and its value is proportional to the similarity of the images.

5. If histogram correlation shows more than 0.6, the image can be considered belonging to the same person.

Pros :-
1. A single step classifier which can be made better as the search queries increase by combining data sets of similar face searches.
2. No face modeling done and hence relies on the simplicity of histogram creation and comparison.
3. Good for mobile platform analysis

Cons:-
1. A naive classifier of 0.6 threshold so chances to get false positives and false negatives are high.
2. No face modeling done so critical features of the face may be neglected due to the basic histogram approach.

Note:- This is still a naive way to filter faces but the algorithm can be made adaptive as with each iteration of searches, we can keep improve the accuracy by combining search results for the same person and filtering out the faces which do not fall into the intersection set.

The github repo(source code) for the above can be found here