Removing Red Eyes with OpenCV and Python

As I wrote in my blog yesterday, I started working on automatic red eye removal with python and OpenCV. Today, I finally managed to get it working (despite of the small amount of fever, which lead to some very interesting solutions :-P). The system works on the few pictures I have tested. Of course, it is not perfect, but should be at least somewhat acceptable.

The First part of the program is from the yesterday’s blog (eye detection), that needs to be run in order to get the information required for the red eye removal (yeah, you really need to have eyes recognized for this to work!), I have included the DetectRedEyes here, because I did some modification to it.

Detecting the Eyes

I’m not going to comment this much. Just to note, that I have removed drawing the rectangles around the eyes and replaced that with adding the information to an eyesList, which should contain the coordinates for the eyes in the big picture. I will skip to the changed eyes part…

def DetectEyes(image, faceCascade, eyeCascade):

“””

“””

“””

# Detect the eyes

eyes = cv.HaarDetectObjects(image, eyeCascade,
cv.CreateMemStorage(0),
haar_scale, min_neighbors,
haar_flags, (20,15))

eyesList = []

# If eyes were found
if eyes:

# For each eye found,
for eye in eyes:

# We save the locations on the BIG picture
# and append them to the eyesList

eyeX1 = eye[0][0]
eyeY1 = eye[0][1]
eyeX2 = eye[0][0] + eye[0][2]
eyeY2 = eye[0][1] + eye[0][3]

correctPosition = (eyeX1 + point1,
eyeY1 + point2,
eyeX2 + point1,
eyeY2 + point2)

eyesList.append(correctPosition)

# Finally, reset the image region of
# interest (otherwise this won’t be drawn correctly

cv.ResetImageROI(image)

return image, eyesList

Removing Red Eyes

def RemoveRedEyes(image, eyes, threshold=2.1):

# Check for existence of the eyes list
if eyes:

# Go through the eyes list [one eye at a time]
for position in eyes:

# Rename the positions, this is just for code readability
eyeX1 = position[0]
eyeY1 = position[1]
eyeX2 = position[2]
eyeY2 = position[3]

# Set the image Region of interest to be the eye area [this reduces processing time]
cv.SetImageROI(image, (eyeX1, eyeY1, eyeX2 – eyeX1, eyeY2 – eyeY1))

Okay, so first to the function. RemoveRedEyes(image, eyes, threshold=2.1). The image is the loaded up image in OpenCV, and the eyes is the list of the eyes in the DetectEyes(). Then We start going through the eyes list. The eyeX1, eyeY1, etc… are just used for making the code easier to read and understand. In the last row, we set the image region of interest to be the area of the eye in question. This reduces the processing time notably, because there is now only small amount of the image to be processed (not the complete 5+ Megapixels).

# Gets the image pixel i,j
# Then the channel value B,G,R order n with pixel[n]
# Get the size of the image (this time image means only the region of # interest, It has been “cropped” at the current time to be smaller)

sizeX, sizeY = cv.GetSize(image)
i = 0
j = 0

# Go through all the pixels in the image
while i < sizeX:
j = 0

while j < sizeY:

# This gets the pixel in question
pixel = cv.Get2D(image, i, j)

# Calculate the intensity compared to blue and green average
redIntensity = pixel[2] / ((pixel[1] + pixel[0]) / 2)

# If the red intensity is greater than 2.0, lower the value
if redIntensity >= threshold:

newRedValue = (pixel[1] + pixel[0]) / 2
# Insert the new red value for the pixel to the image
cv.Set2D(image, i, j,
cv.RGB(newRedValue, pixel[1], pixel[0]))

j += 1

i += 1

# Reset the image region of interest back to full image
cv.ResetImageROI(image)

# Return the resulting image
return image

Now, the hard part was here. After all, understanding that is the key to the SUCESSION! First, we get the size of the image with cv.GetSize(image). Now we use this information to go through all the pixels of the selected region.

pixel = cv.Get2D(image, i, j)

redIntensity = pixel[2] / ((pixel[1] + pixel[0]) / 2)

Here we load the (i,j) coordinate pixel and then calculate the intensity of the red in the picture. The pixel values in (at least current version of) openCV go here in order Blue, Green, Red. So pixel[0] means blue value, pixel[1] green value and pixel[2] means the red value.
Here we calculate the red intensity by dividing the red value by the average of blue and green. If the red is a lot greater than the other values, we certainly have more red. (And because we are talking of the eyes here, this shouldn’t be normal – at least I don’t know any red eyed people).

If redIntensity >= threshold:
newRedValue = (pixel[1] + pixel[0]) / 2
cv.Set2D(image, I, j,
cv.RGB(newRedValue, pixel[1], pixel[0]))

Here, if the red intensity of greater than the threshold given at the beginning (default = 2.0), we should do something. The new red value is given according to the average of the blue and green values, so it is scaled down to the same level. In the last row we just insert the new red value to the pixel in question to the big image.

Now, just run the whole thing:

if __name__ == “__main__”:

image, cascade, eyeCascade = Load()
image, eyes = DetectEyes(image, cascade, eyeCascade)
image2, cascade, eyeCascade = Load()
RemoveRedEyes(image2, eyes)
cv.ShowImage(“Changed image”, image2)
Display(image)

Some Results

So, here are some results of the changes I have managed to get. In this first picture, we can see that the baby has a little bit of red eyes left, because the threshold level is so high. If we would lower it a little bit, say to 1.5, the result would be following:

However, this will cause trouble with other pictures. For example the picture of me, which I used as an example yesterday has too much red color (well, it does not have red eyes, but it could have!). When we change the threshold here down to 1.5, the whole region of interest area gets a little bit messed up.

But Again, if we return back to the 2.1 (just my hunch), we don’t get this error.

Okay, so have fun testing this out. The source code with all the image files and haarcascades is included here.

http://dl.dropbox.com/u/4692161/red_eye_removal.tar.gz

Advertisements

Detecting Eyes With Python & OpenCv

Lately, I have been working on an image processing application for nokia’s n900. One part of the idea is to have automatic red eye detection and removal, to make everything easier for the user. My previous post was about getting the latest opencv version to work with python bindings on n900. I suggest checking it out, in case you are also interested on mobile development. A big thanks for all this goes to http://nashruddin.com/OpenCV_Eye_Detection

Starting out

First, lets start from the top. Import opencv and create a loader for the data.

import cv

def Load():

image = cv.LoadImage(“images/dallas.jpg”)
faceCascade = cv.Load(“../haarcascades/haarcascade_frontalface_alt.xml”)
eyeCascade = cv.Load(“../haarcascades/haarcascade_eye.xml”)
return (image, faceCascade, eyeCascade)

So, here we load the image, a haarcascade for the face recognition and a haarcascade for eye recognition. Then return all these for later use. After this, lets do the display and test everything works fine.

def Display(image):

cv.NameWindow(“Red Eye Test”)
cv.ShowImage(“Red Eye Test”, image)
cv.WaitKey(0)
cv.DestroyWindow(“Red Eye Test”)

Okay, so the Display takes image as a parameter (which has to be loaded before to cv). Then we just create a display window and show the image until the user presses any key. Now you should see an image of me when you just call these two files.

image, faceCascade, eyeCascade = Load()
Display(image)

Into the REAL business

Okay, so now it is time to go to the actual business! First, we should start with creating the detection function.

def DetectRedEyes(image, faceCascade, eyeCascade):

min_size = (20,20)
image_scale = 2
haar_scale = 1.2
min_neighbors = 2
haar_flags = 0

So far nothing very special has happened. In the beginning we just define the minimum size of the detected area (these relate to faces, so if the face is smaller than 20×20, it shouldn’t be detected), how small to scale the image, etc.

# Allocate the temporary images
gray = cv.CreateImage((image.width, image.height), 8, 1)
smallImage = cv.CreateImage((cv.Round(image.width / image_scale),
cv.Round (image.height / image_scale)), 8 ,1)

# Convert color input image to grayscale
cv.CvtColor(image, gray, cv.CV_BGR2GRAY)

# Scale input image for faster processing
cv.Resize(gray, smallImage, cv.CV_INTER_LINEAR)

# Equalize the histogram
cv.EqualizeHist(smallImage, smallImage)

In the first row here we create a gray image and a small image to make processing less resource intensive (the computer still is pretty good at recognizing stuff from smaller images!).

# Detect the faces
faces = cv.HaarDetectObjects(smallImage, faceCascade, cv.CreateMemStorage(0),
haar_scale, min_neighbors, haar_flags, min_size)

Now, here we detect the faces from the picture. We use HaarDetectObjects to find the faces from the smaller image we created beforehand. For more information about the HaarDetectObjects(), check http://opencv.willowgarage.com/documentation/python/object_detection.html#haardetectobjects

# If faces are found
if faces:

for ((x, y, w, h), n) in faces:
# the input to cv.HaarDetectObjects was resized, so scale the
# bounding box of each face and convert it to two CvPoints
pt1 = (int(x * image_scale), int(y * image_scale))
pt2 = (int((x + w) * image_scale), int((y + h) * image_scale))
cv.Rectangle(image, pt1, pt2, cv.RGB(255, 0, 0), 3, 8, 0)

Here We just detect the faces and draw rectangles around the found faces.

# Estimate the eyes position
# First, set the image region of interest
# The last division removes the lower part of the face to lower probability for false recognition
cv.SetImageROI(image, (pt1[0],
pt1[1],
pt2[0] – pt1[0],
int((pt2[1] – pt1[1]) * 0.6)))

Okay, so after finding the faces, we create a region of interest, from where the eyes should be found. The area is actually the same as the rectangle drawn, EXCEPT for the lower part of the image, where 1/3 of the face is left out (normally this means the mouth area). This is just to make the recognition a little bit faster and to lessen the amount of faulty eyes found. Underneath is a screenshot of what the image will look like after setting the region of interest.

Now, all that is left, is to repeat the same kind of operation that was done to the whole picture to find faces, but this time to find eyes from the region of interest.

# Detect the eyes
eyes = cv.HaarDetectObjects(image, eyeCascade,
cv.CreateMemStorage(0),
haar_scale, min_neighbors,
haar_flags, (20,15))

# If eyes were found
if eyes:

# For each eye found
for eye in eyes:

# Draw a rectangle around the eye
cv.Rectangle(image,
(eye[0][0],
eye[0][1]),
(eye[0][0] + eye[0][2],
eye[0][1] + eye[0][3]),
cv.RGB(255, 0, 0), 1, 8, 0)

# Finally, reset the image region of interest (otherwise this won’t
# be drawn correctly
cv.ResetImageROI(image)

This is the final and important part. If you do not call the ResetImageROI, the only thing that will be shown, is the small cropped (region of interest) area of the picture with red squares drawn around the found faces. So, it is important to remember to call this ResetImageROI(). All that is now left, is to return the image.

return image

Now, we should just add all these function calls to main and things should go just fine.

if __name__ == “__main__”:

image, faceCascade, eyeCascade = Load()
image = DetectRedEyes(image, faceCascade, eyeCascade)
Display(image)

This is what you should get as a result:

So, this concludes the part of detecting the eyes. In the next part I will add the possibility for removing the found red eyes (if there are any, in my example picture there are none).