Removing Red Eyes with OpenCV and Python

As I wrote in my blog yesterday, I started working on automatic red eye removal with python and OpenCV. Today, I finally managed to get it working (despite of the small amount of fever, which lead to some very interesting solutions :-P). The system works on the few pictures I have tested. Of course, it is not perfect, but should be at least somewhat acceptable.

The First part of the program is from the yesterday’s blog (eye detection), that needs to be run in order to get the information required for the red eye removal (yeah, you really need to have eyes recognized for this to work!), I have included the DetectRedEyes here, because I did some modification to it.

Detecting the Eyes

I’m not going to comment this much. Just to note, that I have removed drawing the rectangles around the eyes and replaced that with adding the information to an eyesList, which should contain the coordinates for the eyes in the big picture. I will skip to the changed eyes part…

def DetectEyes(image, faceCascade, eyeCascade):

“””

“””

“””

# Detect the eyes

eyes = cv.HaarDetectObjects(image, eyeCascade,
cv.CreateMemStorage(0),
haar_scale, min_neighbors,
haar_flags, (20,15))

eyesList = []

# If eyes were found
if eyes:

# For each eye found,
for eye in eyes:

# We save the locations on the BIG picture
# and append them to the eyesList

eyeX1 = eye[0][0]
eyeY1 = eye[0][1]
eyeX2 = eye[0][0] + eye[0][2]
eyeY2 = eye[0][1] + eye[0][3]

correctPosition = (eyeX1 + point1,
eyeY1 + point2,
eyeX2 + point1,
eyeY2 + point2)

eyesList.append(correctPosition)

# Finally, reset the image region of
# interest (otherwise this won’t be drawn correctly

cv.ResetImageROI(image)

return image, eyesList

Removing Red Eyes

def RemoveRedEyes(image, eyes, threshold=2.1):

# Check for existence of the eyes list
if eyes:

# Go through the eyes list [one eye at a time]
for position in eyes:

# Rename the positions, this is just for code readability
eyeX1 = position[0]
eyeY1 = position[1]
eyeX2 = position[2]
eyeY2 = position[3]

# Set the image Region of interest to be the eye area [this reduces processing time]
cv.SetImageROI(image, (eyeX1, eyeY1, eyeX2 – eyeX1, eyeY2 – eyeY1))

Okay, so first to the function. RemoveRedEyes(image, eyes, threshold=2.1). The image is the loaded up image in OpenCV, and the eyes is the list of the eyes in the DetectEyes(). Then We start going through the eyes list. The eyeX1, eyeY1, etc… are just used for making the code easier to read and understand. In the last row, we set the image region of interest to be the area of the eye in question. This reduces the processing time notably, because there is now only small amount of the image to be processed (not the complete 5+ Megapixels).

# Gets the image pixel i,j
# Then the channel value B,G,R order n with pixel[n]
# Get the size of the image (this time image means only the region of # interest, It has been “cropped” at the current time to be smaller)

sizeX, sizeY = cv.GetSize(image)
i = 0
j = 0

# Go through all the pixels in the image
while i < sizeX:
j = 0

while j < sizeY:

# This gets the pixel in question
pixel = cv.Get2D(image, i, j)

# Calculate the intensity compared to blue and green average
redIntensity = pixel[2] / ((pixel[1] + pixel[0]) / 2)

# If the red intensity is greater than 2.0, lower the value
if redIntensity >= threshold:

newRedValue = (pixel[1] + pixel[0]) / 2
# Insert the new red value for the pixel to the image
cv.Set2D(image, i, j,
cv.RGB(newRedValue, pixel[1], pixel[0]))

j += 1

i += 1

# Reset the image region of interest back to full image
cv.ResetImageROI(image)

# Return the resulting image
return image

Now, the hard part was here. After all, understanding that is the key to the SUCESSION! First, we get the size of the image with cv.GetSize(image). Now we use this information to go through all the pixels of the selected region.

pixel = cv.Get2D(image, i, j)

redIntensity = pixel[2] / ((pixel[1] + pixel[0]) / 2)

Here we load the (i,j) coordinate pixel and then calculate the intensity of the red in the picture. The pixel values in (at least current version of) openCV go here in order Blue, Green, Red. So pixel[0] means blue value, pixel[1] green value and pixel[2] means the red value.
Here we calculate the red intensity by dividing the red value by the average of blue and green. If the red is a lot greater than the other values, we certainly have more red. (And because we are talking of the eyes here, this shouldn’t be normal – at least I don’t know any red eyed people).

If redIntensity >= threshold:
newRedValue = (pixel[1] + pixel[0]) / 2
cv.Set2D(image, I, j,
cv.RGB(newRedValue, pixel[1], pixel[0]))

Here, if the red intensity of greater than the threshold given at the beginning (default = 2.0), we should do something. The new red value is given according to the average of the blue and green values, so it is scaled down to the same level. In the last row we just insert the new red value to the pixel in question to the big image.

Now, just run the whole thing:

if __name__ == “__main__”:

image, cascade, eyeCascade = Load()
image, eyes = DetectEyes(image, cascade, eyeCascade)
image2, cascade, eyeCascade = Load()
RemoveRedEyes(image2, eyes)
cv.ShowImage(“Changed image”, image2)
Display(image)

Some Results

So, here are some results of the changes I have managed to get. In this first picture, we can see that the baby has a little bit of red eyes left, because the threshold level is so high. If we would lower it a little bit, say to 1.5, the result would be following:

However, this will cause trouble with other pictures. For example the picture of me, which I used as an example yesterday has too much red color (well, it does not have red eyes, but it could have!). When we change the threshold here down to 1.5, the whole region of interest area gets a little bit messed up.

But Again, if we return back to the 2.1 (just my hunch), we don’t get this error.

Okay, so have fun testing this out. The source code with all the image files and haarcascades is included here.

http://dl.dropbox.com/u/4692161/red_eye_removal.tar.gz

Advertisements

Detecting Eyes With Python & OpenCv

Lately, I have been working on an image processing application for nokia’s n900. One part of the idea is to have automatic red eye detection and removal, to make everything easier for the user. My previous post was about getting the latest opencv version to work with python bindings on n900. I suggest checking it out, in case you are also interested on mobile development. A big thanks for all this goes to http://nashruddin.com/OpenCV_Eye_Detection

Starting out

First, lets start from the top. Import opencv and create a loader for the data.

import cv

def Load():

image = cv.LoadImage(“images/dallas.jpg”)
faceCascade = cv.Load(“../haarcascades/haarcascade_frontalface_alt.xml”)
eyeCascade = cv.Load(“../haarcascades/haarcascade_eye.xml”)
return (image, faceCascade, eyeCascade)

So, here we load the image, a haarcascade for the face recognition and a haarcascade for eye recognition. Then return all these for later use. After this, lets do the display and test everything works fine.

def Display(image):

cv.NameWindow(“Red Eye Test”)
cv.ShowImage(“Red Eye Test”, image)
cv.WaitKey(0)
cv.DestroyWindow(“Red Eye Test”)

Okay, so the Display takes image as a parameter (which has to be loaded before to cv). Then we just create a display window and show the image until the user presses any key. Now you should see an image of me when you just call these two files.

image, faceCascade, eyeCascade = Load()
Display(image)

Into the REAL business

Okay, so now it is time to go to the actual business! First, we should start with creating the detection function.

def DetectRedEyes(image, faceCascade, eyeCascade):

min_size = (20,20)
image_scale = 2
haar_scale = 1.2
min_neighbors = 2
haar_flags = 0

So far nothing very special has happened. In the beginning we just define the minimum size of the detected area (these relate to faces, so if the face is smaller than 20×20, it shouldn’t be detected), how small to scale the image, etc.

# Allocate the temporary images
gray = cv.CreateImage((image.width, image.height), 8, 1)
smallImage = cv.CreateImage((cv.Round(image.width / image_scale),
cv.Round (image.height / image_scale)), 8 ,1)

# Convert color input image to grayscale
cv.CvtColor(image, gray, cv.CV_BGR2GRAY)

# Scale input image for faster processing
cv.Resize(gray, smallImage, cv.CV_INTER_LINEAR)

# Equalize the histogram
cv.EqualizeHist(smallImage, smallImage)

In the first row here we create a gray image and a small image to make processing less resource intensive (the computer still is pretty good at recognizing stuff from smaller images!).

# Detect the faces
faces = cv.HaarDetectObjects(smallImage, faceCascade, cv.CreateMemStorage(0),
haar_scale, min_neighbors, haar_flags, min_size)

Now, here we detect the faces from the picture. We use HaarDetectObjects to find the faces from the smaller image we created beforehand. For more information about the HaarDetectObjects(), check http://opencv.willowgarage.com/documentation/python/object_detection.html#haardetectobjects

# If faces are found
if faces:

for ((x, y, w, h), n) in faces:
# the input to cv.HaarDetectObjects was resized, so scale the
# bounding box of each face and convert it to two CvPoints
pt1 = (int(x * image_scale), int(y * image_scale))
pt2 = (int((x + w) * image_scale), int((y + h) * image_scale))
cv.Rectangle(image, pt1, pt2, cv.RGB(255, 0, 0), 3, 8, 0)

Here We just detect the faces and draw rectangles around the found faces.

# Estimate the eyes position
# First, set the image region of interest
# The last division removes the lower part of the face to lower probability for false recognition
cv.SetImageROI(image, (pt1[0],
pt1[1],
pt2[0] – pt1[0],
int((pt2[1] – pt1[1]) * 0.6)))

Okay, so after finding the faces, we create a region of interest, from where the eyes should be found. The area is actually the same as the rectangle drawn, EXCEPT for the lower part of the image, where 1/3 of the face is left out (normally this means the mouth area). This is just to make the recognition a little bit faster and to lessen the amount of faulty eyes found. Underneath is a screenshot of what the image will look like after setting the region of interest.

Now, all that is left, is to repeat the same kind of operation that was done to the whole picture to find faces, but this time to find eyes from the region of interest.

# Detect the eyes
eyes = cv.HaarDetectObjects(image, eyeCascade,
cv.CreateMemStorage(0),
haar_scale, min_neighbors,
haar_flags, (20,15))

# If eyes were found
if eyes:

# For each eye found
for eye in eyes:

# Draw a rectangle around the eye
cv.Rectangle(image,
(eye[0][0],
eye[0][1]),
(eye[0][0] + eye[0][2],
eye[0][1] + eye[0][3]),
cv.RGB(255, 0, 0), 1, 8, 0)

# Finally, reset the image region of interest (otherwise this won’t
# be drawn correctly
cv.ResetImageROI(image)

This is the final and important part. If you do not call the ResetImageROI, the only thing that will be shown, is the small cropped (region of interest) area of the picture with red squares drawn around the found faces. So, it is important to remember to call this ResetImageROI(). All that is now left, is to return the image.

return image

Now, we should just add all these function calls to main and things should go just fine.

if __name__ == “__main__”:

image, faceCascade, eyeCascade = Load()
image = DetectRedEyes(image, faceCascade, eyeCascade)
Display(image)

This is what you should get as a result:

So, this concludes the part of detecting the eyes. In the next part I will add the possibility for removing the found red eyes (if there are any, in my example picture there are none).

OpenCV 2.1.0 with Python bindings on Maemo (n900)

Howdy How!

Lately I’ve been really busy with my job at the university. However, I decided to give a short moment of my time to share some fruits of my work. I have been always interested in playing with the Augmented Reality stuff. So I decided to challenge myself a little bit and get some tools to work on n900. I found a version from the repository (2.0.something), which was missing the Python bindings. Now that I have lately worked with python (and still am), I really want those bindings to work!

Well, it seemed that there is only one option, to compile and configure everything to work manually. And that is exactly what I did. Finding all this was a pretty hard, so I decided to post everything here on my blog. There are probably some strange things, because I am no real Linux expert (and this is the first time ever I have created .deb packages), but following my instructions you should get opencv to work on maemo (with the newest PR1.2 firmware). So, keep on reading.

How to get OpenCV to work on n900

The OpenCV needs to be build inside scratchbox and then deployed to the real n900 in question. The installation is based on the OpenCV installation instructions from the wiki: http://opencv.willowgarage.com/wiki/InstallGuide%20:%20Debian

I have built successfully the whole system from the stable package, not from the SVN (for some reason I wasn’t able to get cmake to compile from SVN)

NOTE: This can be built for both the X86 and the ARMEL. Just change to different scratchbox environment with sb-menu

If you want to try things on the emulator, you should compile the X86. Otherwise use ARMEL (so you can deploy it to the n900). Probably might be best to do both?

Working with the scratchbox

If you want to use the latest package, get subversion.

1. Install subversion

fakeroot apt-get install subversion

2. Install the rest of the prerequisites

apt-get install build-essential

apt-get install cmake

apt-get install pkg-config

apt-get install libpng12-0 libpng12-dev libpng++-dev libpng3

apt-get install libpnglite-dev ibpngwriter0-dev libpngwriter0c2

apt-get install zlib1g-dbg zlib1g zlib1g-dev

apt-get install libjasper-dev libjasper-runtime libjasper1

apt-get install pngtools libtiff4-dev libtiff4 libtiffxx0c2 libtiff-tools

apt-get install libjpeg8 libjpeg8-dev libjpeg8-dbg libjpeg-prog

apt-get install ffmpeg libavcodec-dev libavcodec52 libavformat52 libavformat-dev

apt-get install libgstreamer0.10-0-dbg libgstreamer0.10-0 libgstreamer0.10-dev

apt-get install libxine1-ffmpeg libxine-dev libxine1-bin

apt-get install libunicap2 libunicap2-dev

apt-get install libdc1394-22-dev libdc1394-22 libdc1394-utils

apt-get install swig

apt-get install libv4l-0 libv4l-dev

apt-get install python-numpy

3. Get the files and save them to your MyDocs scratchbox directory

(if default installation is done, this can be found in ubuntu from /scratchbox/users//home//MyDocs/

RECOMMENDED: download the latest stable Unix version

http://sourceforge.net/projects/opencvlibrary/files/

untar the file for example to OpenCV-2.1.0/ (the version I have tested this with)

or if you want to try your luck, from svn

https://code.ros.org/svn/opencv/tags/latest_tested_snapshot/

4. create a build directory (the only difference to that one, is that o is not a capital letter

mkdir opencv-2.1.0

cd opencv-2.1.0

5. Now, lets use cmake to generate the makefile

cmake -D CMAKE_BUILD_TYPE=RELEASE -D CMAKE_INSTALL_PREFIX=/usr/local -D BUILD_PYTHON_SUPPORT=ON -D BUILD_EXAMPLES=ON ../OpenCV-2.1.0

This will use the cmake to build the make file

6. Now, we must create the .deb package for the opencv

export DEBULLNAME=”Your name”

dh_make -e your.name@example.org –createorig

When asked for what kind of binary, just type s (= single)

After this you can edit the installation info etc. at ./debian (for example nano ./debian/control and just write a short description here)

7. Now, build the package (you have to be in the opencv-2.1.0/ folder

dpkg-buildpackage -rfakeroot

This will display a lot of output and you should see the build process in percentage running towards 100% (this will take some time, so grab a cup of coffee. For example check http://coffeeproject.com/shop/magento/index.php for some nice stuff)

8. If everything went okay, now you should be able to install the application from the .deb package

cd ..

fakeroot dpkg -i opencv-2.1.0-1_armel.deb

OR

fakeroot dpkg -i opencv-2.1.0-1_i386.deb

Now you should have the whole thing installed on the machine (now scratchbox environment). The problem here is, that python is not configured properly.

9. Configure library path

Because the .deb installer does not install all the libraries correctly (at least with these instructions), you have to configure the path manually.

Export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH

sudo ldconfig (OR fakeroot, if inside scratchbox)

10. Configure Python

Now, finally only one part is still missing! The thing we went all this trouble through, configuring the python to work. In n900 you can replace python2.5 with just basic python command

Python2.5 [in scratchbox python leads to 2.3 version of python]

>>>import sys

>>>print sys.path

>>>sys.path.append(“/usr/local/lib/python2.5/site-packages/

And after this everything should work just fine. Test it

11. Test OpenCV with python

>>>import cv

If no error comes up, this should work now 😀 You can test the installation in scratchbox by going to the original folder OpenCV-2.1.0/samples/python. Normally, when running on PC machine, the python test could be done for example by running python delaunay.py. If you try to run

python2.5 delaunay.py

You will get no error, but nothing will show on the screen (at least when run solely inside the scratchbox and no virtual machine). If you run the same thing in n900, it should display you some nice stuff 😀

Okay, that was it. Have fun tinkering with opencv and python. If you get any errors/bug/missing things, please, post here so we can try to fix those.

-Japskua