Getting Dlib Face Landmark Detection working with OpenCV

Dlib has excellent Face Detection and Face Landmark Detection algorithms built-in. Its face detection is based on Histogram of Oriented Gradients (HOG) feature combined with a linear classifier, on a sliding window detection scheme (Ref. http://dlib.net/) and it provides pre-trained models for face landmark detection. It also provides handy utility functions like dlib.get_frontal_face_detector() to make our lives easier.

Dlib Face Landmark Detection in action

Note: Image used for testing is in the Public Domain - https://en.wikipedia.org/wiki/File:Arnold_Schwarzenegger_edit%28ws%29.jpg

To check out Dlib with it's native functions, you can try out the Dlib example from the official site: http://dlib.net/face_landmark_detection.py.html.It works well, but we can do better.

Although Dlib offers all the simplicity in implementing face landmark detection, it's still no match for the flexibility of OpenCV. (Simply put, Dlib is a library for Machine Learning, while OpenCV is for Computer Vision and Image Processing)

So, can we use Dlib face landmark detection functionality in an OpenCV context? Yes, here's how.

Installing Dlib on Anaconda Python on Windows

Update - 16/Dec/2019: There is now an easier way to install Dlib from the official pip package. Check out the new tutorial How to Build and Install the Latest Version of Dlib on Anaconda on Windows

Dlib is a Machine Learning library, primarily written in C++, but has a Python package also. It has many useful and optimized algorithms useful for machine learning, linear algebra, data structures, image processing, and much more available out-of-the-box.

"Dlib is a modern C++ toolkit containing machine learning algorithms and tools for creating complex software in C++ to solve real world problems. It is used in both industry and academia in a wide range of domains including robotics, embedded devices, mobile phones, and large high performance computing environments. Dlib's open source licensing allows you to use it in any application, free of charge." - dlib.net

One of the most popular features in Dlib is Facial Landmark Detection. Dlib installation ships with a pre-trained shape predictor model named shape_predictor_68_face_landmarks.dat, which as the name suggests, is trained to detect 68 facial keypoints including eyes, eyebrows, mouth, nose, face outline, etc.

Dlib's Facial Landmark Detection in action

You can view the sample code for face landmark detection here at the Dlib website, and download the pre-trained model from http://dlib.net/files/shape_predictor_68_face_landmarks.dat.bz2

(Make sure to unzip the .bz2 file once you download it)

Of course, Dlib is capable of much more than face landmark detection. I'm hoping to dig into some cool features of Dlib in later posts.

But first, we need to install it.

Getting Keras working with Anaconda Python

I've started using the Anaconda Python distribution for most of my Machine Learning. It has pre-built binaries of Python for many platforms and architectures, has hundreds of pre-built and tested Python packages directly available through the conda package manager, and it allows easy creation of virtual isolated environments - with its own Python version and packages - to experiment with.

You can get an idea of the capabilities of Anaconda by going through their Anaconda Test Drive guide.

Getting Keras (with Theano backend) working on any Python distribution is usually straightforward, but you do run into some errors occasionally based on the platform you're on and your environment settings.

So, here are the steps that worked for me to get Keras working on the Anaconda Python distribution:

First, you need to install Anaconda. It's as easy as getting the binary for your platform from Anaconda download page and running it. Once it's installed, the conda command will be available from your terminal or command prompt.

Now you can create an anaconda environment to install Keras and related packages,

 conda create --name keras-test numpy scipy scikit-learn pillow h5py mingw libpython  

'keras-test' is the name of the environment we're creating. You can give it a different name.
You can also create an environment with a different Python version. For example, if you want to create the environment with Python 2.7,

 conda create --name keras-test python=2.7 numpy scipy scikit-learn pillow h5py mingw libpython  

Once the environment is created, activate it.

 activate keras-test  

Then, we'll install Theano from Git, since we want the latest development version,

 pip install --upgrade --no-deps git+git://github.com/Theano/Theano.git  

And then, we install Keras from PIP,

 pip install keras  

Finally, we setup OpenBLAS and configure Theano to use it. My earlier blog post - Getting Theano working with OpenBLAS on Windows - details how to setup Theano with OpenBLAS in detail.

We can test whether the setup was successful by running the Python interpreter and importing Keras package,

 python  
 >>> import keras  
 Using Theano backend.  

Keras loading successfully

If you don't get any errors when the Keras package is loading, then all is set.

Related posts:
Switching between TensorFlow and Theano on Keras
What is the image_dim_ordering parameter in Keras, and why is it important

Related Links:
https://www.continuum.io/downloads
https://docs.continuum.io/anaconda/pkg-docs
http://conda.pydata.org/docs/test-drive.html

Build Deeper: Deep Learning Beginners' Guide is the ultimate guide for anyone taking their first step into Deep Learning.

Get your copy now!

Working Theano configs

Here are the Theano configurations that I have tested and worked.
These were tested on Windows 10 64-Bit, and Windows 7 64-Bit.
(I will update when I test on other OS's and setups)

With GPU support, on CUDA and cuDNN

In order to allow Theano to use the GPU, you need to be on a machine with a supported Nvidia GPU, and have the CUDA toolkit and cuDNN setup. I will cover how to setup CUDA on a different post.

 [global]  
 floatX = float32  
 device = gpu  
   
 [nvcc]  
 flags=-LC:\Users\Thimira\Anaconda3  
 compiler_bindir=C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin  
   
 [dnn]  
 enabled = True  
   
 [lib]  
 cnmem=0.75  
   
 [blas]   
 ldflags=-LC:\Dev_Tools\openblas\bin -lopenblas  

device = gpu tells Theano to use the GPU instead of the CPU.
flags=-LC:\Users\Thimira\Anaconda3 point this to your Python installation (I'm using Anaconda Python)
compiler_bindir=C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin point this to the bin dir of your Visual Studio installation (Note: CUDA only worked with Visual Studio 2013 for me)
[dnn] enabled = True this enables cuDNN
cnmem=0.75 set the memory limit Theano can use of the GPU. Here it's set to 75% of the GPU memory
ldflags=-LC:\Dev_Tools\openblas\bin -lopenblas point to your OpenBLAS installation. Refer to my earlier post Getting Theano working with OpenBLAS on Windows

With only CPU support

Since not everyone have a compatible Nvidia GPU to have CUDA.

 [global]  
 floatX = float32  
 device = cpu  
   
 [blas]  
 ldflags=-LC:\Dev_Tools\openblas\bin -lopenblas  

device = cpu tells Theano to use the CPU.
ldflags=-LC:\Dev_Tools\openblas\bin -lopenblas point to your OpenBLAS installation. Refer to my earlier post Getting Theano working with OpenBLAS on Windows

Build Deeper: Deep Learning Beginners' Guide is the ultimate guide for anyone taking their first step into Deep Learning.

Get your copy now!

Getting Theano working with OpenBLAS on Windows

I wanted to try out Machine Learning with Python, so my first choice was Keras with Theano.

Got Theano installed from Git (to get the latest development version):

 pip install --upgrade --no-deps git+git://github.com/Theano/Theano.git  

Then, I needed to setup Theano with OpenBLAS (otherwise, training Keras models was painfully slow).
Since I was on Windows, I had to look around for instructions on how to setup OpenBLAS properly.

Luckily, OpenBLAS provides binaries for Windows - both 32Bit and 64Bit - although, they may not be for the latest version of OpenBLAS.

Head over to https://sourceforge.net/projects/openblas/files/ and see which release has the binaries already built for Windows. We need both OpenBLAS and MinGW binaries.

At the time of this writing the latest version of OpenBLAS was v0.2.19, which unfortunately doesn't have the Windows binaries released.

But, going back a few releases, we find that the release v0.2.15 includes the binaries - OpenBLAS-v0.2.15-Win64-int32.zip and mingw64_dll.zip.

Download both of the Zip files, and first extract the OpenBLAS Zip to a globally accessible location on your hard disk. (I would suggest a location such as C:\Dev_Tools\openblas\).
Then, extract the mingw Zip, and copy it's contents to the bin directory of your extracted OpenBLAS directory. e.g. If you extracted OpenBLAS to C:\Dev_Tools\openblas\, then copy the contents (3 DLL files) of mingw to C:\Dev_Tools\openblas\bin\.
i.e.: The extracted openblas\bin will have the libopenblas.dll in it. When you extract mingw, it will have 3 more DLLs - libgcc_s_seh-1.dll, libgfortran-3.dll, libquadmath-0.dll. Copy those to openblas\bin also.

Then, add the openblas\bin directory to your system path.

Finally, edit (or create) your .theanorc file with the following settings: (assuming you extracted OpenBLAS to C:\Dev_Tools\openblas\)
Note: If you don't already have a .theanorc file, create a file named .theanorc in the home directory of your user account, e.g. C:\Users\<your user>\.theanorc

 [global]  
 floatX = float32  
 device = cpu  
   
 [blas]  
 ldflags=-LC:\Dev_Tools\openblas\bin -lopenblas  

Now, run your Keras/Theano program and see whether Theano picks up OpenBLAS.


Related Links:
http://www.openblas.net/

Build Deeper: Deep Learning Beginners' Guide is the ultimate guide for anyone taking their first step into Deep Learning.

Get your copy now!