Last updated on July 3, 2021.
Last week we learned how to install and configure dlib on our system with Python bindings.
Today we are going to use dlib and OpenCV to detect facial landmarks in an image.
Facial landmarks are used to localize and represent salient regions of the face, such as:
- Eyes
- Eyebrows
- Nose
- Mouth
- Jawline
Facial landmarks have been successfully applied to face alignment, head pose estimation, face swapping, blink detection and much more.
In today’s blog post we’ll be focusing on the basics of facial landmarks, including:
- Exactly what facial landmarks are and how they work.
- How to detect and extract facial landmarks from an image using dlib, OpenCV, and Python.
In the next blog post in this series we’ll take a deeper dive into facial landmarks and learn how to extract specific facial regions based on these facial landmarks.
To learn more about facial landmarks, just keep reading.
- Update July 2021: Added section on alternative facial landmark detectors, including dlib’s 5-point facial landmark detector, OpenCV’s built-in facial landmark detector, and MediaPipe’s face mesh detector.
Facial landmarks with dlib, OpenCV, and Python
The first part of this blog post will discuss facial landmarks and why they are used in computer vision applications.
From there, I’ll demonstrate how to detect and extract facial landmarks using dlib, OpenCV, and Python.
Finally, we’ll look at some results of applying facial landmark detection to images.
What are facial landmarks?
Detecting facial landmarks is a subset of the shape prediction problem. Given an input image (and normally an ROI that specifies the object of interest), a shape predictor attempts to localize key points of interest along the shape.
In the context of facial landmarks, our goal is detect important facial structures on the face using shape prediction methods.
Detecting facial landmarks is therefore a two step process:
- Step #1: Localize the face in the image.
- Step #2: Detect the key facial structures on the face ROI.
Face detection (Step #1) can be achieved in a number of ways.
We could use OpenCV’s built-in Haar cascades.
We might apply a pre-trained HOG + Linear SVM object detector specifically for the task of face detection.
Or we might even use deep learning-based algorithms for face localization.
In either case, the actual algorithm used to detect the face in the image doesn’t matter. Instead, what’s important is that through some method we obtain the face bounding box (i.e., the (x, y)-coordinates of the face in the image).
Given the face region we can then apply Step #2: detecting key facial structures in the face region.
There are a variety of facial landmark detectors, but all methods essentially try to localize and label the following facial regions:
- Mouth
- Right eyebrow
- Left eyebrow
- Right eye
- Left eye
- Nose
- Jaw
The facial landmark detector included in the dlib library is an implementation of the One Millisecond Face Alignment with an Ensemble of Regression Trees paper by Kazemi and Sullivan (2014).
This method starts by using:
- A training set of labeled facial landmarks on an image. These images are manually labeled, specifying specific (x, y)-coordinates of regions surrounding each facial structure.
- Priors, of more specifically, the probability on distance between pairs of input pixels.
Given this training data, an ensemble of regression trees are trained to estimate the facial landmark positions directly from the pixel intensities themselves (i.e., no “feature extraction” is taking place).
The end result is a facial landmark detector that can be used to detect facial landmarks in real-time with high quality predictions.
For more information and details on this specific technique, be sure to read the paper by Kazemi and Sullivan linked to above, along with the official dlib announcement.
Understanding dlib’s facial landmark detector
The pre-trained facial landmark detector inside the dlib library is used to estimate the location of 68 (x, y)-coordinates that map to facial structures on the face.
The indexes of the 68 coordinates can be visualized on the image below:
These annotations are part of the 68 point iBUG 300-W dataset which the dlib facial landmark predictor was trained on.
It’s important to note that other flavors of facial landmark detectors exist, including the 194 point model that can be trained on the HELEN dataset.
Regardless of which dataset is used, the same dlib framework can be leveraged to train a shape predictor on the input training data — this is useful if you would like to train facial landmark detectors or custom shape predictors of your own.
In the remaining of this blog post I’ll demonstrate how to detect these facial landmarks in images.
Future blog posts in this series will use these facial landmarks to extract specific regions of the face, apply face alignment, and even build a blink detection system.
Detecting facial landmarks with dlib, OpenCV, and Python
In order to prepare for this series of blog posts on facial landmarks, I’ve added a few convenience functions to my imutils library, specifically inside face_utils.py.
We’ll be reviewing two of these functions inside face_utils.py now and the remaining ones next week.
The first utility function is rect_to_bb , short for “rectangle to bounding box”:
def rect_to_bb(rect): # take a bounding predicted by dlib and convert it # to the format (x, y, w, h) as we would normally do # with OpenCV x = rect.left() y = rect.top() w = rect.right() - x h = rect.bottom() - y # return a tuple of (x, y, w, h) return (x, y, w, h)
This function accepts a single argument, rect, which is assumed to be a bounding box rectangle produced by a dlib detector (i.e., the face detector).
The rect object includes the (x, y)-coordinates of the detection.
However, in OpenCV, we normally think of a bounding box in terms of “(x, y, width, height)” so as a matter of convenience, the rect_to_bb function takes this rect object and transforms it into a 4-tuple of coordinates.
Again, this is simply a matter of conveinence and taste.
Secondly, we have the shape_to_np function:
def shape_to_np(shape, dtype="int"): # initialize the list of (x, y)-coordinates coords = np.zeros((68, 2), dtype=dtype) # loop over the 68 facial landmarks and convert them # to a 2-tuple of (x, y)-coordinates for i in range(0, 68): coords[i] = (shape.part(i).x, shape.part(i).y) # return the list of (x, y)-coordinates return coords
The dlib face landmark detector will return a shape object containing the 68 (x, y)-coordinates of the facial landmark regions.
Using the shape_to_np function, we cam convert this object to a NumPy array, allowing it to “play nicer” with our Python code.
Given these two helper functions, we are now ready to detect facial landmarks in images.
Open up a new file, name it facial_landmarks.py , and insert the following code:
# import the necessary packages
from imutils import face_utils
import numpy as np
import argparse
import imutils
import dlib
import cv2
# construct the argument parser and parse the arguments
ap = argparse.ArgumentParser()
ap.add_argument("-p", "--shape-predictor", required=True,
help="path to facial landmark predictor")
ap.add_argument("-i", "--image", required=True,
help="path to input image")
args = vars(ap.parse_args())
Lines 2-7 import our required Python packages.
We’ll be using the face_utils submodule of imutils to access our helper functions detailed above.
We’ll then import dlib . If you don’t already have dlib installed on your system, please follow the instructions in my previous blog post to get your system properly configured.
Lines 10-15 parse our command line arguments:
--shape-predictor: This is the path to dlib’s pre-trained facial landmark detector. You can download the detector model here or you can use the “Downloads” section of this post to grab the code + example images + pre-trained detector as well.--image: The path to the input image that we want to detect facial landmarks on.
Now that our imports and command line arguments are taken care of, let’s initialize dlib’s face detector and facial landmark predictor:
# initialize dlib's face detector (HOG-based) and then create # the facial landmark predictor detector = dlib.get_frontal_face_detector() predictor = dlib.shape_predictor(args["shape_predictor"])
Line 19 initializes dlib’s pre-trained face detector based on a modification to the standard Histogram of Oriented Gradients + Linear SVM method for object detection.
Line 20 then loads the facial landmark predictor using the path to the supplied --shape-predictor.
But before we can actually detect facial landmarks, we first need to detect the face in our input image:
# load the input image, resize it, and convert it to grayscale image = cv2.imread(args["image"]) image = imutils.resize(image, width=500) gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY) # detect faces in the grayscale image rects = detector(gray, 1)
Line 23 loads our input image from disk via OpenCV, then pre-processes the image by resizing to have a width of 500 pixels and converting it to grayscale (Lines 24 and 25).
Line 28 handles detecting the bounding box of faces in our image.
The first parameter to the detector is our grayscale image (although this method can work with color images as well).
The second parameter is the number of image pyramid layers to apply when upscaling the image prior to applying the detector (this it the equivalent of computing cv2.pyrUp N number of times on the image).
The benefit of increasing the resolution of the input image prior to face detection is that it may allow us to detect more faces in the image — the downside is that the larger the input image, the more computaitonally expensive the detection process is.
Given the (x, y)-coordinates of the faces in the image, we can now apply facial landmark detection to each of the face regions:
# loop over the face detections
for (i, rect) in enumerate(rects):
# determine the facial landmarks for the face region, then
# convert the facial landmark (x, y)-coordinates to a NumPy
# array
shape = predictor(gray, rect)
shape = face_utils.shape_to_np(shape)
# convert dlib's rectangle to a OpenCV-style bounding box
# [i.e., (x, y, w, h)], then draw the face bounding box
(x, y, w, h) = face_utils.rect_to_bb(rect)
cv2.rectangle(image, (x, y), (x + w, y + h), (0, 255, 0), 2)
# show the face number
cv2.putText(image, "Face #{}".format(i + 1), (x - 10, y - 10),
cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 255, 0), 2)
# loop over the (x, y)-coordinates for the facial landmarks
# and draw them on the image
for (x, y) in shape:
cv2.circle(image, (x, y), 1, (0, 0, 255), -1)
# show the output image with the face detections + facial landmarks
cv2.imshow("Output", image)
cv2.waitKey(0)
We start looping over each of the face detections on Line 31.
For each of the face detections, we apply facial landmark detection on Line 35, giving us the 68 (x, y)-coordinates that map to the specific facial features in the image.
Line 36 then converts the dlib shape object to a NumPy array with shape (68, 2).
Lines 40 and 41 draw the bounding box surrounding the detected face on the image while Lines 44 and 45 draw the index of the face.
Finally, Lines 49 and 50 loop over the detected facial landmarks and draw each of them individually.
Lines 53 and 54 simply display the output image to our screen.
Facial landmark visualizations
Before we test our facial landmark detector, make sure you have upgraded to the latest version of imutils which includes the face_utils.py file:
$ pip install --upgrade imutils
Note: If you are using Python virtual environments, make sure you upgrade the imutils inside the virtual environment.
From there, use the “Downloads” section of this guide to download the source code, example images, and pre-trained dlib facial landmark detector.
Once you’ve downloaded the .zip archive, unzip it, change directory to facial-landmarks , and execute the following command:
$ python facial_landmarks.py --shape-predictor shape_predictor_68_face_landmarks.dat \ --image images/example_01.jpg
Notice how the bounding box of my face is drawn in green while each of the individual facial landmarks are drawn in red.
The same is true for this second example image:
$ python facial_landmarks.py --shape-predictor shape_predictor_68_face_landmarks.dat \ --image images/example_02.jpg
Here we can clearly see that the red circles map to specific facial features, including my jawline, mouth, nose, eyes, and eyebrows.
Let’s take a look at one final example, this time with multiple people in the image:
$ python facial_landmarks.py --shape-predictor shape_predictor_68_face_landmarks.dat \ --image images/example_03.jpg
For both people in the image (myself and Trisha, my fiancée), our faces are not only detected but also annotated via facial landmarks as well.
Alternative facial landmark detectors
Dlib’s 68-point facial landmark detector tends to be the most popular facial landmark detector in the computer vision field due to the speed and reliability of the dlib library.
However, other facial landmark detection models exist.
To start, dlib provides an alternative 5-point facial landmark detector that is faster than the 68-point variant. This model works great if all you need are the locations of the eyes and the nose.
One of the most popular new facial landmark detectors comes from the MediaPipe library which is capable of computing a 3D face mesh:
I’ll be doing tutorials on the PyImageSearch blog using MediaPipe and face mesh in the near future.
If you want to avoid using libraries other than OpenCV entirely (i.e., no dlib, MediaPipe, etc.), then it’s worth noting that OpenCV does support a built-in facial landmark detector; however, I have not used it before and I cannot comment on its accuracy, ease of use, etc.
What's next? We recommend PyImageSearch University.
86 total classes • 115+ hours of on-demand code walkthrough videos • Last updated: October 2024
★★★★★ 4.84 (128 Ratings) • 16,000+ Students Enrolled
I strongly believe that if you had the right teacher you could master computer vision and deep learning.
Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?
That’s not the case.
All you need to master computer vision and deep learning is for someone to explain things to you in simple, intuitive terms. And that’s exactly what I do. My mission is to change education and how complex Artificial Intelligence topics are taught.
If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to successfully and confidently apply computer vision to your work, research, and projects. Join me in computer vision mastery.
Inside PyImageSearch University you'll find:
- ✓ 86 courses on essential computer vision, deep learning, and OpenCV topics
- ✓ 86 Certificates of Completion
- ✓ 115+ hours of on-demand video
- ✓ Brand new courses released regularly, ensuring you can keep up with state-of-the-art techniques
- ✓ Pre-configured Jupyter Notebooks in Google Colab
- ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
- ✓ Access to centralized code repos for all 540+ tutorials on PyImageSearch
- ✓ Easy one-click downloads for code, datasets, pre-trained models, etc.
- ✓ Access on mobile, laptop, desktop, etc.
Summary
In today’s blog post we learned what facial landmarks are and how to detect them using dlib, OpenCV, and Python.
Detecting facial landmarks in an image is a two step process:
- First we must localize a face(s) in an image. This can be accomplished using a number of different techniques, but normally involve either Haar cascades or HOG + Linear SVM detectors (but any approach that produces a bounding box around the face will suffice).
- Apply the shape predictor, specifically a facial landmark detector, to obtain the (x, y)-coordinates of the face regions in the face ROI.
Given these facial landmarks we can apply a number of computer vision techniques, including:
- Face part extraction (i.e., nose, eyes, mouth, jawline, etc.)
- Facial alignment
- Head pose estimation
- Face swapping
- Blink detection
- …and much more!
In next week’s blog post I’ll be demonstrating how to access each of the face parts individually and extract the eyes, eyebrows, nose, mouth, and jawline features simply by using a bit of NumPy array slicing magic.
To be notified when this next blog post goes live, be sure to enter your email address in the form below!
Download the Source Code and FREE 17-page Resource Guide
Enter your email address below to get a .zip of the code and a FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning. Inside you'll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL!
Superrrrrbbbbbbbbbb blog .searched continuosly about facial landmarks.wellll explained.looking forward for face recongnition using facial landmark measurements……please go ahead soon..!!!!
awesome blog….!!!!!!
Thanks Anto!
thanks for providing such nice code and helping people.
Can you please guide me how to save extracted landmarks in .mat file so i could use it in matlab.
thanks
You should be able to use the savemat function in SciPy.
Good day, please how can I also save extracted landmarks so that i can use them in training a model in python. thanks
I would suggest you read up on basic file I/O with Python. CSV files would be of special interest to you.
It’s great!! Working very well, Thanks a lot!!
Thank you so much Adrian!!
Very good job Adrian.
All of your explanation are very useful.
This one, in special, is very important for my research.
Thank you a lot!!!
Thank you Mário! 🙂 I wish you the best of luck with your research.
This is a clear, clean, and EXCELLENT demystification of the concept.
Any plan to include this concept and the Deep learning version of training and implementation in your up coming Deep learning book?
Thanks Abkul, I’m glad you enjoyed the tutorial!
I don’t plan on covering how to train custom facial landmark detectors via deep learning inside Deep Learning for Computer Vision with Python, but I will consider it for a future tutorial.
This blog is a goldmine. Thank you so much for writing this.
I’m glad you’re enjoying the blog Dimitri, I’m happy to share 🙂
Hi Adrian,
This is a great article. Cant wait for next week’s article on how to access face features individually.
After some experimenting (and with your hint on array slicing), I managed to extract the features. Here is the method I used,
http://www.codesofinterest.com/2017/04/extracting-individual-facial-features-dlib.html
Undoubtedly, this method I used could use more improvements.
So, waiting for your article 🙂
Nice job Thimira. The method I will demonstrate in next week’s blog post is similar, but uses the
face_utilssub-module ofimutilsfor convenience. I’ll also be demonstrating how to draw semi-transparent overlays for each region of the face.Awesome! can’t wait for next weeks post 🙂
Hey Adrian,
I have already configured dlib with your previous week blog and now when i am trying to run “python facial_landmarks.py –shape-predictor shape_predictor_68_face_landmarks.dat \
–image images/example_01.jpg” command my ubuntu terminal is showing error
“python: can’t open file ‘facial_landmarks.py’ : [Errno 2] no such file or directory “.
PS : I have already downloaded your code and files and i am running my code inside that ‘facial-landmarks’ folder. All the files are present as well.
Dear Adrian,
Fixed the previous issue by providing the full path of py file. Thanks for this great blog.
Thanks and Regards
Neeraj Kumar
How did you managed to remove this path issue ?
Where do we have to specify the path ?
Dear Adrian,
I tried working for side faces but its not working, can you please guide what can be the possibilities for side face landmark detection and yes i was also trying working on your example_02.jpg there imutils.resize() method was giving some error.
Attribute error : ‘NoneType’ object has no attribute ‘shape’.
Thanks and Regards
Neeraj Kumar
If you’re getting a “NoneType” error it’s because you supplied an invalid path to the input image. You can read more about NoneType errors in this blog post.
Fixed Buddy. Thanks a ton.
can you please help me out with – how can i detect landmarks in video and compare with existing dataset of images.
I will be discussing landmark detection in video streams in next weeks blog post.
I hope next post you can use infrared camera
I don’t have any plans right now to cover IR cameras, but I’ll add it to my potential list of topics to cover.
Nice article Adrian! Btw shouldn’t the shape points in Figure 2 be 0 based?
Figure 2 was provided by the creators of the iBUG dataset. They likely used MATLAB which is 1-indexed rather than 0-indexed.
I also came across this. I have created an image and printed the index numbers as they are with dlib and python here: http://cvdrone.de/dlib-facial-landmark-detection.html
Isn’t Independent Component Analysis used to find local features of a face? How is that approach different from this? Advantages? Drawbacks?
Adrian, i’m a huge fan! i don’t know how to thank you for this.
I don’t know but i am having trouble running this code. It says that imutils package does not contain face_utils. I think it is not upgrading properly.
Make sure you run:
$ pip install --upgrade imutilsTo make sure you have the latest version of
imutilsinstalled. You can check which version is installed viapip freezeamazing adrian
I hope next time to show us how to recognize emotions on image
I won’t be using facial landmarks to recognize emotions, but I do cover how to recognize emotions via deep learning inside Deep Learning for Computer Vision with Python.
Hello Adrian,
Thank you for your awesome work.
You are really a good and skilled teacher. I could understand and practice a lot through your post. I am working on emotion detection and was wondering if the accuracy could be higher using Deep Learning than landmark approach.
And also concerning the landmark detection, do you think is worthy to use the 194 landmark points proposed with Helen dataset compare to the 68 landmark points?
Best regards.
Yes, deep learning-based methods will be better than facial landmark-based methods for emotion recognition. If you’d like to learn more about how to implement your own emotion recognition system, refer to Deep Learning for Computer Vision with Python.
Hi Adrian ,thanks for this great post
how dlib eye landmarks can be used to detect eye blinks ?
Hi Tony — I’ll be covering how to perform blink detection with facial landmarks and dlib in the next two weeks. Stay tuned!
May we have a tutorial about apply deep learning(CNN) using keras and tensorflow to classify some dataset, thanks sir, you are super!!!
I demonstrate how to use Keras for deep learning applied to MNIST in this post. I’ll also be discussing deep learning applied to custom datasets in lots of detail in my upcoming book, Deep Learning for Computer Vision with Python.
Hey Adrian,thanks for this blog post.IS there a way so that we can create our own custom shape predictor?
Yes, but you will have to use the dlib library + custom C++ code to train the custom shape predictor.
I’ve tried my best to play with the parameters of train_shape_predictor in dlib but the result is never as close as shape_predictor_68_face_landmarks.dat. data that I’ve used is custom data of face similar to that of ibug.
Hi Adrean,
I’m following your post about Facial landmarks with dlib, OpenCV, and Python. It’s really amazing. thank you a lot for such helpful and util code. So, now i’m tring to save all those landmarks in a file ! So that I’m ask you how cando such thing?
Thank you
Sure, that’s absolutely possible. I would use
picklefor this:f = open("landmarks.pickle", "wb") f = write(pickle.dumps(shape)) f.close()This will save the NumPy array
shapewhich contains the (x, y)-coordinates for each facial landmark to disk.Hello, Adrian ,the blog is very useful, thanks for this great blog.
Thank you Yang!
Hello Adrian
I was wondering if you did any tut. on face alignment too ? if so may you provide the link for me
thanks
Hi Ameer — I’ll be doing a blog post on facial alignment in late May/early June.
hey adrian,
i am trying to achieve real-time face – recognition usind dlib. but when i try using the above code with the .compute_face_descriptor(frame , shape) it gives an error that the arguments are not based on c++.please give me a solution sir
Hi Adrian ,
Thanks a lot for your explanation. It is very useful. However I’m newer in python and I’m trying to save those face landmarks in matrix so I can manipulate them instead of the original image. Would you give me some suggestion. How can I do such thing ?
Thank you Adrian
Once you have the
shaperepresenting the facial landmarks you can save them to disk via JSON, pickle, HDF5, etc. I would recommend pickle if you are new to Python. I would also suggest working through Practical Python and OpenCV so you can learn the fundamentals of computer vision and OpenCV.The things which you have discussed is awsome and it is very usefull for me . Actually im strucking a problem is to diferentiate from a person to person in a particular vedio and try to print those unique persons (i.e to find out the unique head counts from a vedio) im getting the duplicates ones can you please sort out my problrem.
Thank you in advance!!
I would suggest applying face recognition to uniquely count each individual.
Hello, i see you used dlib face/object detector for finding face on image transfer it from dlib.rectangle object to bouding values like your “rect_to_bb” funcition do and then with cv2 draw rectangle, but my problem is i need to use my own haar cascade for finding faces/objects and correct me if i am wrong there i need the exact opposite “bb_to_rect” because landmark detector require this rectangle, but i cant find out which is it datatype and how to reproduce this object, “[(386, 486) (461, 561)]” this sample of the found face its seems like 2 tuples but it doesnt, i cant event find out that while i was examining dlib c++ code, I spent on this problem more than 4 hours and with no result, is there any solution or it is approaching to be impossible?
I will look into this and see what I can find.
You can (sort of) implement bb_to_rect on your own. I wanted to do the same, and figured it out after some probing:
For example, if you want to align an image called “frame” whose face you have already detected as a rectangle with (x, y, w, h) values known, you can do it thus:
fa.align(frame, frame, dlib.rectangle(int(x), int(y), int(x + w), int(y + h)))
Here, I used a BGR image as “frame”, I had used
(x, y, w, h) = faceCascade.detectMultiScale(frame, scaleFactor=1.1, minNeighbors=5)
to detect face rectangle, having already defined faceCascade as:
faceCascade = cv2.CascadeClassifier(‘haarcascade_frontalface_default.xml’)
Thanks for sharing Vikram!
Hi,
I am student of you rather than saying a huge fan.
I have a small doubt we are getting coordinates, what are these coordinates represents.
If i want only a specific points in landmarks how can i get them
like
Nose tip
Left-eye left corner
Right-eye right corner
Corner of lips
These points required to estimate the pose of head,
Please help me this
If you want only specific facial landmark regions, please refer to this blog post which discusses the indexes of each of the facial landmark points (and how to extract them).
I decide to run your program again.
python ‘/home/…/facial_landmarks.py’ –shape-predictor ‘/home/…/shape_predictor_68_face_landmarks.dat’ –image ‘/home/…/30.jpg’
I have the error again. Now python see this dat file but I have the error anyway:
Error
Illegal instruction (core dumped)
What can I do?
It’s hard to say what the exact error is without having physical access to your machine, but I think the issue is that you may have compiled the dlib library against a different Python version than you are importing it into. Are you using Python virtual environments?
Yes, I use virtual environemnts. I reinstalled Ubunt 16.04. But anyway I have this error.
I have error in this line:
rects = detector(gray,1)
Output:
Illegal instruction (core dumped)
What can I do to solve this problem?
It sounds like you might have compiled dlib against a different version of Python than you are importing it into. Try uninstalling and re-installing dlib, taking care to ensure the same Python version is being used.
Fantastic stuff. Thanks for all you’ve done!
I am doing face detection / recognition on IR images. This means I cannot use the standard features for detection or recognition. I am trying to build my own detctor using the dlib “train_object_detector.py” and it is working really well – mostly.
I have a training set that are faces of one size and the detector is getting faces of similar sizes but completely missing smaller face images.
So my question is how does the system work to detect faces of different sizes. Do you need to have training samples of all the sizes that you expect to be finding? Or does the system take in the training images and resize them?
If you could clarify how this process works and what kind of training set I need and how it works to find faces of different sizes, I would really appreciate it. I have the recognizer working well, I just need to find the faces.
I am using the python dlib, specifically:
http://dlib.net/train_object_detector.py.html
Thank you, Jon Hauris
HOG-based object detectors are able to detect objects at various scales (provided the objects across the dataset have the same aspect ratio) using image pyramids. In short, you don’t need to worry about the size of your faces as they will be resized to a constant scale during training and then image pyramids applied when detecting on your testing images.
Hi Adrian,
That is nice tutorial for computer vision,
I am wondering: ‘image pyramid’ can use between dlib and opencv, then pass to HOG?
Most HOG-based detectors already include computing the image pyramid — both OpenCV and dlib do so it’s already built in.
Good afternoon, your tutorials are great.
I am developing a code for facial recognition but I am having difficulties in the matter of precision.
To get faster I made 37 proportions based on 12 points. But the variation of the face angle also varies the result. I’m limiting the angle 5 degrees to each side horizontally. I will now try to record the proportions for each angle ie a series of frames for the same face. Thus the comparison would be between several faces at the same angle.
If you can give me a light, I thank you.
Hi,
I got this error when trying to run your code:
usage: facial_landmarks.py [-h] -p SHAPE_PREDICTOR -i IMAGE
facial_landmarks.py: error: argument -p/–shape-predictor is required
An exception has occurred, use %tb to see the full traceback.
Can you help me please! thank you
Please read up on command line arguments before continuing. You need to supply the command line arguments to the script.
How to open shape_predictor_68_landmarks ? Please.
I’m not sure what you mean by “open” the file. You can use the “Downloads” section of this blog post to download the source code + shape_predictor_68_landmarks file.
Hi，Adrian. How to train shape_predictor_68_landmarks model?And do you have train demo?It’s perfect to have directions to train model.I want to train my model.Thanks!
I don’t have any demos of this, but you would need to refer to the dlib documentation on training a custom shape predictor.
Great article, thank you for you efforts.
Is there any way by which i can get more than 68 points of facial landmark. I see some paper mentioning 83 points or more. Is there any library that can help me find some points on forehead ? I am trying to find golden ratio for a face to score it. Thanks!
You would need to train your own custom shape predictor from scratch using dlib. The pre-trained facial landmark predictor provided by dlib only returns 68 points. The Helen dataset comes to mind as a good starting point.
Hi Mayank,
I’m facing the same issue! Would like to train a shape detector for 9 points along hairline. Have you found any solution?
Thanks!
Hi Mayank / Caroline Wang,
I know it is a long time since you left this comment.
Did either of you find a solution to this. I am ideally looking for a pre-trained model which detects hairline, along with other facial features as in the 68 point model.
Iain
Hey Chief,
Thank you so much. the post is so clear and works super cool.
Hi,do you have MATLAB or c++ cood for your work ?I try that use your code but l could not and I could not solve its error.
Thanks a lot
I only provide Python code here on the PyImageSearch blog. I do not have any MATLAB or C++ code.
Hi adrian…I’m having problem with the argument constructor, after giving the path to predictor and image when this line ” args = vars(ap.parse_args())” is executed in ipython console it is giving this error
“In [46]: args = vars(ap.parse_args())
usage: ipython [-h] -p SHAPE_PREDICTOR -i IMAGE
ipython: error: argument -p/–shape-predictor is required
An exception has occurred, use %tb to see the full traceback.
”
please help me with this problem
If you’re using ipython I would suggest leaving out the command line arguments and simply hardcoding the path to the shape predictor file.
I tried with the photo of man who has beard. It did not work well. How can we solve?
Hello, how are you?
I’ve been a big fan of your posts since I started working with python. Even more so when we made the post for Dlib on Raspberry. I was sad because you did not answer me and that you answered all the questions about your post.
I know I asked a very specific question but could answer me by email if that is the case. Even if it is a negative answer.
I do biometrics with haarcascade but I’m trying with landmarks.
I am developing a code for facial recognition but I am having difficulties in the matter of precision.
To get faster I made 37 proportions based on 12 points. But the variation of the face angle also varies. I’m limiting the angle 5 degrees to each side horizontally. I will now try to record the proportions for each angle ie a series of frames for the same face. Thus the comparison would be between several faces at the same angle.
Thank you for your attention.
bye
Hi Moises, I do my best to answer as many PyImageSearch comments as I possibly can, but please keep in mind that the PyImageSearch blog is very popular and receives 100’s of emails and comments per day. Again, I do the best I can, but I can’t always respond to them all.
That said, regarding your specific project (if I understand it correctly), you are performing face recognition but want to use facial landmarks to aid you in the aligning process to obtain better accuracy? Do I understand your question correctly? If so, why not just use the tutorial I provided on face alignment.
Thank you response.
About the project I will explain better.
My code makes reasons like this:
Face 1
Comparison 1 = straight (point1 to point2) / straight (point4 to point8) = “1,2”
Comparison 2 = straight (point3 to point4) / straight (point5 to point6) = “0.8”
Face 2
Comparison 1 = straight (point1 to point2) / straight (point4 to point8) = “1,6”
Comparison 2 = straight (point3 to point4) / straight (point5 to point6) = “1,0”
..
So if the face is straight the comparison is accurate. However if the face is crooked it does not work.
Example of a crooked face facing left:
.. …..
| Head straight but face turned to left
——
Straight face:
…. ….
| Head straight and face straight
——-
The turning of the face is not a problem because the comparison is made by the proportion of the lines. If the head is crooked but the face is straight the code works well.
I hope you have explained it better.
thank you
Keep in mind that the most facial landmarks and face recognition algorithms assume a frontal facing view of the person. Yes, facial landmarks can tolerate a small degree of rotation in the face (as it turns from side to side), but there becomes a point where the face is too angled to apply these techniques. I’m still not sure I understand your question entirely, but I think this might be the situation you are running into.
Hi Adrian,
Thanks for the wonderful tutorial. However if I wish to get a roi composed of eye lid, eye ball together, for example like in eye localization tasks where in the whole eye including continuous portion from eye lids to eye brows to eye is to be cropped, how do I do the same with facial landmarks code above
best regards
Tarun
I’m not sure I understand your question, but since facial landmarks can localize both eyes and eyebrows, you can simply compute the bounding box of both those regions and extract the ROI.
Great tutorial. Thank a lot!
However, I have one confounding problem:
When running the code at:
rects = detector(gray, 1) I get the following error:
__ TypeError: No registered converter was able to produce a C++ rvalue of type char from this Python object of type str __
I investigated the error, upgraded libboost-dev-all (using Ubuntu 17.04), still no resolution
What confounds me is; in a separate work for face recognition, I imported dlib initialized detector and set rects = detector(img , 1) etc…
It works fine.
Redid this exercise on python console (line by line)… error did not show up
Ran the program and the error turns up.
No spelling mistakes…
Any pointers, anything will help…
Thanks for your time
Hi Avi — I have to admit that I haven’t encountered this issue before so I’m not sure what the exact issue is. I would suggest posting the issue on the official dlib forums.
Hello Sir!
Thank you very much. All your explanations in this tutorial are very helpful.
https://pyimagesearch.com/2017/04/03/facial-landmarks-dlib-opencv-python/
I am currently studying at the national school of engineers of Sousse, coming from Tunisia. I use facial markers to recognize emotionsfrom the image. I try to create a template (model) from all the landmarks I extracted from the images in the CK + dataset. I advocate using this model to qualify the emotion of the face by using it.
So I wonder if you could help me and guide me how to save the facial landmarks in a training model and how could I predict this pattern to detect facial emotion from benchmarks.
Thank you for your help
Sincerly
Is there a particular reason why you are using facial landmarks to predict emotions? As I show in Deep Learning for Computer Vision with Python, CNNs can give higher accuracy for emotion recognition and are much easier to use in this context.
HI Wiem
Actualy I am working in facial emotion recogntion , but I have some defficulties in how to extract all the landmarks from the images and save them in a training mode .
Thanks
Again, I don’t recommend using facial landmarks to detect emotions. I would suggest training a CNN to recognize them, like I do inside Deep Learning for Computer Vision with Python.
hi , thanks for this great tutorial, please i want to crop the detected face . i tried the function crop but it doesn’t work.
please if you have any idea, help me.
thanks a lot.
You can drop out the face using NumPy array slicing:
face = image[startY:endY, startX:endX]I cover the basics of image processing, computer vision, and OpenCV (including how to crop images), inside my book, Practical Python and OpenCV. I suggest you start there. I hope that helps!
Hi Adrian, thanks for a useful website. I just tried your code on a movie in which I want to annotate faces (cf. your real time post, feeding in a video file instead of the webcam input).
When faces are detected, it works well, however the bottleneck is definitely face detection. Are there other “out of the box” solutions than the pre-trained HOG + Linear SVM object detector? You mention deep learning based approaches, is it something I could quickly deploy (i.e., are there pre-trained weights somewhere, and a pre-built network architecture, which would do a decent job?). Thank you for any hints!
Are you looking to simply increase the speed of the face detection? Use Haar cascades instead. They are less accurate than HOG + Linear SVM detectors, but much faster.
sorry, I wasn’t clear. The issue is that sometimes faces are not detected. I was wondering if there are other methods readily available that may give me hits where HOG+Linear SVM doesn’t. I am not concerned with speed, this is not a real-time project.
Sure, there are many face detectors that you can use. Haar cascades. Deep learning-based face detectors. You could even use a Google or Microsoft face detection API. It really depends on how far down the rabbit hole you want to go. Ideally, I would suggest gathering more training data that reflects the types of faces and environments the faces are in, then train your own custom face detector.
Hello Sir,
I am a huge fan of you.
You are doing wonder full work.
This code works very perfectly.
I want to know is how can I detect face if the image if the image is rotated by 90, 180 or 270 degrees?
And what if I can do this in a live video from the camera?
The HOG detector is not invariant to rotation so you’ll need to rotate your image by 90, 180, and 270 degrees and then apply the detection to each of the rotated images.
I cover how to apply facial landmarks to real-time video in this post.
Hi adrian, thanks for this page.
How can I implement face detection with dlib in a raspberry Pi ?
Face detection? Or face recognition? This post already covers how to perform face detection (i.e., detecting the location of a face in a given image).
Hi Adrian,
Is it possible to recognize the faces in live cam using OpenCV.I need your suggestions.Currently i am working on face recognization.
Yes, absolutely. I cover face recognition inside the PyImageSearch Gurus course. I would suggest starting there.
HI Andrian , thanks for this page, I want ask you if the precision of detected landmarks is related to the image size; because in this tutorial you change it into 500 ??
Cordialy
Wiem
No, the precision of detected landmarks is not related to the image size. I resized the image because it’s very rare we need to process images larger than 500-700 pixels along their largest dimension. Instead, we reduce the image size so that our algorithms run faster.
how I can do live facial landmark detection through video analysis
Please see this blog post.
Hello Adrian, have you written any blog on how to estimate head pose using facial landmarks? Please let me know. Cant seem to find any elegant solution.
Hi Tommy — I have not written a post on head pose estimation, but I will consider this for a future blog post topic.
This is one of the best articles ive ever come across!
Thanks Arfah!
Hi Adrian,
Thanks for this page. This is a great tutorial. I’m having one error running this code. When I run
python facial_landmarks.py –shape-predictor shape_predictor_68_face_landmarks.dat –image images/example_01.jpg
I’m getting
Illegal instruction (core dumped)
Hi Darshil — I’m sorry it isn’t running on your system. Can you try running this example program to see if it works?
Thank you Adrian for your reply. I tried running the code you sent, it is not giving any error but it is not giving any output either. I am using Ubuntu 14. Can I record and send you a video somewhere ,of what error I’m getting.
Hi Darshil.
cv2.imshow()should display the image on your computer screen. Alternatively you could usecv2.imwrite()to write the image to disk.Hi Adrian, thanks for your reply.
The code you sent is running without any error but it is not giving any output either. I’m using Ubuntu 14. I know what the error is, please help me with that. I tried running facial_landmarks.py, video_facial_landmarks.py, detect_blinks.py and drowsiness_detection.py and all the four are having same error. When the code execution reaches to “rects = detector(gray, 0)” in all the codes, execution stops and it shows “Illegal instruction (core dumped).
Hi Darshil — Did the dlib sample code I linked work without a hitch? Without being on your system, it is hard for me to debug from here. Try sending me an email with details about your configuration including OpenCV version, dlib version, and Python version.
I have dropped you an email. Please help
Am also facing error in parse.arg(). I havent left any space in middle still it shows ASSERTION ERROR
Hi Eyshika — please read up on command line arguments. You DO NOT need to edit any of the code.
Hey there. This is one of the simple and well written tutorial about facial landmarks.I have a question regarding landmarks.
Can we find distance between two landmarks? Suppose we want to find length of our lips so according to 68-landmarks detection we should find distance between 49 to 55. Is there anyway to do ?
Compute the Euclidean distance between the two points — this will give you the pixel distance. The distance in a measurable metric an be computed provided you’ve done a simple calibration.
Hi Adrian great work , can you please suggest a way as to quantify how well the classification or lip extraction is doing , I mean can we plot a Confusion matrix for the above script ????
looking for an early reply
You can evaluate the facial landmarks against a testing set; however, you cannot evaluate the performance without knowing the ground-truth locations of each facial coordinate.
Hi Adrian great work, thank you for you efforts.
I have a mini project about lip reading authentication.
Does you have an idea about extracting letters from an image ?.
cordially
Hi Adrian great work.
I have a project about Lip reading authentication with the method : Follow-up of lips movement by points of interest.
Well, your example help me very match in my project but does you have any idea about extracting a letter from the picture?
I don’t have any experience with lip reading, but I would suggest taking a look at this publication which discusses the topic.
Hi Adrian!
Great work! Thank you for running this blog!
I just tried the code and I am getting an error: facial_landmarks.py: error: the following arguments are required: -i/–image
and then
facial-landmarks>–image images/example_01.jpg
‘–image’ is not recognized as an internal or external command,
operable program or batch file.
Would you be able to help me to resolve it? I didn’t edited the code.
Hey Nick, please see my reply to “mux” (June 1, 2017). Your issue here is that you are not properly supplying the command line arguments. Open up a command line and then execute the following command, just like I do in the blog post:
You’ll want to make sure you are in the same directory as the
facial_landmarks.pyscript.Take a second to read up on how to use the command line and command line arguments and it will help out dramatically. I hope that helps!
Hey, I decided to plot the x-y coordinates from the np array. They appear to be upside down on a scatter plot. Any ideas why?
How did you plot them?
Hey Adrian !
First I want to thank you for this tutorial, it really helped a lot though I’m using C++ and Visual Studio !
I wanted to ask you if it’s possible to detect only eyes using this algorithm ? I mean, I have a camera which can only take video of 2 eyes and the nose, not the entire face :/
I tried using the face landmark detection in those conditions but it just doesn’t go well !
Do you have any ideas on how I can manage to get through this ?
Thank you !
Unfortunately, no. You need to detect the face first in order to localize the eye region. You may consider applying an eye detector, such as OpenCV’s Haar cascades — but then you still need to localize the landmarks of the eye. This would likely entail training your own eye landmark detector.
Hi, your work is good. I want to ask a question I work with Dlib I want to train image when I detect facial landmarks. Can I train image when I use dlib and teach a computer how find landmarks with Dlib it is possible??????
Yes, you can use dlib to train your own shape predictors. I don’t have any tutorials on this but includes an example on it.
hi Adrian Rosebrock,
i want to detect the forehead also. How can i do it?and i want to landmark the whole face components.please advice me..thanks your contribution really work 🙂
If you can detect the face you can use heuristics to detect the forehead. For example, if a forehead is approximately 30-40% as tall as the rest of the face you can first detect the face, compute the bounding box, and then use the bounding box to derive the forehead region based on this percentage.
Thanks Adrian. I am following both of this articles 1. https://pyimagesearch.com/2017/04/03/facial-landmarks-dlib-opencv-python/
and 2. https://pyimagesearch.com/2017/04/10/detect-eyes-nose-lips-jaw-dlib-opencv-python/
my question is if i want to add detect facial landmark first then Detect eyes, nose, lips, and jaw in a single frame what should i do? and also i wanted to know that is it a semantic annotation methodology according to your reference ? https://ibug.doc.ic.ac.uk/resources/facial-point-annotations/
thanks in Advance
You mentioned reading this tutorial but your question is solved by it. If you apply the method you can first detect the face and then find the eyes, nose, lips, and jaw. Again, the tutorial covers your exact question.
Thank you so much
Hi, thank you for the tutorials, big help for my proyect. I have a question, why do you resize the input image? The face detector works better with 500 pixels width or something?
Thanks in advance
We typically don’t process images/frames that are larger than 600-800px across their maximum dimension. The less data there is, the faster the detection algorithms run. While high resolution images are appealing to the human eye they can slow down the pipeline and actually hurt detection/classification performance.
Nice work ! I am currently doing project where subjects in image are around 60 in no and they are at certain distance…but using this code I am only able to detect faces which are nearer.Have you set any constrictions in .dat file?
Try increasing the image pyramid value for
detector(gray, 1)from “1” to “2” or “3”. The code will take longer to run since more image pyramids need to be evaluated but you should be able to detect faces farther away.Hi Adrian,
Thanks for the Face Detector, It works well. Can you please let me know, Can the Face Detector be used to identify faces in CCTV and Faces at 10 meter distances. Please enlighten me.
Thanks
Guru
That would really depend on the quality of the CCTV stream. I would need to see examples of the stream to provide any other suggestions.
Thanks a ton, I will send you the video stream by Monday.
Hi Adrian,
Please find the youtube video, Can you please let me know, If DLIB Face Dedector can find faces for CCTV in this resolution
https://www.youtube.com/watch?v=iSeMVR42thE
Can you please let me know, If I have perform any special steps to get the faces detected.
OpenCV includes Haar cascades which can be used for face detection. Dlib provides face detection as well. I’ll be doing another tutorial on a more advanced, more accurate face detector in a couple weeks. Have you tried applying any of these methods to your videos?
Hello Adrian,
Thank you so much for your tutorials, you’ve really been a great support for my academic projects. I have some questions to ask you about head pose estimation, my goal is to draw a lazer from eyes like Superman on the image plane to find out if the person is watching a screeen or not. I want it to apply the method of head pose estimation on the eye but it’s not precise. Can you please help me ? Thank you so much
I don’t have any tutorials on head pose or eye gaze estimation, but I would suggest starting with this guide as I know other readers have had good luck with it.
Hello Adrian
Thank you so much for your tutorials, you did a really great job. We are students that interest in Opencv and face recogition. We have been following all your steps and now we can get the facial landmarks in our example pictures.In this point we want to know how the coordinates facial landmarks can tell us whose face this is because now we are doing the face recogniton project that webcam can detect the customer’s face and it can give us some information such as name,age,gender but now we just want people’s name. We hope you understand what we ask,sorry for our bad english. Thank you so much
Hi Joe, thanks for the comment (and no worries, your English is very understandable). While (2D) facial landmarks can be used for facial recognition we normally use dedicated facial recognition algorithms for 2D face recognition, including Eigenfaces, Fisherfaces, and LBPs for face recognition. Deep learning-based embeddings can also be used as well. In that case, I would suggest working through either the PyImageSearch Gurus course or Deep Learning for Computer Vision with Python where I cover face recognition and deep learning in detail. I hope that helps point you in the right direction!
Hello! Thanks for this blog, it helps a lot!! But I have an issue
I put this in the terminal (Im using windows 10 btw), all in one line:
python facial_landmarks.py –shape-predictor shape_predictor_68_face_landmarks.dat image img1.jpg
and img1.jpg its in the same folder that facial_landmarks.py
I get error: unrecognized arguments: –image img1.jpg
What I am doing wrong!?
Thanks!
I wrote a blog post on Python + command line arguments. Give it a read and see if it helps resolve the problem.
https://github.com/jrosebr1/imutils/blob/master/imutils/face_utils.py link is not working .
The face_utils.py file was refactored into a proper sub-module:
https://github.com/jrosebr1/imutils/tree/master/imutils/face_utils
Hey, i think current error is
…
image = cv2.imread(args[“image”])
KeyError: ‘image
I’m so sorry, I have already solved my problem.
I like your tutorial so much. It is so helpful for my project.
Thank you~
Congrats on resolving the issue, Loralin. For anyone else who is having this issue I would suggest you:
1. Use the “Downloads” section of this blog post to download the code rather than copying and pasting
2. Read this blog post on command line arguments
Hi Adrian,
Can I use these facial landmarks as features for facial recognition task?
Thanks
Typically no. We would take a detected face and then apply a dedicated facial recognition algorithm such as Eigenfaces, Fisher faces, LBPs for face recognition, or utilize a deep learning facial embedding to generate a vector to quantify the post. The raw (x, y)-coordinates by themselves are not very helpful for facial recognition.
Hi Adrian,
Thanks for this tutorial! It kicked off my deep dive into CV!
I was curious if you knew of any open source libraries for localizing the center of the eyes as well? ie, the pupil location? I tried using contours and finding centroids, but it’s really inaccurate in different lighting. I thought something with HOG or CNNs might be the best way to get good accuracy.
Even better if it’s ok to use potentially for commercial use, thanks Adrian!
Eric
I haven’t tried this method but I know other PyImageSearch readers have had good luck with it.
Hi Adrian,
Thanks for the tutorial and explaination
Can you use dlib to detect hair?
I’m new to dlib
Thanks before
Pardon my english
No, dlib cannot be used for hair detection. It’s actually not an easy problem. You may want to look into semantic segmentation algorithms.
hi Adrian Rosebrock, thanks for your tutorial. that’s helpful.i have some issues in your code. Can you please give me advice.? When an image with closed eye, it will face a problem, it doesn’t crop the eye portion, because the landmark of an opened eye was predefined.so when a closed eye image is given it will not give proper result. i want to crop eye section whether it close or open. the total eye section of human face.please help me out.
The landmark here is dynamic. The eye landmarks will move as the eye opens and closes. It’s why this post on blink detection works. If I understand correctly you would like to crop the eye region regardless if the eye is open or closed?
yes, Sir I wanted to crop the section after a facial landmark detector, that means it will extract the region from ROI after dividing the facial landmark portion
Thanks a lot
Hi Adrian,
How come the landmark detector is able to detect features outside of the bounding box? e.g. in your pictures, the chin and ears are often outside of the bounding box, and yet the landmark detector is still able to find those features.
The bounding box gives context to the actual facial landmark predictor. Based on this context the predictor fits its own model to the face region.
this code is working in windows?
Once you have OpenCV and dlib properly installed, yes, this code will work on Windows.
Thanks Adrian for this great tutorial. Awesome Blog!!
Thank you, Abubakar! I really appreciate that 🙂
Hi Adrian, Thanks for the great tutorial! I have a question regarding the performance of face detection (just getting the bounding boxes). Dlib’s get_frontal_face_detector() seems quite slow (Only 10 FPS on 480p video on a good machine, even slower on my laptop). Since we’re making realtime applications, I wonder if this speed is normal, or am I missing something? I’ve also tried openCV’s haar cascades, but these are not good at detecting even slightly tilted/turned faces. So do you know if I can tweak Dlib to be faster, or is there an alternative for robust face detection in python? Thank you very much.
Dlib’s method uses HOG + Linear SVM for object detection which can be a bit slow. The deep learning face detectors are even more accurate, but also slower. One trick is to simply resize the image to be as small as possible without sacrificing accuracy. The less data there is to process, the faster your pipeline will run.
Is there any implementation of head pose estimation using these facial landmarks?
I do not have any tutorials on head pose estimation but I will certainly consider it for a future topic. Thanks so much for the recommendation!
Hey Adrian. Thanks for this tutorial. I wanna know if it’s possible to make an exe file from this code so that it could be run on any other PC without installing dlib, opencv or having shape predictor file?
If it’s possible can you tell me how?
In short, not easily, especially if you’re planning on using Python. I do not recommend it.
Hello! thank you a lot for all your blogs they are very useful for us. The other time I used the 86 Landmarks from Dlib Library to do emotion recognition and it works very well. Now I am trying to ameliorate this system and add a new thing wish is “Emotion neutralisation ” so like that the system had to do emotion recognition and face recognition at the same time. I thought of using the same model “shape_predictor_68_face_landmarks.dat” that I used to do emotion recognition with the classifier SVM. However, I need your advice on that!! How should I do the classification of the dataset? And, do I need more than the Dlib Library, OpenCV like “Tensorlow” to do the Face recognition?
Hey Wiem, thank you for the kind words, I appreciate it. You actually don’t need facial landmarks to perform emotion recognition, you can train a CNN to perform emotion recognition instead. In fact, I cover emotion recognition inside my book, Deep Learning for Computer Vision with Python. Be sure to take a look!
Thank you Adrian for your suggestion. I appreciate it. Actually, I used facial landmarks to do emotion recognition and it works well. Now, I am trying to ameliorate the system to do also the identification of the face detected using the facial Landmarks too. So, I am asking if it is applicable to do two types of training on one system (training to do emotion recognition + training to do face recognition) so like that it will return to me in the prediction the emotion and the identity of the face detected.
Any help from you would be great.
Thank you Adrian in advance
I wouldn’t recommend trying to combine emotion recognition and face recognition inside a single model. You should use two separate models here. This tutorial on face recognition will help you get started.
Thanks. I always appreciate your great works!
I found out that you publish the book.
Is that book include the content of this article and Face alignment that you posted? (https://pyimagesearch.com/2017/05/22/face-alignment-with-opencv-and-python/)
If so, I really want to buy that book or more like pdf that I want to read it in my tablet.
If not, i want at least table of content in your book.
I’ve actually published a handful of books and courses. If you’re specifically interested face recognition I would actually recommend you work through the PyImageSearch Gurus course where I cover facial recognition in detail.
Hey, Recently I was using dlib for face detection. I came to notice that it didn’t detect the faces in many cases when the face of a person with a dark skin was being provided during testing.
What are the ways one can overcome this problem?
Like training on those dataset where most of the image are of dark colored skinned person?
Or is there any other way to avoid the biased behaviour of the pre-trained dlib face detector?
Any machine learning model is only as good as its training data. Unfortunately, it sounds like there were not as many dark skinned people in the dataset OR the environment itself is quite different. To obtain the best results possible I would suggest you train your own custom object detector on images that you expect your model to be predicting on.
usage: facial_landmarks.py [-h] -p SHAPE_PREDICTOR -i IMAGE
facial_landmarks.py: error: the following arguments are required: -p/–shape-predictor, -i/–image
I got this error when i implement your codes
please help me to get rid of this
It’s okay if you are new to Python and command line arguments but you need to read up on them before you continue. Take the time to read the linked to article and you’ll be up and running 🙂
thank’s andrian for your blog, I have searched many sites about how do dlib facial landmarks work, but I can’t find any explain or flowchart, please let me know where I can find or learn how does Dlib facial landmark work
What specifically regarding the facial landmark algorithm are you trying to understand?
how do facial landmarks determine landmark facial points from the results of face detection that has been done before ?
Proccess facial landmark,
image -> face detection -> facial landmark -> Result
i trying to understand how shape predictor facial landmark proccessing the result of face detection
You’ll want to read the paper One Millisecond Face Alignment with an Ensemble of Regression Trees
Hi Adrian,
Thank you for your insightful work and helping everyone understand computer vision!
I am prototyping a stroke evaluator and would like to map relevancy of the points in figure 2 to each other.
I have a question about the shape array on line 39 (of the facial_landmarks.py code) in reference to the picture in figure 2. Are the x,y coordinates in the shape array in order with the points in figure 2? So would 16th position of the shape array (since it starts at 0) be the 17th point in figure 2?
Thank you for time and your help.
Regards,
Jeff
Keep in mind that Python is zero-indexed while the figure itself is one-indexed. You can very each point by looping over them individually if you like, but otherwise the array slices are correct. You can see how to visualize each facial structure in this post.
Hello Adrian,
Thanks for the amazing content published here.
I have a question regarding the facial landmarks.
We are trying to detect yawning, so basically we detect if the mouth is wide open for more than 3 seconds.
Our struggle is detecting facial landmarks for men with heavy dark beard directly below the mouth. Google “jon hamm heavy beard” to get an idea. The usual detection works fine, until the man opens his mouth, the current model will not consider that mouth open, and the whole logic will fail.
Any ideas dear to enhance the facial landmarks?
One I am thinking about is the training process, doing it again. But I guess it might be a nightmare since we will have to label each image with 68 landmark, and we need to be as precise and accurate as the original guys who labeled the original data set.
What do you think?
Thanks 🙂
Weird, I’ve never encountered that before! That’s certainly problematic. My guess is that most of the people the model was trained on didn’t have a beard, or at least that heavy of a beard. Perhaps trying to train a model with more bearded-people to increase accuracy? As you mentioned it would be extremely time consuming and tedious but keep in mind an ML model is only as good as the data it was trained on.
Thanks Adrian, I appreciate your feedback.
Best 🙂
In this lesson, You refer to:
>In order to prepare for this series of blog posts on facial landmarks, I’ve added a few >convenience functions to my imutils library, specifically inside face_utils.py.
The link is bad, it goes to a 404 error in GitHub page it refers to.
Here is a direct link to the page on GitHub.
Again, I respectfully request an option that allows me to view the comments (Most recent first). Also, is there an option that allows me to be notified via Email when I get a response to a comment I have made?
These two features would greatly increase my experience with the lessons.
Thanx
John
Hi Adrian
Thanks for such an informative article.I one query here:
The dlib’s “get_frontal_face_detector” detect the face even if we hide the “CHIN” with our hand. I can even share the pic with you over google drive if you want.
Is there any way to avoid it So, that it doesn’t detect a face when “CHIN” is not visible.
Regards
Akhil
It’s actually a desirable property of object detectors to work under occlusion.
Great tutorial sir.
I loved it
Thanks, I’m glad you found it helpful!
hi sir, thank you very much for your great tutorials that are very useful.
I downloaded your code and followed the instructions that you mentioned but i faced the problem of getting out of the program at the execution and the emergence of a message window:
python has stopped working
windows can check online solution to the problem and two choice of check online solution or close the program.
can you please help me with the problem.
What version of Python are you using? It sounds like your local machine is configured incorrectly.
Hi,
Congratulations for your great work. What I want to do is determine the angle of the jawline as shown here:
https://face-shape.com/img/round-angled-jaw.jpg
https://face-shape.com/img/find-jawline-angle.jpg
Given the indexes of the 68 coordinates(and especially for this case 2-16 lets say) do you think that it has to do more with a mathematical approach or do I need to train some shape predictor on my own? In any case, could you give me some starting points to cope with this issue?
Thank you
Great question. I would suggest starting with the 68 coordinates first. Don’t train your own model if you don’t have to. Start with pre-existing models first. Specifically, look into fitting a parabola to the facial landmark coordinates.
I was able to determine whether jaw is angular or not from the coordinates, I built this app, you can try it out to see how it performs: https://www.faceshapeapp.com.
Cool, nice job!
I want to create real time virtual makeover with python and opencv(it should apply lipstick when i select lipstick then other products in same manner)
Can u list down the steps so that I can get started with it
Thanks, Adrian. Great post. After walking through the examples, I tried some tests with photos of my own. with similar good results. On one of the photos, one of five faces was not found, even though it’s a standard, forward-facing group photo with no occlusion. Trying to understand what factors could cause this “miss” with the search.
You might want to try a deep learning-based face detector for higher accuracy.
Thanks for such a great blog, i just wanted to know if there is any way to only extract a certain part of face like only the right eye or left eye or mouth ?
Yes, see this tutorial.
Thank you! for the post
What are the “.dat” files and what program did you use to make them ,how can I modify them please explain
You cannot modify the .dat files. They are pre-trained machine learning files. They were generated by the dlib library. They cannot be edited.
Thanks for such a great blog, i just wanted to know if there is any way to detect other parts as well like ear and neck??
Sorry, I do not know of any landmark models for specifically the ear or neck.
Thanks for sharing the code. Can you tell me how to view the .dat extension file.
The .dat file is generated by dlib. The dlib library was used to create the actual facial landmark detector. The .dat file is imply the serialized model.
Hi, Adrian.
It is such a good and helpful post.
Can i access the coordinate of all 68 points?
Thank you.
Yes, see this tutorial of doing exactly that.
Thank you Adrian! I am learning a lot. This is like gold, I want to develop some ideas!
Thanks so much for the kind words, Gaby!
Hi adrian, Is there any way to identify chest in the body?
Take a look at instance segmentation along with pose estimation algorithms.
Hi Adrian,
Do you have blog post on this detecting chest in the human body?
Sir, i am doing a project using this method,but the problem is that it is detecting face of others who are walking near me…… how could i get only one face(my face) that is close to the camera without taking any others face
A simple heuristic would be to find the bounding box with the largest area. Since your face is closet to the camera the bounding box will be the largest.
thank you so mush but i need help
how extract 68 keypoints, with coordinates (x, y), for that face in python
You mean extract the regions of each facial structure? If so, see this post.
I’m using other face recognition to crop out the face. How can I do ‘If Else checking’ when the landmark is being block or face side ways by using the landmark? Because my mouth is being block it’s still able to predict the facial landmark.
The facial landmark detect, given a bounding box of a face, will do it’s best to estimate the facial landmark locations, even if the face or facial structures are partially occluded.
Hi Adrian,
can you suggest me a way to detect whether an important feature (mouth, nose and eyes) is being block?
Hello teacher
I have read the article carefully but because I come from another country, I don’t understand the ideas very well. I have some questions:
1st: How can Dlib know the position of the corner of the eye, nose and chin.
2st: after HOG is extracted from ROI, will SMV classify which HOG is the face compared to the HOG of the house, door, tree?
3st: Does Dlib predict the location of eye landmarks based on HOG extraction?
HI sir,
I am your big fan. I am trying to find a way to calculate the distance between eyes. Maybe between nose and mouth. I have the coordinates for each features like “eye” and “mouth”
Is any way that I can get the distance mapping to the real dimensions like “inches”.
for instance, 5 inches between eyes.
I need the value of real distance for some purpose.
If you follow this tutorial you can adapt it to computing the distance between eyes.
Hello Adrian. Thanks for this tutorial. A have a question.
For optimization purposes, can I only extract 12 (x,y)- coordinates belonging to both right and left eyes? especificaly, in line 35:
shape = predictor(gray, rect)
Thanks in advance
Yes, see this tutorial.
hi Adrian, nice work done!!! I’ve followed your tutorial and and write an ios app with objective-c, dlib, and opengl. It works well when the head’s roll angle is between -30 to 30 degrees, but incorrect alignment of 68 landmarks when otherwise. Is there anything i’m missing? I need your help. Thanks!
Hello, Adrian!
I used your post to start my scientific initiation at university.
Now, I just need to know where is the origin of the coordinate system. I suppose it is at lower left corner, but I am not sure.
I need a lot your help, because I couldn’t find that on dlib documentation.
Thank you!
Hi Adrian,
Thanks for a great post. It’s very helpful. Our team wants to add the facial landmark detection (FLD) to our mobile app. The app is developed in javascript, do you know how I can integrate the FLD feature that written in python to the app? I don’t know javascript and the javascript guy doesn’t know python.
Sorry, I don’t have any experience with JavaScript and computer vision algorithms. I think opencv.js might still be in development though, perhaps check that out.
You can look into face-api.js.
Hey! Thanks a lot for the tutorial! How can I cite your work or use the images from this tutorial in a paper I want to publish?
Thanks for asking. Follow these instructions on how to cite my work.
How can I compare these points with others in other images?
I need to compare distances with several other images and make sure it’s the same person
That sounds like you should be using a dedicated face recognition algorithm instead.
Thanks a ton, Adrian! I am a novice and pretty new to python. Could successfully complete all the steps. All credit to you!
A small niggle for me though – after the image is shown the program execution was waiting for ever and wouldn’t terminate even when i press esc key or ctrl+D. I am using ubuntu latest version. For now i have put a workaround by setting the waitkeyto a fixed value like 5000 which shows the image for 5seconds and exit successfully. From what i searched on the net, waitkey(0) should exit upon any key event..but for some reason my key events were not getting detected.
Looked around further and now i got it! When i press number ‘0’ key it exits properly.
Congrats on resolving the issue!
Hello Sir ,
Thank You for the great tutorial.
I got 68 facial co-ordinates successfully .
I need one help more.
After getting facial coordinate , it is possible to correlate point from different human images. and make a system which give estimate human face size from image in measurable unit like mm,cm.
I read yours https://pyimagesearch.com/2016/03/28/measuring-size-of-objects-in-an-image-with-opencv/ Blog .
I don’t have any reference object in my images.
Can you please give me kick of any reference so i can get human face size in measurable unit.
Thank you is not sufficient , For helping others in learning.
Gentlemen, good morning,
Performing some tests with dlib with two cameras one IP and another HD has intelbras with DVR with speed problems in ip camera, while a HD responds in 2 seconds in ip camera responds in 20 seconds, or delays this nonsense with ip cameras … already test The ip camera on different networks, but the HD camera has good speed like a webcam.
Does anyone have any idea what this might be? Is there any solution for camera IP?
Does this line of descriptor that identifies makes a very slow application with the use of ip camera, HD camera and WEBCAM speed because it occurs?
Face descriptor = Face recognition.compute_face_descriptor (image, Facepoints)
Thiago
Thanks for you treated ❤️ tutorial and I have a couple of questionI am working on a web based project of assigning the appropriate false eyelash according to the eye type or shape (round, monolid, hooded, downturned, upturned and almond)
– is there any method written in Python to determine the eye type?
– I want to create a website in Python and deploy the eye shape model on it . What is the best choice? Therefore the client can upload his image and the appropriate eyelash is specified respectively. You recommendation in how to do that
_
I don’t know of a model used to automatically detect the eye type. Do you have a dataset of such examples?
As for web framework, I personally like Django and Flask.
Hi Adrain,
Great tutorial. What if the image has a partial face with only eyes and nose. For example, can the blink detection work if the video/image has only eyes visible. If so , do we need to handle or modify the landmarks ?
Provided that (1) the face can be detected and (2) the eyes can be localized, yes, it will work.
Sir you are my motivation.
I have searched a lot. but this is something that gave me huge benefit.
Thanks a lot sir.
Thanks Khayam 🙂
Hey, I tried to use dlib with thermal facial images but it is not working, is there any facial landmarks detector for thermal images , eagerly waiting for your answer.
Sorry, I don’t believe there is.
Hi Adrian, After extracting the landmark features, can I go ahead and pass the feature vectors to a neural net for face detection? I want to know if facial landmark features is all I need for facial detection? or do I need to perform other processing for optimal results? Thanks
I think you’re confusing face detection with face recognition. Face detection simply detects where in a face an image is (it does not identify it). Face recognition takes the detected face and recognizes it. I would suggest you read my face recognition tutorials.
Hello sir,
I am a student in Viet Nam. I know you as the author of Pyimagesearch – a famous and useful website!
I and my friend have participated in the National Science and Technology Competition. Our project is about Facial Landmark. Whereby, our product (Automative Safety System for Cars) has two main parts: Warning drowsiness system and Warning asphyxiation system. We need to complete about four questions:
1. Facial Landmark’s origin and its operating principle?
2. What are the Symptoms of drowsiness?
3. What is the “dangerous threshold” of CO2 concentration, CO, temperature, humidity, which affect negatively human health?
4. Frequency of blinking
Can you support us?
Thanks for your reading!
Hey Adrian,
It is a great article, it helped me a lot.
Do you know which machine learning algorithm is used in dlib to train data on facial landmarks.dat?
Refer to this tutorial as it discusses the algorithm used to train the shape predictor.
Hi Adrian,
My name is Raymond Briones and my team and I are currently working on an eye-blink tracking system. We have implemented your code, however, we encountered some issues regarding accuracy. These issues include accuracy for the count of the blinks, head movement (when looking down) counts as a blink, and facial detection outputting false positives (to the count of the number of blinks). When trying out the code, we had a big issue with the program outputting false positives where the system accounts for blinks even though the person did not blink.
Is there any way we can improve the accuracy, for example, using a webcam with zoom capability, or applying zoom in the code itself? Any input would be greatly appreciated.
Hey Adrian !
I was wondering how i can detect only the eyes and the eyebrow using the same code. Would it be enough to just change the range in shape_to_np file?
See this tutorial.
Hey Adrian!
You mentioned that except 64 point landmark detector there’s also a model which happens to place 194 point landmarks on face, so I searched the link for HELEN dataset page you provided, but i couldn’t find the trained model or .dat file like the one for 64 point landmark shape predictor.
Do I need to train a model from scratch to apply 194 point landmarks from HELEN dataset? i dont understand? If yes then how? Btw love the above article!
Yes, you would need to train your own custom dlib shape predictor.
Hey Adrian,
Thanks for a great post!
Can you give me a brief explanation of how the facial landmarks predictor works?
What is the process that is done from the images in the dataset?
Do we extract features in the same way as with HOG?
Inside the post I link to the original publication on facial landmarks. I suggest giving it a read if you want more details on how it works. I may cover it in more detail in the future but I cannot guarantee if/when that may be.