Let me tell you an embarrassing story of how I wasted three weeks of research time during graduate school six years ago.
It was the end of my second semester of coursework.
I had taken all of my exams early and all my projects for the semester had been submitted.
Since my school obligations were essentially nil, I started experimenting with (automatically) identifying prescription pills in images, something I know a thing or two about (but back then I was just getting started with my research).
At the time, my research goal was to find and identify methods to reliably quantify pills in a rotation invariant manner. Regardless of how the pill was rotated, I wanted the output feature vector to be (approximately) the same (the feature vectors will never be to completely identical in a real-world application due to lighting conditions, camera sensors, floating point errors, etc.).
After the first week I was making fantastic progress.
I was able to extract features from my dataset of pills, index them, and then identify my test set of pills regardless of how they were oriented…
…however, there was a problem:
My method was only working with round, circular pills — I was getting completely nonsensical results for oblong pills.
How could that be?
I racked my brain for the explanation.
Was there a flaw in the logic of my feature extraction algorithm?
Was I not matching the features correctly?
Or was it something else entirely…like a problem with my image preprocessing.
While I might have been ashamed to admit this as a graduate student, the problem was the latter:
I goofed up.
It turns out that during the image preprocessing phase, I was rotating my images incorrectly.
Since round pills have are approximately square in their aspect ratio, the rotation bug wasn’t a problem for them. Here you can see a round pill being rotated a full 360 degrees without an issue:
But for oblong pills, they would be “cut off” in the rotation process, like this:
In essence, I was only quantifying part of the rotated, oblong pills; hence my strange results.
I spent three weeks and part of my Christmas vacation banging my head against the wall trying to diagnose the bug — only to feel quite embarrassed when I realized it was due to me being negligent with the cv2.rotate function.
You see, the size of the output image needs to be adjusted, otherwise, the corners of my image would be cut off.
How did I accomplish this and squash the bug for good?
To learn how to rotate images with OpenCV such that the entire image is included and none of the image is cut off, just keep reading.
Rotate images (correctly) with OpenCV and Python
In the remainder of this blog post I’ll discuss common issues that you may run into when rotating images with OpenCV and Python.
Specifically, we’ll be examining the problem of what happens when the corners of an image are “cut off” during the rotation process.
To make sure we all understand this rotation issue with OpenCV and Python I will:
- Start with a simple example demonstrating the rotation problem.
- Provide a rotation function that ensures images are not cut off in the rotation process.
- Discuss how I resolved my pill identification issue using this method.
A simple rotation problem with OpenCV
Let’s get this blog post started with an example script.
Open up a new file, name it rotate_simple.py , and insert the following code:
# import the necessary packages
import numpy as np
import argparse
import imutils
import cv2
# construct the argument parse and parse the arguments
ap = argparse.ArgumentParser()
ap.add_argument("-i", "--image", required=True,
help="path to the image file")
args = vars(ap.parse_args())
Lines 2-5 start by importing our required Python packages.
If you don’t already have imutils, my series of OpenCV convenience functions installed, you’ll want to do that now:
$ pip install imutils
If you already have imutils installed, make sure you have upgraded to the latest version:
$ pip install --upgrade imutils
From there, Lines 8-10 parse our command line arguments. We only need a single switch here, --image , which is the path to where our image resides on disk.
Let’s move on to actually rotating our image:
# load the image from disk
image = cv2.imread(args["image"])
# loop over the rotation angles
for angle in np.arange(0, 360, 15):
rotated = imutils.rotate(image, angle)
cv2.imshow("Rotated (Problematic)", rotated)
cv2.waitKey(0)
# loop over the rotation angles again, this time ensuring
# no part of the image is cut off
for angle in np.arange(0, 360, 15):
rotated = imutils.rotate_bound(image, angle)
cv2.imshow("Rotated (Correct)", rotated)
cv2.waitKey(0)
Line 14 loads the image we want to rotate from disk.
We then loop over various angles in the range [0, 360] in 15 degree increments (Line 17).
For each of these angles we call imutils.rotate , which rotates our image the specified number of angle degrees about the center of the image. We then display the rotated image to our screen.
Lines 24-27 perform an identical process, but this time we call imutils.rotate_bound (I’ll provide the implementation of this function in the next section).
As the name of this method suggests, we are going to ensure the entire image is bound inside the window and none is cut off.
To see this script in action, be sure to download the source code using the “Downloads” section of this blog post, followed by executing the command below:
$ python rotate_simple.py --image images/saratoga.jpg
The output of using the imutils.rotate function on a non-square image can be seen below:
As you can see, the image is “cut off” when it’s rotated — the entire image is not kept in the field of view.
But if we use imutils.rotate_bound we can resolve this issue:
Awesome, we fixed the problem!
So does this mean that we should always use .rotate_bound over the .rotate method?
What makes it so special?
And what’s going on under the hood?
I’ll answer these questions in the next section.
Implementing a rotation function that doesn’t cut off your images
Let me start off by saying there is nothing wrong with the cv2.getRotationMatrix2D and cv2.warpAffine functions that are used to rotate images inside OpenCV.
In reality, these functions give us more freedom than perhaps we are comfortable with (sort of like comparing manual memory management with C versus automatic garbage collection with Java).
The cv2.getRotationMatrix2D function doesn’t care if we would like the entire rotated image to kept.
It doesn’t care if the image is cut off.
And it won’t help you if you shoot yourself in the foot when using this function (I found this out the hard way and it took 3 weeks to stop the bleeding).
Instead, what you need to do is understand what the rotation matrix is and how it’s constructed.
You see, when you rotate an image with OpenCV you call cv2.getRotationMatrix2D which returns a matrix M that looks something like this:
This matrix looks scary, but I promise you: it’s not.
To understand it, let’s assume we want to rotate our image 
degrees about some center 
coordinates at some scale (i.e., smaller or larger).
We can then plug in values for 
and 
:
and 
That’s all fine and good for simple rotation — but it doesn’t take into account what happens if an image is cut off along the borders. How do we remedy this?
The answer is inside the rotate_bound function in convenience.py of imutils:
def rotate_bound(image, angle):
# grab the dimensions of the image and then determine the
# center
(h, w) = image.shape[:2]
(cX, cY) = (w // 2, h // 2)
# grab the rotation matrix (applying the negative of the
# angle to rotate clockwise), then grab the sine and cosine
# (i.e., the rotation components of the matrix)
M = cv2.getRotationMatrix2D((cX, cY), -angle, 1.0)
cos = np.abs(M[0, 0])
sin = np.abs(M[0, 1])
# compute the new bounding dimensions of the image
nW = int((h * sin) + (w * cos))
nH = int((h * cos) + (w * sin))
# adjust the rotation matrix to take into account translation
M[0, 2] += (nW / 2) - cX
M[1, 2] += (nH / 2) - cY
# perform the actual rotation and return the image
return cv2.warpAffine(image, M, (nW, nH))
On Line 41 we define our rotate_bound function.
This method accepts an input image and an angle to rotate it by.
We assume we’ll be rotating our image about its center (x, y)-coordinates, so we determine these values on lines 44 and 45.
Given these coordinates, we can call cv2.getRotationMatrix2D to obtain our rotation matrix M (Line 50).
However, to adjust for any image border cut off issues, we need to apply some manual calculations of our own.
We start by grabbing the cosine and sine values from our rotation matrix M (Lines 51 and 52).
This enables us to compute the new width and height of the rotated image, ensuring no part of the image is cut off.
Once we know the new width and height, we can adjust for translation on Lines 59 and 60 by modifying our rotation matrix once again.
Finally, cv2.warpAffine is called on Line 63 to rotate the actual image using OpenCV while ensuring none of the image is cut off.
For some other interesting solutions (some better than others) to the rotation cut off problem when using OpenCV, be sure to refer to this StackOverflow thread and this one too.
Fixing the rotated image “cut off” problem with OpenCV and Python
Let’s get back to my original problem of rotating oblong pills and how I used .rotate_bound to solve the issue (although back then I had not created the imutils Python package — it was simply a utility function in a helper file).
We’ll be using the following pill as our example image:
To start, open up a new file and name it rotate_pills.py . Then, insert the following code:
# import the necessary packages
import numpy as np
import argparse
import imutils
import cv2
# construct the argument parse and parse the arguments
ap = argparse.ArgumentParser()
ap.add_argument("-i", "--image", required=True,
help="path to the image file")
args = vars(ap.parse_args())
Lines 2-5 import our required Python packages. Again, make sure you have installed and/or upgraded the imutils Python package before continuing.
We then parse our command line arguments on Lines 8-11. Just like in the example at the beginning of the blog post, we only need one switch: --image , the path to our input image.
Next, we load our pill image from disk and preprocess it by converting it to grayscale, blurring it, and detecting edges:
# load the image from disk, convert it to grayscale, blur it, # and apply edge detection to reveal the outline of the pill image = cv2.imread(args["image"]) gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY) gray = cv2.GaussianBlur(gray, (3, 3), 0) edged = cv2.Canny(gray, 20, 100)
After executing these preprocessing functions our pill image now looks like this:
The outline of the pill is clearly visible, so let’s apply contour detection to find the outline of the pill:
# find contours in the edge map cnts = cv2.findContours(edged.copy(), cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE) cnts = imutils.grab_contours(cnts)
We are now ready to extract the pill ROI from the image:
# ensure at least one contour was found if len(cnts) > 0: # grab the largest contour, then draw a mask for the pill c = max(cnts, key=cv2.contourArea) mask = np.zeros(gray.shape, dtype="uint8") cv2.drawContours(mask, [c], -1, 255, -1) # compute its bounding box of pill, then extract the ROI, # and apply the mask (x, y, w, h) = cv2.boundingRect(c) imageROI = image[y:y + h, x:x + w] maskROI = mask[y:y + h, x:x + w] imageROI = cv2.bitwise_and(imageROI, imageROI, mask=maskROI)
First, we ensure that at least one contour was found in the edge map (Line 26).
Provided we have at least one contour, we construct a mask for the largest contour region on Lines 29 and 30.
Our mask looks like this:
Given the contour region, we can compute the (x, y)-coordinates of the bounding box of the region (Line 34).
Using both the bounding box and mask , we can extract the actual pill region ROI (Lines 35-38).
Now, let’s go ahead and apply both the imutils.rotate and imutils.rotate_bound functions to the imageROI , just like we did in the simple examples above:
# loop over the rotation angles
for angle in np.arange(0, 360, 15):
rotated = imutils.rotate(imageROI, angle)
cv2.imshow("Rotated (Problematic)", rotated)
cv2.waitKey(0)
# loop over the rotation angles again, this time ensure the
# entire pill is still within the ROI after rotation
for angle in np.arange(0, 360, 15):
rotated = imutils.rotate_bound(imageROI, angle)
cv2.imshow("Rotated (Correct)", rotated)
cv2.waitKey(0)
After downloading the source code to this tutorial using the “Downloads” section below, you can execute the following command to see the output:
$ python rotate_pills.py --image images/pill_01.png
The output of imutils.rotate will look like:
Notice how the pill is cut off during the rotation process — we need to explicitly compute the new dimensions of the rotated image to ensure the borders are not cut off.
By using imutils.rotate_bound , we can ensure that no part of the image is cut off when using OpenCV:
Using this function I was finally able to finish my research for the winter break — but not before I felt quite embarrassed about my rookie mistake.
What's next? We recommend PyImageSearch University.
86 total classes ā¢ 115+ hours of on-demand code walkthrough videos ā¢ Last updated: October 2024
ā ā ā ā ā 4.84 (128 Ratings) ā¢ 16,000+ Students Enrolled
I strongly believe that if you had the right teacher you could master computer vision and deep learning.
Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?
Thatās not the case.
All you need to master computer vision and deep learning is for someone to explain things to you in simple, intuitive terms. And thatās exactly what I do. My mission is to change education and how complex Artificial Intelligence topics are taught.
If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here youāll learn how to successfully and confidently apply computer vision to your work, research, and projects. Join me in computer vision mastery.
Inside PyImageSearch University you'll find:
- ✓ 86 courses on essential computer vision, deep learning, and OpenCV topics
- ✓ 86 Certificates of Completion
- ✓ 115+ hours of on-demand video
- ✓ Brand new courses released regularly, ensuring you can keep up with state-of-the-art techniques
- ✓ Pre-configured Jupyter Notebooks in Google Colab
- ✓ Run all code examples in your web browser ā works on Windows, macOS, and Linux (no dev environment configuration required!)
- ✓ Access to centralized code repos for all 540+ tutorials on PyImageSearch
- ✓ Easy one-click downloads for code, datasets, pre-trained models, etc.
- ✓ Access on mobile, laptop, desktop, etc.
Summary
In today’s blog post I discussed how image borders can be cut off when rotating images with OpenCV and cv2.warpAffine .
The fact that image borders can be cut off is not a bug in OpenCV — in fact, it’s how cv2.getRotationMatrix2D and cv2.warpAffine are designed.
While it may seem frustrating and cumbersome to compute new image dimensions to ensure you don’t lose your borders, it’s actually a blessing in disguise.
OpenCV gives us so much control that we can modify our rotation matrix to make it do exactly what we want.
Of course, this requires us to know how our rotation matrix M is formed and what each of its components represents (discussed earlier in this tutorial). Provided we understand this, the math falls out naturally.
To learn more about image processing and computer vision, be sure to take a look at the PyImageSearch Gurus course where I discuss these topics in more detail.
Otherwise, I encourage you to enter your email address in the form below to be notified when future blog posts are published.
Download the Source Code and FREE 17-page Resource Guide
Enter your email address below to get a .zip of the code and a FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning. Inside you'll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL!
amazing article as always.
do you know any ml/ deep learning NN architectures that are rotation invariant inherently, without image preprocessing and thus creating multiple train exemplars leading to the same classification (as I suppose you do) ?
I have googled the topic without success.
thanks for everything
This is still an active area of research. Focus your search on “steerable filters + convolutional neural networks” and you’ll come across some of the more recent publications. The problem here is that rotation invariance can vary based on what type of dataset you’re working with. For example, rotation invariance for natural scene images (broad classification and therefore easier) is much easier than obtain than say rotation invariance for fine-grained classification (such as pill identification).
This is very useful. Thanks, Adrian, and Happy New Year!
Happy New Year to you as well David!
Excellent! Any easy to way to return a mask for the portions of the rotated image that are not part of the image itself?
Hi Adrian,
Thanks for posting this. I have a question for you. What if you were interested in the opposite scenario? That is, if you were doing object tracking and you wanted to calculate the rotation angle as the object is rotating. For example, you may take a reference image of an object, and then track the object realtime using the webcam while the object is rotating back and forth. I am interested in the angle of the object in the current frame relative to the angle of the object in the reference frame. For simplicity, let’s for now assume that the object only rotates along the axis of the camera, and does not change size. Could you point me to the right direction for this?
Thanks in advance.
There are multiple ways to accomplish this, each of them based on the actual shape of the object you are working with. The best solution would be to determine the center of the object and a second identifiable point of some sort on the object that can be detected across many frames. Exactly which method you use is really dependent on the project. Once you have these points you can measure how much the object has rotated between frames.
For irregular objects you could simply compute the mask + bounding box and then compute the minimum-enclosing rectangle which will also give you the angle of rotation.
Hi Adrian,
Thanks for the reply. Let’s say that we are trying to create a more general algorithm under the following scenario: we would like to detect the rotation of different objects, but in all cases the object is circular and has a detectable pattern to it that’s not symmetric (therefore it would be possible to tell the angle). However, let’s say the pattern itself is not always the same. Consider for instance company logos that are circular. Usually the pattern here is an annulus and the detectable features are not necessarily the same from one logo to another, except for that the features are located in an annulus around the center. I was thinking about taking a reference on the annulus and then tracking the rotational angle. However, I’m not sure if there is a better approach, and how to make this approach computationally efficient. If SIFT or SURF algorithms are used, I fear they would not be efficient so I was hoping there would be a better method.
The standard approach here would be to use SIFT/SURF, keypoint matching, and RANSAC. You mentioned wanting to create a “general algorithm”, but in reality I don’t think this is advisable. Most successful computer vision applications focus on a specific problem and attempt to solve it. Take logo recognition for example — we’ve become better at logo recognition but it’s not solved. I would suggest you start with SIFT/SURF and see how far it gets you in your particular problem, but try to stay away from solving “general” problems.
Hi Adrian,
Thanks for the code, very clear and helpful. I have a question. Since Python is starting indexing from zero shouldn’t the center be calculated as:
(cX, cY) = ((w-1) // 2.0, (h-1) // 2.0)
I am trying to do some mapping between coordinates after rotation using your code. But I don’t get the correct mapping unfortunately.
Thanks for sharing this with us.
What if rotation center is not the center of image?
If the rotation center is not the center of the image then you would need to update the translation components of the matrix to reflect this.
If you wanted a reversible version of this, would it be best to pad the original image and rotate the “bad” way, or to use the “good” way and crop once it was reversed?
Hi Chris — I’m not sure what you mean by a “reversible” verison?
Hey Adrian. First of all thank you for all your great posts, really helpful and fun to read.
I think that Chris wanted to say how would you approach to “undo” such rotation. Say, if you rotated it 10 degrees counter-clockwise with rotate_bounds you will get the new sized imaged and the content correctly rotated (not clipped).
What if you want to rotate 10 degrees clockwise? To “undo” such rotation… I tried your approach, and doing this results in the image correctly rotated (like its original un-rotated version) but with a size much larger than the original one had, and a buffer of zeros around it.
What could be an approach to avoid that? Perhaps doing rotation as you explained, but then clip or resize the image to the shape the original one had?
Cheers
Hi Adrian, thank you for this post- I have an 3 *3 Rotation matrix, How can I apply this rotation matrix for an original image and find the transformed image. The 3 *3 matrix is obtained using angles measured by x,y,z direction of the camera.
Hi Adrian,
Thank you for the great post. I wonder is there a quick way to map a pixel coordinate at (x,y) in an image rotated by rotate_bound function back to the coordinate in the original image?
Thanks!
Ur understanding is awesome man.. very perfect
Thank you Haroon š
Java implementation without minus on the angle.
https://github.com/manumanmax/maogene/blob/master/camera/src/utils/CVUtils.java
Thanks for sharing!
I was trying to implement your code but i faced one problem , once i run above code , finely the image loaded but how to rotate it in any direction ? I’m beginner in this field.
When i run the code , image open finely. After what ? Is it rotate by its own in any direction such as clock wise or anti-clock wise ? Or , i should use mouse click but it didn’t work.
Any suggestion ?
Hey Sam — thanks for the comment, although I’m a bit confused by the question. You do not use your mouse to rotate the image. The amount in which the image is rotation is indicated by the rotation angle in the code.
Hi Adrian,
I’m having the same problem as Sam. The image opens but doesn’t rotate. Ive tried both python rotate_simple.py –image images/saratoga.jpg and python rotate_pills.py –image images/pill_01.png.
I’m using windows. Could that be a problem?
Danny
Got it, you need to increase waitkey time.
Or you can click the active window and press any key on your keyboard. The
cv2.waitKey(0)call will wait until you click the window opened by OpenCV and then hit a key.Hi Adrian,
Upon applying the rotate_bound function to my images, it appears that the color of the image was also altered. Any idea why this is?
That’s quite odd. Can you be more specific on what “altered” means in this context?
Maybe it is because that cv2.imread uses BGR to load images.
Try the following code:
img = cv2.imread(imagepath)
RGB_img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
That wouldn’t matter, but now that you bring it up, I bet George is trying to use a function other than cv2.imshow to display the image.
Hi Adrian.
When using the rotate_bound, is there a way to get the mapping of each pixel to its new location?
The homography matrix generated by OpenCV can be use to map the location of one pixel in the original image to the location of the pixel in the new image.
Hey, Following techniques given here- I want to detect a phone in image and automatically rotate image to make phone in portraight or landscape mode. Using your document scanner tutorial I can find countour, but how to auto rotate image?
Once you have detected the phone itself you can extract the ROI and call
imutils.rotate_boundto rotate the actual phone region.Hi Adrian,
I had a question regarding the rotation and display function. I’m developing an application using OpenCV and PyQT. The problem is that OpenCV has a rotation function and even PyQT.
So is it better to use the rotation function of PyQT or OpenCV?
Also, for eg loading an image, the OpenCV format is in BGR whereas for PyQt it’s RGB.
Which function is faster and better while loading an image, is it better to use OpenCV function imshow, or is it better to convert the image channels from BGR to RGB and then load it in PyQT?
Sorry, I’m not very familiar with PyQT so I unfortunately don’t have much advice there. I personal suggestion would be for OpenCV to handle all image processing functions and only then hand off to PyQT to display the image, but again, I don’t have much knowledge over PyQT.
Hi Adrian,
Awesome explanation on the topic. I have a doubt though.
How does warpAffine() work? I have been trying to write the C++ code that does the same as warpAffine but haven’t been able to. Any ideas on the same?
Thanks in advance
Under the hood the “cv2.warpAffine” function is performing a affine transformation.
Hi Adrian,
I’m using your rotate_bound function on a 3d image volume, and would like to pass in (cAx1, cAx2) [axes 1 and axes 2] as an argument, so that I can externally call the function and rotate the image 3 consecutive times:
1) first on the x and z rotation plane
2) then on the x and y rotation plane
3) lastly on the y and z rotation plane.
Your code is written for a rotation on the x and y plane. How would I go about calculating (cX, cZ), and (cY, cZ) [lines 44 & 45]? What should be done differently? Please excuse me if the answer to my question is evident, as I have little background in this area.
Thanks,
isa
Hm, I suppose what I am searching for is the axis argument in line 50:
M = cv2.getRotationMatrix2D((cX, cY), -angle, axis=1.0) ?
I don’t think OpenCV has a built-in 3D rotation function but you may want to refer to this GitHub repo which seems to address your question.
Hi Adrian,
How to correctly perform an arbitrary affine transformation (without cropping) ?
Hello Adrian,
First thanks for your amazing tutorials, they are the best!
My question is: In figure 8, why is the pill mask filled in white? Should it be just white edges?
No, we used
cv2.drawContours(mask, [c], -1, 255, -1)to draw the contours. The final “-1” means that we want the contour filled in.Hi, thanks for the great post.
I have one question. Is there a way to change the black background color to white?
Thank you.
Have you segmented your image and created a binary mask? If so, “cv2.bitwise_not” would flip the background to white.
Hi Adrian,
Thanks for the useful post. I have a question – I want to crop a part of an image, say for instance draw a rectangle at a desired angle over a small portion in a large image and then crop that part.
Can you please suggest how i can crop it. I have the center point of the rectangle , height , width and angle at which it is tilted
Am not able to paste the image to show how it looks like. Hoping that my question is clear
You can do that using NumPy array slicing. I cover how to extract portions of an image inside Practical Python and OpenCV. Definitely start there!
Using Numpy slicing I used the below code. But It doesn’t crop the rectangle at an angle
cropped = img[y:y+h, x:x+w]
It just crops a straight rectangle. Please advice
You cannot crop a rotated rectangle with OpenCV. You would first crop the “normal” rectangle and then use masking to mask out the regions you do not want.
This is a really good explanation.
I’m facing the same issue but with Affine transformation.
How can I get a correct image ? please advice
Thanks Adrinannnn you save my project God Bless Youu and your Family
Thanks Bruno!
Mr. Adrian, i have a question. I was change the bitwise_and() with the bitwise_not(), but the background not changed to white.
Hi, I would like to put a green background instead of a black background. Like, when you rotate the image, a black background increases in the back of the image. How can I put it in green color?
Hi Adrian,
thanks a lot for your inspiring post! I struggling actually also with an rotation problem a bit different you have done:
I have a binary image with a white rectangle. The image dimensions are sufficient so rotating the image up to 90Ā° doesnt cut off anything. So far so good!
At the starting point (0Ā° rotation) I count the white pixels.
Then I rotate the image using scipy.ndimage.rotate with reshape=False from 0Ā°:1Ā°:90Ā° counting the white pixels of the rotated image and estimating the difference regarding no rotation.
What do I get: for some angles, the rotated rectangle has more and sometimes less white pixels than the rectangle without any rotation.
Do you know am algorithm or any hint to get a pixelinvariant rotation?? IĀ“m struggling since days without any idea.
Best regards!