One of my favorite parts of running the PyImageSearch blog is a being able to link together previous blog posts and create a solution to a particular problem — in this case, real-time panorama and image stitching with Python and OpenCV.
Over the past month and a half, we’ve learned how to increase the FPS processing rate of builtin/USB webcams and the Raspberry Pi camera module. We also learned how to unify access to both USB webcams and the Raspberry Pi camera into a single class, making all video processing and examples on the PyImageSearch blog capable of running on both USB and Pi camera setups without having to modify a single line of code.
And just to weeks ago, we discussed how keypoint detection, local invariant descriptors, keypoint matching, and homography matrix estimation can be used to construct panoramas and stitch images together.
Today we are going to link together the past 1.5 months worth of posts and use them to perform real-time panorama and image stitching using Python and OpenCV. Our solution will be able to run on both laptop/desktops systems, along with the Raspberry Pi.
Furthermore, we’ll also apply our basic motion detection implementation from last week’s post to perform motion detection on the panorama image.
This solution is especially useful in situations where you want to survey a wide area for motion, but don’t want “blind spots” in your camera view.
Keep reading to learn more…
Real-time panorama and image stitching with OpenCV
As I mentioned in the introduction to this post, we’ll be linking together concepts we have learned in the previous 1.5 months of PyImageSearch posts and:
- Use our improved FPS processing rate Python classes to access our builtin/USB webcams and/or the Raspberry Pi camera module.
- Access multiple camera streams at once.
- Apply image stitching and panorama construction to the frames from these video streams.
- Perform motion detection in the panorama image.
Again, the benefit of performing motion detection in the panorama image versus two separate frames is that we won’t have any “blind spots” in our field of view.
Hardware setup
For this project, I’ll be using my Raspberry Pi 2, although you could certainly use your laptop or desktop system instead. I simply went with the Pi 2 for it’s small form factor and ease of maneuvering in space constrained places.
I’ll also be using my Logitech C920 webcam (that is plug-and-play compatible with the Raspberry Pi) along with the Raspberry Pi camera module. Again, if you decide to use your laptop/desktop system, you can simply hook-up multiple webcams to your machine — the same concepts discussed in this post still apply.
Below you can see my setup:
Here is another angle looking up at the setup:
The setup is pointing towards my front door, kitchen, and hallway, giving me a full view of what’s going on inside my apartment:
The goal is to take frames captured from both my video streams, stitch them together, and then perform motion detection in the panorama image.
Constructing a panorama, rather than using multiple cameras and performing motion detection independently in each stream ensures that I don’t have any “blind spots” in my field of view.
Project structure
Before we get started, let’s look at our project structure:
|--- pyimagesearch | |---- __init__.py | |--- basicmotiondetector.py | |--- panorama.py |--- realtime_stitching.py
As you can see, we have defined a pyimagesearch module for organizational purposes. We then have the basicmotiondetector.py implementation from last week’s post on accessing multiple cameras with Python and OpenCV. This class hasn’t changed at all, so we won’t be reviewing the implementation in this post. For a thorough review of the basic motion detector, be sure to read last week’s post.
We then have our panorama.py file which defines the Stitcher class used to stitch images together. We initially used this class in the OpenCV panorama stitching tutorial.
However, as we’ll see later in this post, I have made a slight modifications to the constructor and stitch methods to facilitate real-time panorama construction — we’ll learn more about these slight modifications later in this post.
Finally, the realtime_stitching.py file is our main Python driver script that will access the multiple video streams (in an efficient, threaded manner of course), stitch the frames together, and then perform motion detection on the panorama image.
Updating the image stitcher
In order to (1) create a real-time image stitcher and (2) perform motion detection on the panorama image, we’ll assume that both cameras are fixed and non-moving, like in Figure 1 above.
Why is the fixed and non-moving assumption so important?
Well, remember back to our lesson on panorama and image stitching.
Performing keypoint detection, local invariant description, keypoint matching, and homography estimation is a computationally expensive task. If we were to use our previous implementation, we would have to perform stitching on each set of frames, making it near impossible to run in real-time (especially for resource constrained hardware such as the Raspberry Pi).
However, if we assume that the cameras are fixed, we only have to perform the homography matrix estimation once!
After the initial homography estimation, we can use the same matrix to transform and warp the images to construct the final panorama — doing this enables us to skip the computationally expensive steps of keypoint detection, local invariant feature extraction, and keypoint matching in each set of frames.
Below I have provided the relevant updates to the Sticher class to facilitate a cached homography matrix:
# import the necessary packages import numpy as np import imutils import cv2 class Stitcher: def __init__(self): # determine if we are using OpenCV v3.X and initialize the # cached homography matrix self.isv3 = imutils.is_cv3() self.cachedH = None
The only addition here is on Line 11 were I define cachedH , the cached homography matrix.
We also need to update the stitch method to cache the homography matrix after it is computed:
def stitch(self, images, ratio=0.75, reprojThresh=4.0): # unpack the images (imageB, imageA) = images # if the cached homography matrix is None, then we need to # apply keypoint matching to construct it if self.cachedH is None: # detect keypoints and extract (kpsA, featuresA) = self.detectAndDescribe(imageA) (kpsB, featuresB) = self.detectAndDescribe(imageB) # match features between the two images M = self.matchKeypoints(kpsA, kpsB, featuresA, featuresB, ratio, reprojThresh) # if the match is None, then there aren't enough matched # keypoints to create a panorama if M is None: return None # cache the homography matrix self.cachedH = M[1] # apply a perspective transform to stitch the images together # using the cached homography matrix result = cv2.warpPerspective(imageA, self.cachedH, (imageA.shape[1] + imageB.shape[1], imageA.shape[0])) result[0:imageB.shape[0], 0:imageB.shape[1]] = imageB # return the stitched image return result
On Line 19 we make a check to see if the homography matrix has been computed before. If not, we detect keypoints and extract local invariant descriptors from the two images, followed by applying keypoint matching. We then cache the homography matrix on Line 34.
Subsequent calls to stitch will use this cached matrix, allowing us to sidestep detecting keypoints, extracting features, and performing keypoint matching on every set of frames.
For the rest of the source code to panorama.py , please see the image stitching tutorial or use the form at the bottom of this post to download the source code.
Performing real-time panorama stitching
Now that our Stitcher class has been updated, let’s move on to to the realtime_stitching.py driver script:
# import the necessary packages
from __future__ import print_function
from pyimagesearch.basicmotiondetector import BasicMotionDetector
from pyimagesearch.panorama import Stitcher
from imutils.video import VideoStream
import numpy as np
import datetime
import imutils
import time
import cv2
# initialize the video streams and allow them to warmup
print("[INFO] starting cameras...")
leftStream = VideoStream(src=0).start()
rightStream = VideoStream(usePiCamera=True).start()
time.sleep(2.0)
We start off by importing our required Python packages. The BasicMotionDetector and Stitcher classes are imported from the pyimagesearch module. We’ll also need the VideoStream class from the imutils package.
If you don’t already have imutils installed on your system, you can install it using:
$ pip install imutils
If you do already have it installed, make sure you have upgraded to the latest version (which has added Python 3 support to the video sub-module):
$ pip install --upgrade imutils
Lines 14 and 15 then initialize our two VideoStream classes. Here I assume that leftStream is a USB camera and rightStream is a Raspberry Pi camera (indicated by usePiCamera=True ).
If you wanted to use two USB cameras, you would simply have to update the stream initializations to:
leftStream = VideoStream(src=0).start() rightStream = VideoStream(src=1).start()
The src parameter controls the index of the camera on your system.
Again, it’s imperative that you initialize leftStream and rightStream correctly. When standing behind the cameras, the leftStream should be the camera to your lefthand side and the rightStream should be the camera to your righthand side.
Failure to set these stream variables correctly will result in a “panorama” that contains only one of the two frames.
From here, let’s initialize the image stitcher and motion detector:
# initialize the image stitcher, motion detector, and total # number of frames read stitcher = Stitcher() motion = BasicMotionDetector(minArea=500) total = 0
Now we come to the main loop of our driver script where we loop over frames infinitely until instructed to exit the program:
# loop over frames from the video streams
while True:
# grab the frames from their respective video streams
left = leftStream.read()
right = rightStream.read()
# resize the frames
left = imutils.resize(left, width=400)
right = imutils.resize(right, width=400)
# stitch the frames together to form the panorama
# IMPORTANT: you might have to change this line of code
# depending on how your cameras are oriented; frames
# should be supplied in left-to-right order
result = stitcher.stitch([left, right])
# no homograpy could be computed
if result is None:
print("[INFO] homography could not be computed")
break
# convert the panorama to grayscale, blur it slightly, update
# the motion detector
gray = cv2.cvtColor(result, cv2.COLOR_BGR2GRAY)
gray = cv2.GaussianBlur(gray, (21, 21), 0)
locs = motion.update(gray)
Lines 27 and 28 read the left and right frames from their respective video streams. We then resize the frames to have a width of 400 pixels, followed by stitching them together to form the panorama. Remember, frames supplied to the stitch method need to be supplied in left-to-right order!
In the case that the images cannot be stitched (i.e., a homography matrix could not be computed), we break from the loop (Lines 41-43).
Provided that the panorama could be constructed, we then process it by converting it to grayscale and blurring it slightly (Lines 47 and 48). The processed panorama is then passed into the motion detector (Line 49).
However, before we can detect any motion, we first need to allow the motion detector to “run” for a bit to obtain an accurate running average of the background model:
# only process the panorama for motion if a nice average has # been built up if total > 32 and len(locs) > 0: # initialize the minimum and maximum (x, y)-coordinates, # respectively (minX, minY) = (np.inf, np.inf) (maxX, maxY) = (-np.inf, -np.inf) # loop over the locations of motion and accumulate the # minimum and maximum locations of the bounding boxes for l in locs: (x, y, w, h) = cv2.boundingRect(l) (minX, maxX) = (min(minX, x), max(maxX, x + w)) (minY, maxY) = (min(minY, y), max(maxY, y + h)) # draw the bounding box cv2.rectangle(result, (minX, minY), (maxX, maxY), (0, 0, 255), 3)
We use the first 32 frames of the initial video streams as an estimation of the background — during these 32 frames no motion should be taking place.
Otherwise, provided that we have processed the 32 initial frames for the background model initialization, we can check the len of locs to see if it is greater than zero. If it is, then we can assume “motion” is taking place in the panorama image.
We then initialize the minimum and maximum (x, y)-coordinates associated with the locations containing motion. Given this list (i.e., locs ), we loop over the contour regions individually, compute the bounding box, and determine the smallest region encompassing all contours. This bounding box is then drawn on the panorama image.
As mentioned in last week’s post, the motion detector we use assumes there is only one object/person moving at a time. For multiple objects, a more advanced algorithm is required (which we will cover in a future PyImageSearch post).
Finally, the last step is to draw the timestamp on panorama and show the output images:
# increment the total number of frames read and draw the
# timestamp on the image
total += 1
timestamp = datetime.datetime.now()
ts = timestamp.strftime("%A %d %B %Y %I:%M:%S%p")
cv2.putText(result, ts, (10, result.shape[0] - 10),
cv2.FONT_HERSHEY_SIMPLEX, 0.35, (0, 0, 255), 1)
# show the output images
cv2.imshow("Result", result)
cv2.imshow("Left Frame", left)
cv2.imshow("Right Frame", right)
key = cv2.waitKey(1) & 0xFF
# if the `q` key was pressed, break from the loop
if key == ord("q"):
break
# do a bit of cleanup
print("[INFO] cleaning up...")
cv2.destroyAllWindows()
leftStream.stop()
rightStream.stop()
Lines 82-86 make a check to see if the q key is pressed. If it is, we break from the video stream loop and do a bit of cleanup.
Running our panorama builder + motion detector
To execute our script, just issue the following command:
$ python realtime_stitching.py
Below you can find an example GIF of my results:
On the top-left we have the left video stream. And on the top-right we have the right video stream. On the bottom, we can see that both frames have been stitched together into a single panorama. Motion detection is then performed on the panorama image and a bounding box drawn around the motion region.
The full video demo can be seen below:
What's next? We recommend PyImageSearch University.
86 total classes • 115+ hours of on-demand code walkthrough videos • Last updated: October 2024
★★★★★ 4.84 (128 Ratings) • 16,000+ Students Enrolled
I strongly believe that if you had the right teacher you could master computer vision and deep learning.
Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?
That’s not the case.
All you need to master computer vision and deep learning is for someone to explain things to you in simple, intuitive terms. And that’s exactly what I do. My mission is to change education and how complex Artificial Intelligence topics are taught.
If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to successfully and confidently apply computer vision to your work, research, and projects. Join me in computer vision mastery.
Inside PyImageSearch University you'll find:
- ✓ 86 courses on essential computer vision, deep learning, and OpenCV topics
- ✓ 86 Certificates of Completion
- ✓ 115+ hours of on-demand video
- ✓ Brand new courses released regularly, ensuring you can keep up with state-of-the-art techniques
- ✓ Pre-configured Jupyter Notebooks in Google Colab
- ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
- ✓ Access to centralized code repos for all 540+ tutorials on PyImageSearch
- ✓ Easy one-click downloads for code, datasets, pre-trained models, etc.
- ✓ Access on mobile, laptop, desktop, etc.
Summary
In this blog post, we combined our knowledge over the past 1.5 months of tutorials and:
- Increased FPS processing rate using threadng.
- Accessed multiple video streams at once.
- Performed image stitching and panorama construction from these video streams.
- And applied motion detection on the panorama image.
Overall, we were able to easily accomplish all of this on the Raspberry Pi. We can expect even faster performance on a modern laptop or desktop system.
See you next week!
If you enjoyed this post, please be sure to signup for the PyImageSearch Newsletter using the form below!
Download the Source Code and FREE 17-page Resource Guide
Enter your email address below to get a .zip of the code and a FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning. Inside you'll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL!
