A few weeks ago I demonstrated how to perform real-time object detection using deep learning and OpenCV on a standard laptop/desktop.
After the post was published I received a number of emails from PyImageSearch readers who were curious if the Raspberry Pi could also be used for real-time object detection.
The short answer is “kind of”…
…but only if you set your expectations accordingly.
Even when applying our optimized OpenCV + Raspberry Pi install the Pi is only capable of getting up to ~0.9 frames per second when applying deep learning for object detection with Python and OpenCV.
Is that fast enough?
Well, that depends on your application.
If you’re attempting to detect objects that are quickly moving through your field of view, likely
not.
But if you’re monitoring a low traffic environment with slower moving objects, the Raspberry Pi could indeed be fast enough.
In the remainder of today’s blog post we’ll be reviewing two methods to perform deep learning-based object detection on the Raspberry Pi.
Raspberry Pi: Deep learning object detection with OpenCV
Today’s blog post is broken down into two parts.
In the first part, we’ll benchmark the Raspberry Pi for real-time object detection using OpenCV and Python. This benchmark will come from the exact code we used for our laptop/desktop deep learning object detector from a few weeks ago.
I’ll then demonstrate how to use multiprocessing to create an alternate method to object detection using the Raspberry Pi. This method may or may not be useful for your particular application, but at the very least it will give you an idea on different methods to approach the problem.
Object detection and OpenCV benchmark on the Raspberry Pi
The code we’ll discuss in this section is is identical to our previous post on Real-time object detection with deep learning and OpenCV; therefore, I will not be reviewing the code exhaustively.
For a deep dive into the code, please see the original post.
Instead, we’ll simply be using this code to benchmark the Raspberry Pi for deep learning-based object detection.
To get started, open up a new file, name it real_time_object_detection.py , and insert the following code:
# import the necessary packages from imutils.video import VideoStream from imutils.video import FPS import numpy as np import argparse import imutils import time import cv2
We then need to parse our command line arguments:
# construct the argument parse and parse the arguments
ap = argparse.ArgumentParser()
ap.add_argument("-p", "--prototxt", required=True,
help="path to Caffe 'deploy' prototxt file")
ap.add_argument("-m", "--model", required=True,
help="path to Caffe pre-trained model")
ap.add_argument("-c", "--confidence", type=float, default=0.2,
help="minimum probability to filter weak detections")
args = vars(ap.parse_args())
Followed by performing some initializations:
# initialize the list of class labels MobileNet SSD was trained to
# detect, then generate a set of bounding box colors for each class
CLASSES = ["background", "aeroplane", "bicycle", "bird", "boat",
"bottle", "bus", "car", "cat", "chair", "cow", "diningtable",
"dog", "horse", "motorbike", "person", "pottedplant", "sheep",
"sofa", "train", "tvmonitor"]
COLORS = np.random.uniform(0, 255, size=(len(CLASSES), 3))
# load our serialized model from disk
print("[INFO] loading model...")
net = cv2.dnn.readNetFromCaffe(args["prototxt"], args["model"])
We initialize CLASSES , our class labels, and corresponding COLORS , for on-frame text and bounding boxes (Lines 22-26), followed by loading the serialized neural network model (Line 30).
Next, we’ll initialize the video stream object and frames per second counter:
# initialize the video stream, allow the camera sensor to warm up,
# and initialize the FPS counter
print("[INFO] starting video stream...")
vs = VideoStream(src=0).start()
# vs = VideoStream(usePiCamera=True).start()
time.sleep(2.0)
fps = FPS().start()
Wwe initialize the video stream and allow the camera warm up for 2.0 seconds (Lines 35-37).
On Line 35 we initialize our VideoStream using a USB camera If you are using the Raspberry Pi camera module you’ll want to comment out Line 35 and uncomment Line 36 (which will enable you to access the Raspberry Pi camera module via the VideoStream class).
From there we start our fps counter on Line 38.
We are now ready to loop over frames from our input video stream:
# loop over the frames from the video stream while True: # grab the frame from the threaded video stream and resize it # to have a maximum width of 400 pixels frame = vs.read() frame = imutils.resize(frame, width=400) # grab the frame dimensions and convert it to a blob (h, w) = frame.shape[:2] blob = cv2.dnn.blobFromImage(cv2.resize(frame, (300, 300)), 0.007843, (300, 300), 127.5) # pass the blob through the network and obtain the detections and # predictions net.setInput(blob) detections = net.forward()
Lines 41-55 simply grab and resize a frame , convert it to a blob , and pass the blob through the neural network, obtaining the detections and bounding box predictions.
From there we need to loop over the detections to see what objects were detected in the frame :
# loop over the detections
for i in np.arange(0, detections.shape[2]):
# extract the confidence (i.e., probability) associated with
# the prediction
confidence = detections[0, 0, i, 2]
# filter out weak detections by ensuring the `confidence` is
# greater than the minimum confidence
if confidence > args["confidence"]:
# extract the index of the class label from the
# `detections`, then compute the (x, y)-coordinates of
# the bounding box for the object
idx = int(detections[0, 0, i, 1])
box = detections[0, 0, i, 3:7] * np.array([w, h, w, h])
(startX, startY, endX, endY) = box.astype("int")
# draw the prediction on the frame
label = "{}: {:.2f}%".format(CLASSES[idx],
confidence * 100)
cv2.rectangle(frame, (startX, startY), (endX, endY),
COLORS[idx], 2)
y = startY - 15 if startY - 15 > 15 else startY + 15
cv2.putText(frame, label, (startX, y),
cv2.FONT_HERSHEY_SIMPLEX, 0.5, COLORS[idx], 2)
On Lines 58-80, we loop over our detections . For each detection we examine the confidence and ensure the corresponding probability of the detection is above a predefined threshold. If it is, then we extract the class label and compute (x ,y) bounding box coordinates. These coordinates will enable us to draw a bounding box around the object in the image along with the associated class label.
From there we’ll finish out the loop and do some cleanup:
# show the output frame
cv2.imshow("Frame", frame)
key = cv2.waitKey(1) & 0xFF
# if the `q` key was pressed, break from the loop
if key == ord("q"):
break
# update the FPS counter
fps.update()
# stop the timer and display FPS information
fps.stop()
print("[INFO] elapsed time: {:.2f}".format(fps.elapsed()))
print("[INFO] approx. FPS: {:.2f}".format(fps.fps()))
# do a bit of cleanup
cv2.destroyAllWindows()
vs.stop()
Lines 82-91 close out the loop — we show each frame, break if ‘q’ key is pressed, and update our fps counter.
The final terminal message output and cleanup is handled on Lines 94-100.
Now that our brief explanation of real_time_object_detection.py is finished, let’s examine the results of this approach to obtain a baseline.
Go ahead and use the “Downloads” section of this post to download the source code and pre-trained models.
From there, execute the following command:
$ python real_time_object_detection.py \ --prototxt MobileNetSSD_deploy.prototxt.txt \ --model MobileNetSSD_deploy.caffemodel [INFO] loading model... [INFO] starting video stream... [INFO] elapsed time: 54.70 [INFO] approx. FPS: 0.90
As you can see from my results we are obtaining ~0.9 frames per second throughput using this method and the Raspberry Pi.
Compared to the 6-7 frames per second using our laptop/desktop we can see that the Raspberry Pi is substantially slower.
That’s not to say that the Raspberry Pi is unusable when applying deep learning object detection, but you need to set your expectations on what’s realistic (even when applying our OpenCV + Raspberry Pi optimizations).
Note: For what it’s worth, I could only obtain 0.49 FPS when NOT using our optimized OpenCV + Raspberry Pi install — that just goes to show you how much of a difference NEON and VFPV3 can make.
A different approach to object detection on the Raspberry Pi
Using the example from the previous section we see that calling net.forward() is a blocking operation — the rest of the code in the while loop is not allowed to complete until net.forward() returns the detections .
So, what if net.forward() was not a blocking operation?
Would we able to obtain a faster frames per second throughput?
Well, that’s a loaded question.
No matter what, it will take approximately a little over a second for net.forward() to complete using the Raspberry Pi and this particular architecture — that cannot change.
But what we can do is create a separate process that is solely responsible for applying the deep learning object detector, thereby unblocking the main thread of execution and allow our while loop to continue.
Moving the predictions to separate process will give the illusion that our Raspberry Pi object detector is running faster than it actually is, when in reality the net.forward() computation is still taking a little over one second.
The only problem here is that our output object detection predictions will lag behind what is currently being displayed on our screen. If you detecting fast-moving objects you may miss the detection entirely, or at the very least, the object will be out of the frame before you obtain your detections from the neural network.
Therefore, this approach should only be used for slow-moving objects where we can tolerate lag.
To see how this multiprocessing method works, open up a new file, name it pi_object_detection.py , and insert the following code:
# import the necessary packages from imutils.video import VideoStream from imutils.video import FPS from multiprocessing import Process from multiprocessing import Queue import numpy as np import argparse import imutils import time import cv2
For the code walkthrough in this section, I’ll be pointing out and explaining the differences (there are quite a few) compared to our non-multprocessing method.
Our imports on Lines 2-10 are mostly the same, but notice the imports of Process and Queue from Python’s multiprocessing package.
Next, I’d like to draw your attention to a new function, classify_frame :
def classify_frame(net, inputQueue, outputQueue): # keep looping while True: # check to see if there is a frame in our input queue if not inputQueue.empty(): # grab the frame from the input queue, resize it, and # construct a blob from it frame = inputQueue.get() frame = cv2.resize(frame, (300, 300)) blob = cv2.dnn.blobFromImage(frame, 0.007843, (300, 300), 127.5) # set the blob as input to our deep learning object # detector and obtain the detections net.setInput(blob) detections = net.forward() # write the detections to the output queue outputQueue.put(detections)
Our new classify_frame function is responsible for our multiprocessing — later on we’ll set it up to run in a child process.
The classify_frame function takes three parameters:
net: the neural network object.inputQueue: our FIFO (first in first out) queue of frames for object detection.outputQueue: our FIFO queue of detections which will be processed in the main thread.
This child process will loop continuously until the parent exits and effectively terminates the child.
In the loop, if the inputQueue contains a frame , we grab it, and then pre-process it and create a blob (Lines 16-22), just as we have done in the previous script.
From there, we send the blob through the neural network (Lines 26-27) and place the detections in an outputQueue for processing by the parent.
Now let’s parse our command line arguments:
# construct the argument parse and parse the arguments
ap = argparse.ArgumentParser()
ap.add_argument("-p", "--prototxt", required=True,
help="path to Caffe 'deploy' prototxt file")
ap.add_argument("-m", "--model", required=True,
help="path to Caffe pre-trained model")
ap.add_argument("-c", "--confidence", type=float, default=0.2,
help="minimum probability to filter weak detections")
args = vars(ap.parse_args())
There is no difference here — we are simply parsing the same command line arguments on Lines 33-40.
Next we initialize some variables just as in our previous script:
# initialize the list of class labels MobileNet SSD was trained to
# detect, then generate a set of bounding box colors for each class
CLASSES = ["background", "aeroplane", "bicycle", "bird", "boat",
"bottle", "bus", "car", "cat", "chair", "cow", "diningtable",
"dog", "horse", "motorbike", "person", "pottedplant", "sheep",
"sofa", "train", "tvmonitor"]
COLORS = np.random.uniform(0, 255, size=(len(CLASSES), 3))
# load our serialized model from disk
print("[INFO] loading model...")
net = cv2.dnn.readNetFromCaffe(args["prototxt"], args["model"])
This code is the same — we initialize class labels, colors, and load our model.
Here’s where things get different:
# initialize the input queue (frames), output queue (detections), # and the list of actual detections returned by the child process inputQueue = Queue(maxsize=1) outputQueue = Queue(maxsize=1) detections = None
On Lines 56-58 we initialize an inputQueue of frames, an outputQueue of detections, and a detections list.
Our inputQueue will be populated by the parent and processed by the child — it is the input to the child process. Our outputQueue will be populated by the child, and processed by the parent — it is output from the child process. Both of these queues trivially have a size of one as our neural network will only be applying object detections to one frame at a time.
Let’s initialize and start the child process:
# construct a child process *indepedent* from our main process of
# execution
print("[INFO] starting process...")
p = Process(target=classify_frame, args=(net, inputQueue,
outputQueue,))
p.daemon = True
p.start()
It is very easy to construct a child process with Python’s multiprocessing module — simply specify the target function and args to the function as we have done on Lines 63 and 64.
Line 65 specifies that p is a daemon process, and Line 66 kicks the process off.
From there we’ll see some more familiar code:
# initialize the video stream, allow the cammera sensor to warmup,
# and initialize the FPS counter
print("[INFO] starting video stream...")
vs = VideoStream(src=0).start()
# vs = VideoStream(usePiCamera=True).start()
time.sleep(2.0)
fps = FPS().start()
Don’t forget to change your video stream object to use the PiCamera if you desire by switching which line is commented (Lines 71 and 72).
Once our vs object and fps counters are initialized, we can loop over the video frames:
# loop over the frames from the video stream while True: # grab the frame from the threaded video stream, resize it, and # grab its dimensions frame = vs.read() frame = imutils.resize(frame, width=400) (fH, fW) = frame.shape[:2]
On Lines 80-82, we read a frame, resize it, and extract the width and height.
Next, we’ll work our our queues into the flow:
# if the input queue *is* empty, give the current frame to # classify if inputQueue.empty(): inputQueue.put(frame) # if the output queue *is not* empty, grab the detections if not outputQueue.empty(): detections = outputQueue.get()
First we check if the inputQueue is empty — if it is empty, we put a frame in the inputQueue for processing by the child (Lines 86 and 87). Remember, the child process is running in an infinite loop, so it will be processing the inputQueue in the background.
Then we check if the outputQueue is not empty — if it is not empty (something is in it), we grab the detections for processing here in the parent (Lines 90 and 91). When we call get() on the outputQueue , the detections are returned and the outputQueue is now momentarily empty.
If you are unfamiliar with Queues or if you want a refresher, see this documentation.
Let’s process our detections:
# check to see if our detectios are not None (and if so, we'll
# draw the detections on the frame)
if detections is not None:
# loop over the detections
for i in np.arange(0, detections.shape[2]):
# extract the confidence (i.e., probability) associated
# with the prediction
confidence = detections[0, 0, i, 2]
# filter out weak detections by ensuring the `confidence`
# is greater than the minimum confidence
if confidence < args["confidence"]:
continue
# otherwise, extract the index of the class label from
# the `detections`, then compute the (x, y)-coordinates
# of the bounding box for the object
idx = int(detections[0, 0, i, 1])
dims = np.array([fW, fH, fW, fH])
box = detections[0, 0, i, 3:7] * dims
(startX, startY, endX, endY) = box.astype("int")
# draw the prediction on the frame
label = "{}: {:.2f}%".format(CLASSES[idx],
confidence * 100)
cv2.rectangle(frame, (startX, startY), (endX, endY),
COLORS[idx], 2)
y = startY - 15 if startY - 15 > 15 else startY + 15
cv2.putText(frame, label, (startX, y),
cv2.FONT_HERSHEY_SIMPLEX, 0.5, COLORS[idx], 2)
If our detections list is populated (it is not None ), we loop over the detections as we have done in the previous section’s code.
In the loop, we extract and check the confidence against the threshold (Lines 100-105), extract the class label index (Line 110), and draw a box and label on the frame (Lines 111-122).
From there in the while loop we’ll complete a few remaining steps, followed by printing some statistics to the terminal, and performing cleanup:
# show the output frame
cv2.imshow("Frame", frame)
key = cv2.waitKey(1) & 0xFF
# if the `q` key was pressed, break from the loop
if key == ord("q"):
break
# update the FPS counter
fps.update()
# stop the timer and display FPS information
fps.stop()
print("[INFO] elapsed time: {:.2f}".format(fps.elapsed()))
print("[INFO] approx. FPS: {:.2f}".format(fps.fps()))
# do a bit of cleanup
cv2.destroyAllWindows()
vs.stop()
In the remainder of the loop, we display the frame to the screen (Line 125) and capture a key press and check if it is the quit key at which point we break out of the loop (Lines 126-130). We also update our fps counter.
To finish out, we stop the fps counter, print our time/FPS statistics, and finally close windows and stop the video stream (Lines 136-142).
Now that we’re done walking through our new multiprocessing code, let’s compare the method to the single thread approach from the previous section.
Be sure to use the “Downloads” section of this blog post to download the source code + pre-trained MobileNet SSD neural network. From there, execute the following command:
$ python pi_object_detection.py \ --prototxt MobileNetSSD_deploy.prototxt.txt \ --model MobileNetSSD_deploy.caffemodel [INFO] loading model... [INFO] starting process... [INFO] starting video stream... [INFO] elapsed time: 48.55 [INFO] approx. FPS: 27.83
Here you can see that our while loop is capable of processing 27 frames per second. However, this throughput rate is an illusion — the neural network running in the background is still only capable of processing 0.9 frames per second.
Note: I also tested this code on the Raspberry Pi camera module and was able to obtain 60.92 frames per second over 35 elapsed seconds.
The difference here is that we can obtain real-time throughput by displaying each new input frame in real-time and then drawing any previous detections on the current frame.
Once we have a new set of detections we then draw the new ones on the frame.
This process repeats until we exit the script. The downside is that we see substantial lag. There are clips in the above video where we can see that all objects have clearly left the field of view…
…however, our script still reports the objects as being present.
Therefore, you should consider only using this approach when:
- Objects are slow moving and the previous detections can be used as an approximation to the new location.
- Displaying the actual frames themselves in real-time is paramount to user experience.
What's next? We recommend PyImageSearch University.
84 total classes • 114+ hours of on-demand code walkthrough videos • Last updated: February 2024
★★★★★ 4.84 (128 Ratings) • 16,000+ Students Enrolled
I strongly believe that if you had the right teacher you could master computer vision and deep learning.
Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?
That’s not the case.
All you need to master computer vision and deep learning is for someone to explain things to you in simple, intuitive terms. And that’s exactly what I do. My mission is to change education and how complex Artificial Intelligence topics are taught.
If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to successfully and confidently apply computer vision to your work, research, and projects. Join me in computer vision mastery.
Inside PyImageSearch University you'll find:
- ✓ 86 courses on essential computer vision, deep learning, and OpenCV topics
- ✓ 86 Certificates of Completion
- ✓ 115+ hours of on-demand video
- ✓ Brand new courses released regularly, ensuring you can keep up with state-of-the-art techniques
- ✓ Pre-configured Jupyter Notebooks in Google Colab
- ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
- ✓ Access to centralized code repos for all 540+ tutorials on PyImageSearch
- ✓ Easy one-click downloads for code, datasets, pre-trained models, etc.
- ✓ Access on mobile, laptop, desktop, etc.
Summary
In today’s blog post we examined using the Raspberry Pi for object detection using deep learning, OpenCV, and Python.
As our results demonstrated we were able to get up to 0.9 frames per second, which is not fast enough to constitute real-time detection. That said, given the limited processing power of the Pi, 0.9 frames per second is still reasonable for some applications.
We then wrapped up this blog post by examining an alternate method to deep learning object detection on the Raspberry Pi by using multiprocessing. Whether or not this second approach is suitable for you is again highly dependent on your application.
If your use case involves low traffic object detection where the objects are slow moving through the frame, then you can certainly consider using the Raspberry Pi for deep learning object detection. However, if you are developing an application that involves many objects that are fast moving, you should instead consider faster hardware.
Thanks for reading and enjoy!
And if you’re interested in studying deep learning in more depth, be sure to take a look at my new book, Deep Learning for Computer Vision with Python. Whether this is the first time you’ve worked with machine learning and neural networks or you’re already a seasoned deep learning practitioner, my new book is engineered from the ground up to help you reach expert status.
Just click here to start your journey to deep learning mastery.
Download the Source Code and FREE 17-page Resource Guide
Enter your email address below to get a .zip of the code and a FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning. Inside you'll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL!
Thanks, dr.adrian for this great article!
It is working fine with me, but I have a small question.
I want to detect insects in real time using raspberry, do you recommend any pre-trained module that can do the object detection for insects not just persons, dogs, sofa …etc? if there isn’t what should u do in general to achieve my aim?
thanks again
I’m not aware of a pre-trained model that specifically detects insects. I would suggest fine-tuning an existing, pre-trained neural network. I discussing fine-tuning inside Deep Learning for Computer Vision with Python.
Hello,
actually I am looking to do project with pie to detect the preloaded or default setup image and sending a control signal if it was detected, could suggest me please how to do with open cv is there any code with you.
Very impressive! My own experiments on RPi was about 3 or 4 seconds per frame. So almost 1 fps is quite an improvement from that. With a moderately wide-angle lens that could already be useful, unless you must have the object almost completely fill the frame. Does this code fully utilize all 4 cores on the RPi 3, or is there potentially some additional parallelization possible?
There are always more optimizations that can be made, it’s just a matter of if it’s worth it. The fully utilize all cores to their maximum potential we would need OpenCL (which to my knowledge) The Raspberry Pi does not support.
Thanks. I have yet to try your code, but if (for example) it only uses 2 cores and it’s CPU-bound rather than memory or I/O bound, you ought to get a speedup by simply instantiating two separate processes, one looking at odd frames and one doing even frames. But maybe it’s not that easy; you might run out of memory.
Speaking of OpenCL on Raspbery Pi: it is not 100% complete, but:
09-OCT-2017 : I present to you VC4CL (VideoCore IV OpenCL):
https://www.raspberrypi.org/forums/viewtopic.php?t=194952
Awesome, thanks for sharing.
Came across this article by accident. Great site. I realise this is a python based site but what are the speed improvements were this to be implemented in C++ for comparison sake?
Hi Jay — I’m glad you enjoyed the blog post. Python is just a wrapper around the original C/C++ code for OpenCV. So the speed will be very similar.
My pi3 runs this script at 98% CPU load, and core temperature reaching it’s peak. How do I bring this down?
In short, you can’t without reducing the frames per second processing pipeline. If you are concerned about peak temperature you should only process one frame every 10-15 seconds.
Hi Adrian,
Thanks for the reply, so how do i reduce the frames per second processing pipeline?
I would recommend inserting a
time.sleepcall at the end of each iteration of the loop.Hi Adrian,
Thanks a ton for remembering us Pi enthusiasts. I first got interested in image analysis after someone stole your beer, but was afraid you would lose interest in the Pi after purchasing the beast.
Looking forward to diving in to this post and reading the new book.
Cheers,
Dayle
Hi Dayle — I certainly have not lost interested in the Raspberry Pi, I’ve just primarily been focusing on deep learning tutorials lately 🙂
How is this method different from using Squezenet for object detection on a raspberry pi?
The one you posted a couple of weeks ago?
Also what are the pros and cons of using squezenet over this method?
SqueezeNet is an image classifier. It takes an entire image and returns a single class label. It does no object detection or localization.
The SSD and Faster R-CNN frameworks can be used for object detection. It requires that you take an architecture (SqueezeNet, VGGNet, etc.) and then train it using the object detection framework. This will minimize the joint loss between class label prediction AND localization.
The gist is that vanilla SqueezeNet and SSD are two totally different frameworks.
If you’re interested in learning more about deep learning (and how these architectures differ), I would definitely suggest working through Deep Learning for Computer Vision with Python where I cover these methods in detail.
i have this error , there is a problem here -> args = vars(ap.parse_args())
usage: real_time_object_detection.py [-h] -p PROTOTXT -m MODEL [-c CONFIDENCE]
real_time_object_detection.py: error: argument -p/–prototxt is required
Please read up on command line arguments.
I am also getting same error as above mentioned.
usage: pi_object_detection.py [-h] -p PROTOTXT -m MODEL [-c CONFIDENCE]
pi_object_detection.py: error: argument -p/–prototxt is required
I read out your suggestion of “command line argument” but didn’t able to understand how to use it.
So, help me ,how would i remove this error?
Hey Aman — this blog assumes some basic knowledge of working the command line. If you’re new to the command line that’s okay — we all start somewhere! Take the time to familiarize yourself with command line arguments before continuing. Otherwise, be sure to Google “command line name of your operating system” and and read up on how to use the command line for your OS.
hey aman did you find a solution to this error please if you have a solution tell me
Impressive tutorial.This article helped me a lot,thank you!
Fantastic, I’m glad to hear it Komoriii! 🙂
Great post as always, Adrian! I have learned a lot about computer vision from the content on your site.
Regarding doing AI on the Pi, I would personally not do detection and recognition on an edge device. At least not until they ship a Pi with an AI chip and a decent GPU! And maybe not even then, due to the high power (electricity) consumption of AI. I’d much rather use the Pi as a sensor + basic signal processor, WiFi over all the video / sensor signals to a CPU box, and run all the algorithms on that box.
So I guess I agree with your conclusions.
Hi Sachin — thanks for the comment. I actually discuss the tradeoffs of using the Raspberry Pi for deep learning in this post. In general, I do agree with you that a Raspberry Pi should not be used for deep learning unless under very specific circumstances.
Any specific CPU box that you think is a good (relatively cheap) option for doing the post-processing?
What type of post-processing are you referring to? The type of post-processing you are doing would impact my suggestion.
Hi Adrian, i love ur work, Sir can you please tell me how i can compute :the (x, y)-coordinates of the bounding box for the object if i’m using Squeezenet instead of MobileNet SSD caffe Model on my raspberry pi 3…..what i supposed to change in “box = detections[0, 0, i, 3:7] * np.array([w, h, w, h])” so that it will work with detecting object in squeezenet with highest probablity (I’m able to find the index of object with highest probability till now with your previous post on deep learning) any help is appreciated 🙂 [i have raspberrian stretch with opencv3.30 -dev installed(neon optimized)]
Are you using your own custom trained SqueezeNet using the SSD framework? Keep in mind that you cannot swap out networks trained for image classification and use them for object detection.
Thanks for reply :), i figured that out eventually (Previously i was using SqueezeNet v1.1 imageclassifier instead of SSD framework) but i found another great SqueezeNet-SSD (based on Squeezenet v1.1)
https://github.com/chuanqi305/SqueezeNet-SSD
I benchmarked it using this script but see merely any difference from MobileNetSSD …..both have same FPS around 1.72FPS(opencv optimized) on normal script and 29.4FPS(opencv optimized) using this Multithreaded script…..Through Squeezenet v1.1 (around 0.4 ms on raspberry pi 3) is way faster than any other image classifier, why this Squeezenet-SSD is slower ? I’m totally confused :\
SqueezeNet v1.1 is slower because it utilizes ResNet-like modules. These increase accuracy, but slow the network down a bit.
I think that’s not the point. It’s the difference between SqueezeNet-SSD and SqueezeNet that causes the speed difference.
Thanks for the note, I must have misunderstood the original comment when I read it the first time. In general, yes, image classification networks will tend to run faster than object detection networks.
trivial point, no need to publish, but you say ‘net.forwad()’ vice ‘net.forward()’ at least twice
Thanks for letting me know, David! I have updated the post.
Hello Adrian,
Thanks for this great article!
Actually, I’m a starter in deep learning, but I want to use the Raspberry Pi for deep learning object detection.
One question: Does x-nor net improves the speed results?
(https://pjreddie.com/media/files/XNOR_ECCV2.pdf)
I haven’t used XNOR net to benchmark it, but from the paper the argument is that you can use XNOR net to speedup the network. You end up saving (approximately) 32x memory and 58x faster convolutional operations.
hi Adrian,
Can we only detect people or car (i.e. specific class) by changing the python code?
Yes. Check the
idxof the predicted class and filter out the ones you are uninterested in.Hi Adrian, will this in any way improve the performance i.e will it decrease required processing power from the pi?
No, it will not speedup or slowdown the Pi. A full forward pass of the network still needs to be performed.
Hi Adrian,
I’ve been wanting to do this for months, and it was this that got me to your website, so thank you!
I have been trying to tweak the code so that I can grab the frame when an object is detected and save that as a .jpg in a folder as:
/Pictures/{label}/{label}_{confidence}.jpg
following the ‘Accessing the Raspberry Pi Camera with OpenCV and Python’ tutorial, so I can have my own dataset by using the Pi to do all the hard work.
However I keep getting an mmalError message.
How would you go about taking a frame from the Pi when it detects an object and saving that frame in a folder with that object’s class so you can have a dataset to work with?
Thanks!
I would suggest debugging this line-by-line. Try to determine what line is throwing the error by inserting “print” statements. If you can provide that, I can try to point you in the right direction.
From there, you can use the
cv2.imwriteto save your image to disk. You can format your filename using the detected label and associated probability returned bynet.forward.Hi jsmith,
were you able to solve this? Pls help Im looking out for exactly same solution.
Hi Adrian,
I’m having issues that net.forward() seems to return inconsistent results. For example, I have two frames. One frame with my cat, one frame without. If I process frame-without-cat, the cat is not found. If I process frame-with-cat, it is correctly found. However, if I do this:
detect frame-without-cat
detect frame-with-cat
detect frame-without-cat
I get the following results:
cat not detected
cat detected
cat detected
Which is inconsistent, as the third and first frame should have the same result. However, if for each detection I reload the model, this issue does not occur. It looks as though the net retains previous detections?
Do you have any idea what this could be? If need be, I can provide example data and source.
Hi Roald — this is indeed strange; however, I would double-check your images and 100% verify that you are passing in the correct images as you expect. The network should not be retaining any type of memory from previous detections. Secondly, check the confidence (i.e., probability) of your false-positives and see if you can increase the confidence to filter out these weak detections.
Hi Adrian,
Downloaded the example for this post and ran it:
$ $ python3 real_time_object_detection.py
usage: real_time_object_detection.py [-h] -p PROTOTXT -m MODEL [-c CONFIDENCE]
real_time_object_detection.py: error: the following arguments are required: -p/–prototxt, -m/–model
How do I specify the path for the protext file and the model file to ap.add_argument. They are all in the same folder.
Please read up on command line arguments. This will enable you to learn more about command line basics. Furthermore, I also present examples on how to run the Python script via the command line inside this blog post.
Hi, i also got the same problem while running the python program.
Have you solved this or still stucked in this problem
Did you solve
the problem?
i want to track a ball is that code reliable to do the task
I cover ball/object tracking inside this post.
Hello Adrian Sir,
I’m great fan of all your articles and I ought learn more from you.
I’m presently running on ubuntu mate on raspberry pi 3, I even optimized pi, the way you told in previous post. I removed the unnecessary applications of ubuntu mate which I’m not using. The code runs without any error. But the problem is GPU rendering, I get the frame, but I can’t visualize the video it’s recording. it continuously lags after code starts running.
The Raspberry Pi will only be able to process ~1 frame per second using this deep learning-based object detection method so the lag is entirely normal. Is there another type of lag you are referring to?
Hi Adrian
I tried to run the codes above, but the result was
$ python pi_object_detection.py –prototxt MobileNetSSD_deploy.prototxt.txt –model MobileNetSSD_deploy.caffemodel
[INFO] loading model…
[INFO] starting process…
[INFO] starting video stream…
…
(h, w) = image.shape[:2]
AttributeError: ‘NoneType’ object has no attribute ‘shape’
and I also read the post on 26th Dec 2016, still have no clue
what should I do?
Hi Reed — this error usually occurs when the webcam didn’t properly grab an image. You could put the following between lines 80 and 81:
if frame is None:continue
now it gives me:
NameError: name ‘frame’ is not defined
hello reed
you only have to uncomment the line no. 76 and comment line no. 75 in your ”pi_object_detection.py” file.
nothing any more to add anything !
enjoy !!
hi,
will u please tel me how to detect human(person) within a fraction of second.
i used this coding technique, found idx values and compared with threshold but its having one to two second delay.
if idx == 15 and threshold>0.2:
print ‘human detected’
please suggest how to detect human within fraction of second.
Hi,
please help to find human within fraction of second using raspberry pi-3.
I would suggest taking a look at this blog post to start. Even with an optimized OpenCV install you are not going to be able to detect objects in a fraction of a second on the Raspberry Pi, it’s simply too slow.
Thanks for ur reply,
Will u please give me suggestions, which board is used to do this operation means with a fraction of second detecting human…. Which board is better to use
I would suggest trying the Jetson TX1 and TX2.
hi,
nice work
Thanks Sachin!
Hi Adrian,
Fantastic Work. Thanks a lot for sharing the code. I used an IP camera that streams h264/Jpeg camera over RTSP and was able to see fps of 0.6 as compared to 0.9 you see. Not Bad I guess. Exploring avenues to increase the fps count.
Nice job, Dharshan!
Hi, can you share code to RTSP input?
hi does it have any image processing in matlab soft
Are you referring to the Raspberry Pi? The Raspberry Pi does not include any out-of-the-box image processing software for MATLAB.
Hi Adrian! I wanted to know if it is needed to have opencv 3.3 to run this programme since everytime i try to intall it my rasp gets frozen. i have installed previous versions of opencv without problem.
Nice work by the way 🙂
Hi Andres — yes, OpenCV 3.3 is required for this blog post. If you’re having trouble getting OpenCV installed on your Pi, please see this tutorial.
Thx for this great blog, I got it working on my robot with raspPI3&arduino. I am streaming the output frames to a webclient, because the robot has no monitor attached. However, it is very slow given all the other processes that are running in my robot script. So it takes 10 seconds to identify the objects…. Is it possible to limit the number of known objects to look for, for example to look only for persons, sofa, potted plant and chair? Would this speed up the detection process? (since i don’t have airplanes or cars in my living room where the robot is supposed to manouvre..
The speed of the object detection has nothing to do with the number of classes — it has to do with the depth/complexity of the network (within reason). The deeper/more complex the network, the slower it will run. I have a tutorial on optimizing the Raspberry Pi but in general, if you want to deploy a deep learning object detector, the Raspberry Pi is not the right hardware. I would recommend an NVIDIA Jetson.
Hey. That was a great tutorial. And eventually worked for me. I want to make this thing to run at my raspberry pi boots. I tried it in linked way (https://pastebin.com/zjyEq99c) but I failed. Is there any way to do it? I’m looking for the help.
I would recommend following this tutorial where I demonstrate how to run the script from boot. Be sure to take a look at the comments as well where we discuss a few other alternative methods.
Hi Adrian,
Really Nice. Keep up the good work.
I am using Pi & open cv for the first time in my life and i simply followed your mentioned steps from the installation tutorial and it really worked.
Then I integrated this py code on my stretch os with external Logitech webcam (C920-C) and it did work for the first time and i was really happy to see it working.
Later, I restarted Pi to check the behaviour again and all of sudden it stopped working and started throwing error on command window.Now i am stuck..
Error message :
“VIDEOIO ERROR: V4L2: Pixel format of incoming image is unsupported by OpenCV
Unable to stop the stream: Device or resource busy”.
Then i started researching on google, found many blogs and its a common issue, but unfortunately i could not crack it.
Do i need to change the setting in ” /home/pi/opencv-3.3.0/CMakeLists.txt” from OFF to ON in ” OCV_OPTION(WITH_LIBV4L “Use libv4l for Video 4 Linux support” ON ”
and recompile the openCV.
If yes, should i use the same commands from #Step5 to #Step7 of compile and Install openCv, the whole process will take 10h again to do it. 🙁
Do you have any other solution for this to solve it easily.
Thanks!
Dear Adrian,
Thank for this website it helps me to understand how to do new things. however I am running into this problem and I cannot figure it out:
(cv)pi@raspberry:~/real-time-object-detection $ python real_time_object_detection.py –prototxt MobileNetSSD_deploy.prototxt.txt –model MobileNetSSD_deploy.caffemodel
[INFO] loading model…
[INFO] starting video stream…
OK
Traceback (most recent call last):
File “real_time_object_detection.py”, line 48, in
frame = imutils.resize(frame, width=400)
…
(h, w) = image.shape[:2]
AttributeError: ‘NoneType’ object has no attribute ‘shape’
this file exists /home/pi/.virtualenvs/cv/local/lib/python2.7/site-packages/imutils/convenience.py
and I am pretty sure I did not messed up the installation.
Do you have any advice?
Best regards,
It sounds like your Raspberry Pi is unable to read the frame from your camera sensor as
vs.read()is returning None. Are you using the Raspberry Pi camera module? Or a USB camera? I would also suggest reading up on NoneType errors as I discuss here.How did you solve it? I have same issue.
Thank You Adrian. But I wonder if I can detect the cube. Is there any packages like Caffe that can detect the cube of a particular dimension. Thank You in advance.
Hey Benni — are you trying to detect just a cube? Or the dimensions associated with a cube?
Hi Adrian, thanks for the great tutorial. I am following your tutorials from long time. I followed this tutorial and it worked as expected, Thanks a lot.
Does the detection time depend on image resolution. Can we reduce the detection time if connect any low resolution USB camera to raspberry pi.
The input resolution to the neural network is fixed in this case (300x300px). Reducing the resolution from the USB camera to the Pi will speedup the frame acquisition rate but keep in mind that the bottleneck here isn’t frame I/O — it’s the speed of the SSD performing object detection.
How can i track a detected object? E.g. I want to track the movement of first detected person among all people present in front of camera.
I would suggest taking a look at tracking algorithms, in particular “correlation tracking”. The dlib library has an implementation of correlation tracking.
hey, error = ‘module object has no attribute
‘
Hi Adrian. Maybe you know how to stream this real time object detection to the local area website (web browser)?
Hi Dom — I don’t have any tutorials on streaming the output frames to a web browser but I’ll add this to my queue for 2018. Thank you for the suggestion!
How to stream object detection in web? Maybe i can make it with flask or django? Thanks.
Hey Gary — I do not have any tutorials for RTSP or MJPEG streaming but I’ll add it to my queue!
Hi, Adrian. How is the recent progress in MJPEG research? I am looking forward to viewing your frames in the form of MJPEG stream!
Hey Yin, thanks for following up. To be honest I haven’t had a chance to dive into it yet. I’ve been wrapping up a series of other posts on deep learning but I will try to do it sometime in 2018.
Hi Adrian .thanks for the great tutorial. I download the code and run ‘pi_object_detection.py’ on raspberry pi and I have an error…”no module named ‘imutils’ ” then I open a terminal and type ‘pip install imutils’
and recieve ‘Successfully installed imutils-0.4.5’ but there is also same error
can you help me please
‘
Are you using a Python virtual environments? If so, make sure you use the “workon” command to access your Python virtual environment, install imutils, and then execute the script:
Adrian,
Have you considered the possibilities of the new Google AIY Vision kit?
Seems to be some kind of co-processor/microcontroller add on board (vision bonnet) that is supposed to let a PiZero-W run deep learning object detection and classification models.
I got one, but I need to return it as the connector for the flat cable broke when I tried to open it, so I didn’t get to first base 🙁
Presumably it could work with the Pi3 as well.
Google provides a “compiler” that takes a tensorflow “frozen model” and produces code to run on the vision bonnet.
Obviously my hope here would be to get a higher frame rate than Pi3 can provide. The cost of the vision bonnet kit (~$45) and a PiZero-W (~$10) would not be too much above the price of a Pi3 if we can increase the frame rate significantly.
I am interested in playing around with the Google AIY Vision Kit but I haven’t purchased one yet. I’m still a bit concerned with using a Pi Zero. With only a single core responsible for all system + user operations your frame rate will suffer even if you are using threading to decrease I/O latency.
I am using your Raspberry Pi image from your course, and the above code, and get the following error. Googling the error doesn’t give me a clear path to resolving this, can you please advise?
File “pi_object_detection.py”, line 55, in
net = cv2.dnn.readNetFromCaffe(args[“prototxt”], args[“model”])
AttributeError: ‘module’ object has no attribute ‘dnn’
Hey Mark — you need OpenCV 3.3 or greater to access the “dnn” module that contains the deep learning features. Any OpenCV version prior to OpenCV 3.3. does not include them.
I’m getting this error as well. I thought we downloaded OpenCV3.4 but when I do the version check it’s 3.1.0. I used your instructions on installing OpenCV on Raspbian and that was also the version that was displayed in your tutorial. Am I missing something?
Unfortunately, it sounds like you may have downloaded OpenCV 3.1 rather than OpenCV 3.3 or OpenCV 3.4. I would go back to the OpenCV install tutorial you used and ensure you download the source code to OpenCV 3.3 or OpenCV 3.4 and then recompile. I hope that helps!
Good morning and Happy New Year,
i am getting this error message:
[INFO] loading model…
Traceback (most recent call last):
File “pi_object_detection.py”, line 56, in
net = cv2.dnn.readNetFromCaffe(args[“prototxt”], args[“model”])
AttributeError: ‘module’ object has no attribute ‘dnn’
Any idea ? Thanks in advance!
Hi Paul — you need to have OpenCV 3.3 installed on your system. Anything older than OpenCV 3.3 does not have the “dnn” module.
Can I use readNetFromTensorflow(.pb, .prototxt) instread readNetFromCaffe? It can detect object same Caffe or not?
The
readNetFromTensorflowfunction still doesn’t work in all cases. Whether or not it detects the same objects as the Caffe example in this blog post depends on what your TensorFlow network was trained on.Thank so much Adrian. I have weight.pb file from trained my own dataset in Tensorflow. But I don’t know how to train dataset with Caffe SSD_Mobilenet Model. Do you have plan to post tutorial train own images image dataset with Caffe?
You can try importing your TensorFlow weights via
cv2.dnn.readNetFromTensorflowbut it’s very likely that it will fail. The TensorFlow import methods are not (currently) as reliable and robust as the Caffe ones. As for training your own custom object detectors, take a look at Deep Learning for Computer Vision with Python where I cover how to train your own custom deep learning-based object detectors in detail.https://github.com/opencv/opencv/issues/10043
as I can see there is one tested ssd mobilenet TF model that can be imported without issues but be careful with the version of the model.
Thank you for sharing this, Sergei!
hello Adrian,
i have been following your tutorial & i really liked it , i want to detect a street pole with sign on it.. can you tell me how to do edit the above pole accordingly, thank you
The model provided with this example was trained on the COCO dataset. You would need to train your own custom model to recognize street poles and signs. To start, use this tutorial to gather example images of what you’re trying to detect. I then demonstrate how to train your own custom object detectors inside my book, Deep Learning for Computer Vision with Python — recognizing street signs is even one of the included tutorials!
As I discuss in this blog post the Raspberry Pi can be very slow for deep learning. If you are using a deeper, larger model the inference time will also be larger.
As for transmitting a frame from your Pi to a PC you could use SFTP, Dropbox API, or even dedicated messaging passing libraries such ZeroMQ or RabbitMQ.
I m getting an error like module object has no attribute dnn
in the line net=cv2.dnn…
And i have open cv version 3.3.1
That is indeed strange. Can you fire up a Python shell and verify your OpenCV version just to be safe?
Additionally, are you using a Python virtual environment? Do you have any other OpenCV versions installed on your Raspberry Pi? And how did you install OpenCV on your Raspberry Pi — did you use one of my install tutorials on PyImageSearch?
Well it seems i didn’t have open cv installed in pi.I did it by your method and it is successfully done.One more problem is i created the real_time_obeject_detection file and downloaded the caffemodel etc.when i m executing i m getting error no module named imutils.
Also the mobilenet ssd prototxt and caffemodel are saved in my downloads so should i make another file in pi3 to access them?I mean how the program will access that without knowing where it is saved?
Make sure you install the “imutils” library:
$ pip install imutilsYou can supply the paths to your prototxt and caffemodel files via the command line arguments, exactly as I do in the blog post. When you execute the script simply pass the paths to the prototxt/caffemodel files where they live on your system.
Hi,Adrian. I use OpenCV and Tensorflow (Object Detetion API) on Raspberry Pi. But when I run code it have error gtk2.0 and gtk3.0 duplicate in same project. Do you know how to resolve it?
Hey Kelvin, what is the exact error message? I would need to see the exact error to provide a suggestion.
Hello Adrian, have you looked into neural compute stick developed by movidius intel. It is USB3 interfaced and works on raspberry pi 3. It has support for caffe and tensorflow too and claim to greatly speed up object detection using deep learning. Perhaps, you can make a tutorial on that soon. Cheers
I have! Look for a few posts on the Movidius Neural Compute Stick in February 🙂
Any info to share yet on the Movidius Neural Compute Stick yet?
I’ll be doing a blog post on the NCS either this coming Monday (Feb. 12th) or the following Monday (Feb. 19th).
$pi_object_detection.py [-h] -p PROTOTXT -m MODEL [-c CONFIDENCE]
pi_object_detection.py: error: argument -p/–prototxt is required
I am getting this error!!.
Please kindly tell me the steps after i download the code in my pc how to execute it on raspberrypi3.
Also i observed in video
you are in opencv virtual env and then inside pi-object-detection directory from where you executed files how can i do that?
You need to supply the command line arguments as I do in the blog post:
Notice how I have supplied values for
--prototxtand--modelvia command line arguments. If you’re new to command line arguments make sure you read up on them.I gave the same commands and was getting the error.After i downloaded the files it is saved in F: drive if i provide the path like –prototxt f:\MobileNetSSD_depploy.prototxt.txt in raspberrypi terminal i am still getting error messages .My doubt is can raspberry access the files from system f or any drives or we need to make a directory in raspberrypi first and copy all this files and then provide the path like home/pi/dir??
I’m a bit confused here. You downloaded the .prototxt and .caffemodel files to your F: drive on your Windows machine and now you’re trying to execute the code on your Raspberry Pi? You need to download/transfer the .prototxt and .caffemodel files to your Raspberry Pi before you can run the script on your Raspberry Pi. Your Raspberry Pi cannot access the hard drive on your laptop/desktop.
To use picamera i changed 35 and 36th lines but getting these errors
File “pi_object_detection.py”, line 75, in
…
from picamera.array import PiRGBArray
ImportError: No module named ‘picamera’
You need to install the “picamera” library:
$ pip install "picamera[array]"Additionally, read this guide on accessing your Raspberry Pi camera module.
Beside detection can this process count an object like if I want to use it a surveillance cam which not only saves video but can do a accounting of numbers of defined object going in and out.
Yes. Just create a counter for each object in your class labels and in the “for” loop on Line 58 you will increment the counter for the respective class label.
The problem becomes tracking each object and not re-counting it at each iteration. For that you should use a dedicated object tracking algorithm such as correlation trackers or centroid trackers.
Hello Sir,
this is a great project and now I’m working over it. Good demonstration. But I’m facing some problem . Like at 1st stacked at ….
[INFO] loading model…
[INFO] starting process…
[INFO] starting video stream…
…
(h, w) = image.shape[:2]
AttributeError: ‘NoneType’ object has no attribute ‘shape’
Then I added ..
if frame is None:
continue
between line 80 and 81 .
But now facing another problem. And it is
[INFO] loading model…
[INFO] starting process…
[INFO] starting video stream…
and it’s not showing any error even. Raspberry pi is in working state and cpu usage showing around 56 to 60%. One thing is raspberry pi is working normally. This will be very helpful if you tell me the problem and that will do lot to me. Thank you sir.
Hey Kawsur — make sure you take a look at the comments section. I’ve addressed the NoneType error a handful of times in the comments. Additionally, make sure you see this blog post where I discuss NoneType errors and how to solve them in more detail.
I fixed that none type error. But as I said it’s not showing me any error now. Just loading and loading. I checked my pi camera and it’s also working. Will you please tell me what I need to add to take frame from pi camera. I checked your pi_object_detection code and there is nothing about frame input. Maybe I need to add some line about taking frame from pi camera .
how did yu fixed it ??,,plz help me …
Hello sir,
Good Morning . It’s been running now. But one problem is now it’s showing a warning like ** (Frame:1162): WARNING **: Error retrieving accessibility bus address: org.freedesktop.DBus.Error.ServiceUnknown: The name org.a11y.Bus was not provided by any .service files
then camera star to detect object. Is there any problem with it?
This is a warning, not an error, and can be safely ignored.
How did you fixed it?
Hey Kawsur how did you fixed it ?
Hi Adrian,
Thank you very much for creating this amazing blog, I have learned a lot from you and implemented most of your projects.
I have a question, Could you please tell me if there’s more predefined classes that I could add like making pi able to detect objects like geometrical shapes and is it possible to detect text from live stream?
I am concerned because I am now working on a license plate recognition using opencv since I can’t get openalpr to work with live streaming.
Thanks for your help.
Greetings from Egypt.
I would suggest taking a look at the “Caffe Model Zoo” for more pre-trained models and seeing if any of them match your use case. More likely than not, you will likely need to train your own model.
Hi Adrian,
i am getting the error
usage: real_time_object_detection.py [-h] -p PROTOTXT -m MODEL [-c CONFIDENCE]
real_time_object_detection.py: error: the following arguments are required: -p/-
-prototxt, -m/–model
I’ve addressed this question a handful of times (please read the comments or ctrl + f the page and search for your error). See my replies to “Ahmad”, “aman”, and “Noble”. Thank you!
Hey, I am getting a error and I have no clue how to fix it, could you take a look?
When I run the py file I get the following error
[INFO] loading model…
Traceback (most recent call last):
File “real_time_object_detection.py”, line 31, in
net = cv2.dnn.readNetFromCaffe(args[“prototxt”], args[“model”])
AttributeError: ‘module’ object has no attribute ‘dnn’
Thank you
You need OpenCV 3.3 or greater for this post. OpenCV 3.3+ contains the “dnn” module. Previous versions of OpenCV do not.
Amazing work!
Just a few questions,
1. How do I add more classes in the mobileNet SSD? (I wanted to check for counterfeit currency in our country and hence would need to add my images to the trained dataset)
2. CPU load is at 98% while running the object detection program, will the pi cam help reducing that in any way? if not, how do I bring it down?
Thanks Adrian!
1. You would need to either train the model from scratch or fine-tune it (assuming you have already gathered your dataset of counterfeit currency, of course). I cover how to train your own custom deep learning object detectors inside Deep Learning for Computer Vision with Python. I would suggest starting there.
2. Unfortunately no, deep learning models are very computationally intensive. It does not have anything to do with the Pi camera. It’s simply the amount of computation required by the network for the detection.
Hi Adrian, a big fan of your work, actually i like to add on to Roger’s query of CPU load of 98 percent.
1. I was using this model to detect only people for a surveillance system based on pi camera v2.The problem is continual streaming,after 30 minutes there is a sign on the right side cropping up warning regarding the temperature of the core getting high.
2.So my query is how can we use this system to stream video continuously without the possibility of Pi getting damaged/burnt?
3. I have a lot of other queries as well and would love to interact with you if you have the time.
Hi Adrian, thanks for the excellent guide on object detection. I’ve got everything right, but when I run this code
spberry Pi: Deep learning object detection with OpenCVShell
$ python pi_object_detection.py \
–prototxt MobileNetSSD_deploy.prototxt.txt \
–model MobileNetSSD_deploy.caffemodel
I’m getting the following error
[INFO] loading model…
[INFO] starting process…
[INFO] starting video stream…
Illegal Instructor
Upto starting video stream is going right, but then I’m getting illegal instructor. I’ve searched the internet for a solution I couldn’t find the answer. Could you look into this error. I have a Raspberry Pi Model B+ V1.2 and Ras Pi camera module v2 noir and running raspbian stretch.
Hey Shriram — I would suggest inserting “print” statements throughout the code to debug what line is causing the issue. That will help determine what the problem is.
The code is running till line no. 80
80 frame = vs.read()
When I include a print statement after the above line of code, it isn’t getting printed and throws Illegal Instructor error. I believe the following line of code throws the error
81 frame = imutils.resize(frame, width=400)
Could you help out with the problem?
Great job diagnosing the problem, Shriram. I wonder if it could be an issue related to the VideoStream class used in conjunction with your camera. Perhaps a threading issue of some sort. Just to clarify — are you using a Raspberry Pi camera module or a USB webcam?
I’m using a Raspberry Pi camera module V2 (NoIR) and Raspberry Pi Model B+ V1.2 .
Try starting with this tutorial and see if you can access the NoIR camera via the raw “picamera” library first. Once you debug that move on to VideoStream.
I tried the tutorial you mentioned and the NoIR is working perfectly for ‘picamera’ library and VideoStream was also good. I found that when I remove the code,
frame = imutils.resize(frame, width=400)
the programs runs, but in the video output, object detection isn’t being done. imutils.resize is the one causing the ‘Illegal instructor’. Can you help me out with this error? I tried it with some other values for width, but it didn’t execute.
I even tried using cv2.resize, but then found that it can resize only images.
I’m not sure what the exact reason is for this error. Does cv2.resize produce the same “illegal instruction” error as well?
Your Pi Reboots? Wow, that is odd behavior! I haven’t encountered this before, but it sounds like your Pi might have a firmware issue or it’s immediately getting overheated and dying. I would suggest trying with a fresh install of Raspbian.
Hi Adrian.I followed your tutorial, and it works so well. Thanks.
But, my video stream is upside down. How can I rotate it 180 degrees/ flip it vertically?
I tried to put these near the beginning of the loop, but it doesnt work…
I’m using Pi camera version 2
(code removed due to formatting reasons)
If you are using the Raspberry Pi camera module you should take a look at the documentation and use either .hflip or .vflip. If you want a pure OpenCV approach, just use cv2.flip.
Hi Adrian
i keep getting a
cv2.error:(-2) Failed:fs.is_open()
can’t open “MobileSSD_deploy.caffemodel” in function ReadProtoFromBinaryFile
never mind Adrian, fixed it, thanks so much for your site, it’s being really helpful for my final year project
Congrats on resolving the issue, Jibola. Would you mind share your solution? I imagine it was that your path to the .caffemodel was invalid but I know a lot of other readers new to the command line have the same error and your solution, expressed in your own words, would help them.
how do you solve this one?
You need to double-check your command line arguments — your input path to the model files are likely incorrect..
Hi,Adrian.I need your help. I use yolo v2 to train my datasets,and i got the tiny-yolo-voc_final.weights finally,but in this passage use .caffemodel,what shold i do? I have used darkflow transforming the.weights to .h5,if i want use your programme on my pi,I can apply it derectly or need to train my datasets useing caffe again? thank you very,much! I’m stupid and crazy.
Hey Chopin — it’s been awhile since I’ve used DarkNet to train a network so I’m not sure on the best path to transform the resulting models to a Caffe model. I simply haven’t done it so I’m unfortunately not the right person to ask. In the worst case scenario you would need to re-train using Caffe rather than DarkNet, but you should spend some time researching the DarkNet to Caffe conversion and if it’s possible. I’m sorry I couldn’t be of more help here!
okay. thank you again.you are a gentlman.
How can i make raspberry pi say name to the object
You would need a “text to speech” Python library. I’m not sure which is the best one for that.
please you can change your code for say the name to object
I am happy to provide these tutorials for free to you but any additional functionality will need to be implemented in you. Do your research, experiment, and invest the time and I am confident that you’ll be able to do it 🙂
Is there any way to live stream the output frame window to a local webpage??
Hi Adrian,
Can you please let me know, the process to deploy this python code in Rasberry Pi … So that I can perform the object detection on the go …
Thanks
Guru
What specifically do you mean by “deploy” in this context? Can you elaborate?
Great work on this too, Adrian. I must say you have a good amount of patience to write this much with minute details.
I just have one question. The scrip you have written here in the article, does it run on RasPi to achieve real time object detection or it runs on a computer using RasPi for just sending video stream?
Thank you for the kind words, Nishant.
The script I used in this post actually executes on the Raspberry Pi. It does not send the video anywhere. All video + object detection is performed on the Raspberry Pi.
Your post helped me a lot..Thank you
Also, I have a question.
Can i live stream the output window frame of this object detection code to a webpage hosted locally on my raspberry pi. If there any way to achieve this?
Hey Sunil — I don’t have any posts on real-time streaming to a webpage but I’ll try to cover this in the future. Thank you for the suggestion.
I keep on getting this error everytime i run the py file. please help me
OpenCV Error: Unspecified error (FAILED: fs.is_open(). Can’t open “MobileNetSSD_deploy.prototxt.txt”) in ReadProtoFromTextFile, file /home/pi/opencv-3.3.0/modules/dnn/src/caffe/caffe_io.cpp, line 1113
Traceback (most recent call last):
File “pi_object_detection.py”, line 54, in
net = cv2.dnn.readNetFromCaffe(args[“prototxt”], args[“model”])
cv2.error: /home/pi/opencv-3.3.0/modules/dnn/src/caffe/caffe_io.cpp:1113: error: (-2) FAILED: fs.is_open(). Can’t open “MobileNetSSD_deploy.prototxt.txt” in function ReadProtoFromTextFile
Nvm i solved this issue
Yep, you would need to either train your own model or fine-tune an existing one.
I am getting this same exact error. I was wondering if you could go into greater detail on how to resolve it?
Hey Jack — this still sounds like a path issue. Be sure to read up on command line arguments before continuing. I think you may have supplied an invalid path to the file.
would you like to share your manner that you solve this probelm? i just encounter the same question.i use the vgg model for fine-tune and use the opencv loading this model.
Hi Adrian! I need to detect other things like (crumpled papers, candy wrapper, and other trash) can I just add it to my Classes or do I need to use other pre trained model? Thank you
got any luck with this ?
Do you know of any pre-trained models for traffic symbols i.e Start, Stop signs that I can use instead of the model you used?
Hey Huzzi — I have a model pre-trained on traffic signs (ex. stop sign, pedestrian ahead, etc.) inside Deep Learning for Computer Vision with Python.
I am getting this error-
File “pi_object_detection.py”, line 55, in
net = cv2.dnn.readNetFromCaffe(args[“prototxt”], args[“model”])
AttributeError: ‘module’ object has no attribute ‘dnn’
Make sure you are running OpenCV 3.3+. The “dnn” module was not added until OpenCV 3.3+. You may need to compile and reinstall OpenCV.
Hello Adrian,
Thanks for this blog and valuable information!
I tried to run forward pass to get detection using net.forward() . It works perfectly fine when there are object(s) trained on the image, although, when there isn’t an object, the net returns to me the detection of the last image that had the object. Has somebody stumbled on this issue?
It’s hard to say what could be causing that issue. My guess would be a latency between the frame read, the network predicting the object, and the frame being displayed to your thread. Keep in mind that the Pi is really slow for object detection so there may be a number of issues going on.
Hi Adrian sir,
I want to ask you a question that what should i do to make my rasberry pi cam focus on one object at a time.for example if I want to make a Mechanical arm to pick utensils like plate,cup,bottle ,etc.but want it to pick one object at a time so I want it to focus on one object at a time the what modification should i make the above code or what library should i use to do that.
There are a huge amount of modifications you would make, unfortunately far too many for me to list out in a single comment. You should do some research on robotics, in particular depth cameras, servos, and associated components. Detecting the actual object is just one small piece of this puzzle.
Can we detect specific class, capture the detected frame and upload to dropbox API?
Yes. You can check to see if the class is one you are interested in and if so, write the specific frame to disk via “cv2.imwrite”. You can then upload the file to Dropbox. See this blog post for an example of using the Dropbox API.
[INFO] loading model . . .
[INFO] starting process . . .
[INFO] starting video stream . . .
Illegal instruction
Help please 🙁
i’ve a USB webcam
Try inserting “print” statements to debug exactly where the “Illegal instruction” line is being thrown. Unfortunately it’s impossible to know based on your output.
have you debug this error.??
Hi Adrian,
Awesome series of posts. This allowed me to get OpenCV all up and running on my Pi. Thanks!
Only question I have is will it be quicker if it’s only detecting people? If so I’m assuming I have to redo the model so it’s only got person detection? The reason I’m asking is because I removed some of the class names in the CLASSES array and it didn’t work as expected (labelled me as a sheep!) which lead me to believe you have to use all the classes the model was trained for (and they have to be defined in a specific order?)
Can I just tweak the model files so that it only have the person trained for?
I’m hoping this all makes sense!
James.
Hey James — you cannot add or remove labels from the CLASSES list. That will not change how the network behaves. This has become a question often asked in the object detection posts here on PyImageSearch blog so I’m writing a blog post on the very topic. It will release in early May so please keep an eye out for it!
i have error object detection.py: error: the following arguments are required: -p/–prototxt, -m/–model
What is wrong?
If you’re new to command line arguments that’s okay, but you will need to read up on them before continuing. It’s a very quick fix but you need to read the post I just linked to first.
Hi,
I am getting this error
Traceback (most recent call last):
…
AttributeError: ‘module’ object has no attribute ‘dnn’
Can you please help me with this?
Thanks.
You need at least OpenCV 3.3 to utilize the “dnn” module. The “dnn” module is not included in previous versions of OpenCV. Check your OpenCV version and upgrade if necessary.
Awesome posts.thank you for the nice work!
I have a problem here, when I test the file of ‘A different approach to object detection on the Raspberry Pi ‘, which is ‘the pi_object_detection.py’. the result shows that:
TypeError: can’t pickle cv2.dnn_Net objects
Do you know how to fix that.
Can you clarify what your OpenCV version is? I haven’t ran into that error before.
Adrian,
Wishing you a very happy married life.
I seem to be running into exactly the same problem with TypeError: can’t pickle cv2.dnn_Net objects. I am using OpenCV 3.4.2 both on the PC as well as on the Pi. Interestingly, it works perfectly on the Pi but gives this error on the PC. I think it is something to do with Python’s Pickle. But I am using the same version of Python 3 on both Pi and PC. So not exactly sure why this error is coming up on the PC.
Adding in this Reply to help someone out there.
Thank you for the comment and sharing your experience Sudeep, I appreciate it. I’m sure with enough information we’ll be able to resolve the error. But it’s super strange the code works fine on the Pi but fails on the PC.
I also faced similar issues when i run the code in windows, then i tried the same code In ubuntu os it works properly.
Did u fix this error?
I am too getting same error
Can this code run any Net trained on SSD. I trained a shufflenet SSD on the same VOC dataset but I get an error saying OpenCV doesnt have the shufflenet_chanel_param
No, this code will not work for “any” network as OpenCV must have the equivalent layer implemented in its library. OpenCV would need to have the equivalent “shuffnet layer” implemented in the “dnn” module. It does not, hence the error.
thanks for the reply, i changed to squeezenet since the layers exist in the open cv library but everything takes too long, the Frame box opens after 30seconds and the video is stuck on a frame for over a minute, was the MobilSSD used here optimised in some way or is it that the pi can’t handle Squeezenet
The only optimizations done were to use my optimized OpenCV install. No further optimizations were done.
Hi, Adrian. I use command below:
#./mjpg_streamer -i “./input_raspicam.so” -o “./output_http.so -w ./www”
to transfer video collected by Raspberry Pi to web page,
so I can view picamera video on web page under my WLAN .
But now, I wanna view your “frames” result on web page, how can I export your results to web pages? I think that will be cool !
Hi Yin, I replied to your other comment on this post. Please see it.
hi adrian , please help
i tried ur code but i have problem
et = cv2.dnn.readNetFromCaffe(args[“prototxt”], args[“model”])
AttributeError: ‘module’ object has no attribute ‘dnn’
i dont know what do to no have this problem , please help me
You need to ensure you have at least OpenCV 3.3 installed. You can follow my OpenCV install tutorials to install a new, updated version of OpenCV on your system.
hi Adrain, i’m obtained a tiny yolo caffemodel from here
https://github.com/cinastanbean/caffe-yolo-1
but opencv gives the error Assertionfailed blobs.size() >= 3 when i run it with your code,
i’m assuming this error comes from the cv2.dnn.BlobfromImage . do you have any idea on the parameter i could change from your code to support tiny yolo. I already changed the size to (224,224)
Changing my caffemodel and prototxt file solves this but a new problem has occured. it detects evrything has background ,it’s not runing through the array , if i take away background from the list of classes ,it just detects everything has an aeroplane which is the next item on the list . Yolo isnt the only net that ive had this issue with using your code
Hi James — the list of classes, is just a list of strings. Do not modify it because what’s really important is the index in the list of each of those strings. In other words, you cannot simply remove elements from the classes list and expect the object detector to work properly. If you change the ordering of the list, then the indexes won’t match up. There’s a blog post coming out on 5/14/2018 explaining this.
Hey James — I’m not sure what the error is here. Hopefully in the future I can write a tutorial on YOLO + OpenCV, I would really like to.
Hello adrian
I ran the code. And the project worked. When a human object is detected, I can capture and record a photo. But what I want to do is how to record a 3-second video when a human object is detected. I could not … could you help …
Thank you …
You’ll want to refer to this blog post on saving video clips with OpenCV.
Hi,
I am getting this error
Traceback (most recent call last):
…
(h,w)=image.shape[:2]
AttributeError: ‘NoneType’ object has no attribute ‘shape’
Can you please help me with this?
Thanks.
See this blog post on “NoneType” errors with OpenCV and how to resolve them.
hi Adrian, can i use this to detect potholes on the road? if cant, could u help me to detect it. thank you
If you would like to use this method to detect potholes in a road you would first need to train a deep learning detector on pothole images. Do you have such example images?
Hi, I have completed the opencv install and am now wondering where to I execute the command. I tried to do it in the cv environment but got no such file or directory.
I have the script in the file in the downloads. Do I have to put it into the opencv thing somehow or can I locate it from the cv environment and then execute the command?
thanks!
I fixed both issues but I have one more. It has to do with the imutils module. Even though I imported it and checked the version in the (cv) environment. It still says no module. I have tried almost everything on the internet and nothing works. Do you know the problem?
thanks!
please respond ASAP.
You’ll want to make sure you install the “imutils” into your Python virtual environment:
Hi, I have one more question. All I want to detect is bird because if it detects other things beside bird my if statement will not work and it could possibly increase my FPS. I am now only getting 0.31 FPS.
I tried deleting all of the classes beside bird but I got errors. Do you know a quick and easy way to do this?
thanks
Please respond ASAP
This blog post will help you learn the fundamentals of deep learning object detection, including adding or removing classes.
hi sir
did you find any new optimization technique to reduce detection lag in this post or also sir can we reduce lag by overclocking the pi 3b
This isn’t true “lag” it’s that the network takes considerable amount of time to make predictions on the resource limited Raspberry Pi. You may want to look into using the Movidius NCS.
Hello Adrian,
Would it be a feasible idea to mount picamera on Raspberrypi robot car and do object detection using deep learning?
Thanks
Yes, you could certainly do that.
Thanks for the reply!
Hello Adrian, Thanks a lot for the practical tutorial.
I had one problem though. The video stream will freeze if the Pi connects to internet and sync the clock, while the example program is running.
This might be caused by imutils videostream bug, but I’m not sure, I tried restart videostream by calling vs.stop() and vs.start() again but the videostream still freezes. If I close and restart the whole program, videostream will function again, but will freeze if the Pi tries to sync with internet clock.
So, I wonder if there is any way to prevent imutils videostream from freezing when the Pi sync its internet clock?
Thanks in advance.
Wow, that’s odd. I have never encountered an issue where the Pi freezes when syncing the clock. It sounds like there may be an issue with your install of Raspbian. Could you try on a fresh install?
Hey Adrian
I have this confusing question about raspberry pi..
I did not get answers from Quora and other sites for it and finally i bring it to you.
If I use raspberry pi without a monitor and keyboard I know that I can use my laptop keyboard and display instead and in my understanding that is all for the purpose of setting up the raspberry pi. If I need to use it for outdoor purpose, like I will install it in a transportation bus, to count the people leaving and entering ( That is actually my project and I finish all the things except the hardware),
1. is it possible to use it without a computer ( after setting up and installing all the thing) ?that means the program sketch that will accomplish the task of counting is no more in my laptop. it has to be on the pi and the pi is not connected physically or over the network with my laptop.. because the Pi is a single computer.
2. If that is possible, does it mean I have to load the code on the SD card and that will work fine? Thank you …..
Yes, you can do this. You would install your code on your Raspberry Pi and then typically have the script run on reboot.
thank you very mutch
Hi Adrian!
Really appreciate your hard work! Hope to learn from you here!
Thanks Both 🙂
Hello Adrian
I have a problem when executing the camera is on but I have a grey frame with no images ?
What type of camera are you using? Is it a USB camera or a Raspberry Pi camera module?
Hi Adrian
Thank you for your wonderful tutorials.
How do I load an image instead of the video stream?
Refer to this post.
im having an issue that i can’t seem to figure out. im getting this error message
Traceback (most recent call last):
…
ImportError: No module named ‘cv2’
any help is greatly apreciated
It sounds like you do not have the OpenCV library installed on your system. Make sure you follow one of my OpenCV install guides to install OpenCV on your Raspberry Pi.
One doubt: can I use this example with a Raspberry pi camera?
Yes. The blog post already shows you how to do that. See Lines 35 and 36.
Hi Adrian
I get this error although I’ve installed your imutils:
…
ImportError: No module named imutils.video
It sounds like either:
1. You did not install imutils
2. The ‘pip install imutils’ command exited with an error
3. You installed imutils into a separate Python version (perhaps a Python virtual environment) than the one you are using
Double-check the three above.
Hi Adrian
I’m sorry for my delay but i had many other problems.
I checked your advice, but i can’t find a solution: I’m sure I’ve installed imutils without any error and i’m not using a virtual environment. How can I check if I’ve installed imutils into a separate version of python than the one i’m using?
You can run “pip freeze” to verify if “imutils” is listed in your Python install.
Please help
$ python real_time_object_detection.py
…
AttributeError: ‘module’ object has no attribute ‘dnn’
You need to install OpenCV 3.3 or greater to access the “dnn” module. It sounds like you do not have OpenCV 3.3+ installed.
I don’t have any tutorials on fire detection but I would certainly look to cover it in a future tutorial. Before you can even deploy your model to detect fire you need to train it. You need to gather example images that contain fire. If you’re new to training your own models from scratch I would suggest starting with this tutorial.
Hello Adrian, thank you for the post and the code. I followed the tutorial but I keep getting the following ouput:
[INFO] loading model…
[INFO] starting process…
[INFO] starting video stream…
Unable to init server: Could not connect: Connection refused
(Frame:1113): Gtk-WARNING **: cannot open display:
Could you let me know what the issue might be?
Thanks in advanced!
It sounds like you are accessing your Pi over SSH. In that case you need to enable X11 forwarding:
$ ssh -X pi@ your_ip_addressHello Adrian, can we add text to speech for the labels detected by say, gtts and pygame?
I haven’t tried any text to speech libraries on the Pi, but yes, provided you can correctly detect the label you could pass the string of text into a text to speech library.
How can I create a caffe model to detect other objects?
I cover how to train your own custom object detectors inside Deep Learning for Computer Vision with Python — I would suggest starting there.
hello Adrian,
your tutorial was awesome 🙂
and how can i build own model ?
Thanks Ilham, I’m glad you liked the tutorial 🙂 Are you specifically asking how to train your own deep learning object detector? If so, I would suggest starting by reading reading this guide on the basics of object detection. From there you should read Deep Learning for Computer Vision with Python where you can learn how to train your own object detectors (including code).
Hello adrian!
Im having some issues while running the “–model MobileNetSSD_deploy.caffemodel”
the error that appears is “ImportError: No module named imutils.video”
I already install imutils version: 0.5.1 but the errors keeps showing..
Any idea about this?
THANKS!
It sounds like you haven’t properly installed the imutils library. Are you using Python virtual environments? If so, make sure you access them via the “workon” command, install imutils, and then run the script:
Hey Adrian!
Im working on a school project, to detect numbers of fruits (eg apple) within a image.
Will be using RPI 3B+ with a picamera. My planned steps are as follows:
1) Activate Picamera to capture a image, and save it into a folder.
2) Load the image file, and run it on the object detection module.
Hopefully, the object detection module will be able to identify the number of fruits within the image, track and count the numbers, and return the number to a simple database.
Will I be able to leverage on your above codes to identify different fruits such as apples and bananas?
Thanks!
Hey Daniel, you would actually need to train your own custom object detector to detect various types of fruits. The model included in this tutorial is not trained on fruits. I would suggest you read this tutorial on the fundamentals of deep learning object detection so you can form a plan moving forward with your school project. Best of luck with it!
Hello, I have a question: Can I use the SSD in Windows 10?
Provided you have OpenCV properly installed on your Windows machine, yes, it will work.
Hi Adrian
I have a question , How can I stream my raspberry camera share on web and control movement , for example people counting in street or in office ?
thanks in advance
I do not have any tutorials on streaming the output from the Raspberry Pi but I will be covering it in my upcoming Raspberry Pi + computer vision book.
Hi Adrian,
Would there be a way to make this program verbally state what object it has detected?
Thank you!
You would need to look into a text-to-speech API.
can tesseract support opencv 3.3.0
I assume you are referring to this tutorial? If so, yes, OpenCV 3.3.0 and Tesseract can work together.
Hi Adrian.
Thanks a lot for this very good tutorial!
Just a few question, I’m trying to detect car in an image using cv2.imread(“imagename.jpg”) but nothing is detected. I’ve trying with a lot a images (and videos), same results.
By using debug in LiClipse I’ve seen that the program is executed but not respond after calling the line net.setInput(blob).
Note that I’m running program on Windows.
Please have you an idea?
Thanks
Its a great work…It helped me
I’m so happy it helped you! 🙂
Hi Adrian! Nice Work!
i am working on my project for obstacle detection in a path and your project is somehow related to mine but the difference is that i just want to have the detection of the object or the blob without classifiying it. My project aims to just detect obstacles in a path for the blind using video stream. So, classifying it doesnt really needed. Do you have an idea?
Thanks a lot!!
Hey did you find any solution for your problem I am also looking for similar problem
Hi Adrian.
I have a program error.
from imutils import VideoStream
importError : cannot import name ‘VideoStream’
Can you help me?
Thanks
You need to install the “imutils” library:
$ pip install imutilshey adrian!!
im using ivport v2 board with rpi to use 2 csi cameras simultaneouly …but your code works only for 1 camera…
Follow this tutorial to use multiple cameras on your Raspberry Pi.
Hi Adrian, thanks a lot for this post-tutorial. It’s Great!!! I am a very new to python, raspberry and object detection but this was really easy to understand. I followed it step by step and managed to achieve Object detection through my Picamera, brilliant. The only issue I had was “make -j4” which I resolved with “make” at 1024 and not at 2048.
1. Can you please give me some hint on how to customise the models in order to detect objects of my interest?
2. If I got it right, thanks to this Python code we are loading models from Caffe. Can I create a model and load it? Would it be better to modify an existing one, if possible?
3. If creating and loading a new one is the way, can you tell me how to proceed?
4. If modifying an existing one is the way instead, can you tell me how to proceed?
5. Whichever is of the two above, how do you then class label the new model? MobileNet SSD?
6. If you already post the process can you point me to it with the link?
I understand that is a time, and CPU, consuming process. Is a windows 10 laptop sufficient AMD architecture and CPU? An IMAC can perform the task? If none of the two, how can I achieve that?
Looking forward to hear from you
Alberto
If you’re new to object detection I would recommend reading this gentle guide to object detection which addresses your questions. I also cover deep learning-based object detection in detail inside my book, Deep Learning for Computer Vision with Python.
Hello,
Thanks for sharing this interesting topic. I am actually doing a final year project and i have to take info from motion sensors and the signal will be processed to the microcontroller and will then tell the camera to start recording the scene. The camera will take a picture and then start recording a video for 5 to 6 minutes. Image processing will be done with the picture taken, differentiating between human and animal. then the owner will receive sms , saying the intruder is either a human or animal. Can you help me please?
P.S: raspberry pi 3 model b+ will be used
Thank you
That sounds like an awesome final year project. I actually cover how to build a face recognition system that sends SMS messages using a Raspberry Pi inside the PyImageSearch Gurus course. I would suggest using those lessons as a starting point for you project.
Thank you Adrian.
I have another question. Which pi is better for image processing? 3b or 3b+?
The 3B+ is slightly faster but both are suitable for OpenCV, just keep in mind that the Pi is very much underpowered compared to your laptop/desktop or a system with a GPU.
Hi Adrian,
If ever my camera detects intruder entering my garden, using object detection and then sends an sms to the owner to let him know about the nature of the intrusion, either an animal or human. Considering that i’m using raspberry pi. Can you advise me on that ?
Thank you
I’ll actually be covering that exact project in my upcoming Computer Vision + Raspberry Pi book, stay tuned!
Thank you, is the Pi NoIR camera good enough for human detection?
It really depends on your scene/environment, but yes, the Pi NoIR camera can be used for human detection.
To be used in a garden , using Haar cascade classifier feature, both day and night time.
Hi Adrian,
Do you have python codes for HOG + Linear SVM detector using raspberry pi ?
I cover HOG + Linear SVM object detection, including on the Pi, inside the PyImageSearch Gurus course.
Hey adrian i want to ignore all the class except car and motorcycle in the multiprocessing method source code. where is the place to put this code :
if CLASSES[idx] in IGNORE:
continue
i get this code from your previous post. but I am confused about placing it
I cover your exact question in this tutorial — please give it a read.
hey thank you for sharing i want to implement this code but i think you did not mentied the In put and out put of raspberry that you used in your program
ps; did you use pc camera ot picamera
I used my USB webcam attached to a Pi. You could use the Raspberry Pi camera module as well (see Lines 35 and 36, including the comments above them).
hi adrian can you help me, i’m trying to make people detection only, how can i restrict the other object and only focus to the people detection, i already try to delete the CLASSES but i got error, please help me, Thank you
I cover your exact question in this gentle guide to deep learning-based object detection.
Hey I tried the following example to initially run in PC it worked fine and I’m just impressed with the decent efficiency but can you suggest me how can I use other pretrained model in this and can you suggest me some to tryout?
Are you looking specifically for object detectors? Image classifiers? Or any model that will run on the Raspberry Pi?
Hey Adrian,
I was wondering if it would be possible to use this code but setup a “super computer” out of raspberry pi’s i.e. 4 or 5 pis linked together to process the data faster?
No, you are far better off investing your time or money into an Movidius NCS for the Pi or using something like a Jetson TX2 for embedding deep learning. Networking together a bunch of Pis won’t help performance.
sir
i wanted to make a waste segregation system for my undergrad project which segregate them in two types mainly
1. bio degradable
2. non biodegradable
can u plz suggest me any resource where i could learn and any training module
Congrats on working on your undergrad project, that’s very exciting.
As for a resource, make sure you read through Deep Learning for Computer Vision with Python where I show you how to train your own custom image classifiers, object detectors, and instance segmentation projects — all of them will help you with your project.
Hi Adrian i’m a student from Indonesia and I’m trying to finish my degree and i’m tryingto make final project about detecting people in a room, but my fps is so low it’s arround 0.30fps, is it possible because the camera i use or is it because the raspberry ?
If it’s because the camera what kind of usb camera you recommend so i can get a better fps….
Please Reply ?
Ps : i use pi camera
Thank you
The Raspberry Pi itself will be slow for deep learning-based object detection. You may want to try using a Movidius NCS or Google Coral to speed it up.
I’ll be covering my other best practices on how to perform Deep Learning on a Raspberry Pi in my upcoming book, stay tuned!
Great job ! i would like to make the same thing but to recognize the line form ( if it is continious or not ) . any help please ?
how to use same code for two or more usb web cam?
To use multiple cameras you can use this post.
Hi Adrian, thanks for the tutorial, is that the pre-trained model you put in the code file, only have the classes you stated in the code? Also how to train more different classes into the model?
I would suggest you refer to this tutorial.
Thank you so much Mr. Adrian for these great tutorials 🙂
The classes you mentioned in this tutorial, are just the classes can be detected by MobileNet SSD pre-trained model?
Correct, the classes mentioned in this tutorial come from the pre-trained MobileNet SSD.
hi adrian , please help
I already solve the videostram error however
(Frame:1417): Gtk-WARNING **: cannot open display:
Are you SSH’ing into your Pi?
If so, be sure to enable X11 forwarding:
$ ssh -X pi@your_ip_addressHi Adrian! All of your posts are really helpful, first of all i want to thank you for everything because you help me in my school project. It is my very first time learn about computer vision and you explained everything well and easy to understand. I want to ask you something, so in our school project we want to make animal detection using raspberry pi, and as you explained the training animal data are “bird, cat, cow, dog, horse, and sheep’.
1. Is it possible to add another animal for the training data and delete the others that i won’t use?
2. After detect some animal, for example cow, it will trigger our actuators like buzzer and led. Can you tell me where can i put my code for the actuators?
Once again thank you very much! You are awesome!
1. Yes, that’s called fine-tuning. See this tutorial to get your bearings followed by Deep Learning for Computer Vision with Python where I cover fine-tuning in detail.
2. See this tutorial where I use a buzzer triggered off of actions as well.
Thank you so much once again Adrian! hope everything great will always be with you because of your kindness. Cheers!
Greetings Adrian. Thank you for your marvelous explanation. I wonder, because using rasbpi is only giving a low FPS. Is it possible to send a raw video stream from rasbpi to our desktop, and do the deep learning thing in the desktop? Thanks.
Yes. See this tutorial.
Hi,
I’m trying your code on Raspberry Pi. Just for information, doing:
sudo pip3 install opencv-python==3.3.0.10
it’s possible to use your scripts with python3 without install opencv3 from source.
But the question is:
with you scripts my CPU% is going more than 100. I tried also with a time.sleep inside the frame loop. Do you have any suggestion? Also in your case you have this problem? Thanks
1. If you want to use pip to install OpenCV you should follow my tutorial.
2. Deep learning object detectors are very computationally expensive and the RPi is underpowered. Yes, your CPU will be utilized entirely.
hello sir how did you define the list of classes in sequence do you have summary of model..??
The list of classes is what the model was originally trained to detect.
So, I’ve only just started studying the field, and I’m trying to understand where the specific number of 20 objects (plus background, making 21) comes from, considering that COCO has 80 classes. I see variations of this particular example/tutorial on many sites, but it always seems to come to 20 classes, so I’m guessing that number is somehow significant.
So, would I be at least in the ballpark of being correct in guessing that it comes from the Mobilenet v1 architecture, and somewhere in the hidden layers exist only 21 nodes left (or alternatively, a number of nodes that is only just sufficient for encoding that number in a binary format), after some of them are reserved for representing the necessary box coordinates? So that if you wanted to get a bigger number of classifications from that 80 class set, you would choose a different network architecture (say, inception) and the trade-off would be larger memory consumption and/or lower performance?
The model used here was trained on a subset of the COCO dataset, not the entire dataset. You could train this model on the entire 80 class COCO dataset.
Ah, so it didn’t originally receive the other classes as inputs, even as negative examples. In other words, if I wanted to make sure that an SSD model properly distinguishes between, say, a bicycle and a motorcycle, they both absolutely need to be classes in the list AND classes in the training dataset. And I should only leave out classes that I know to be vastly different from the objects that I do want to detect, or objects that I don’t expect to ever encounter in the camera feed.
Hi Adrian, a real quick question. Can the video input is from youtube live stream?
No, you would need to download the YouTube video to disk first. There may be Python packages out there that convert a YouTube stream to individual NumPy arrays but I haven’t checked (you should do a search yourself and come back and let us know).
Hi Adrian,
An amazing tutorial, just had a couple of questions,
I have been unable to utilise the optimised OpenCV install as I end up with a CMake error when attempting to compile OpenCV 3.3.0. I finally ended up making non-optimised install of OpenCV 4, and received no complication. I assume the error arose from the version of OpenCV being used, and am curious as to whether the optimisation still applies to installs of OpenCV 4.
Additionally, I’m currently using this code as the basis for a robotics project that uses RPi to identify and collect bottles. I’m not overly concerned with FPS (though it would certainly be useful), and can’t help but notice a certain level of inaccuracy, with the program often classifying horizontally-laid bottles as sofas or chairs. I was expecting this, but was wondering if replacing the current MobileNet model with one oriented around bottles would help with accuracy. If so, do you have any recommendations about where I may be likely to find such a thing? Any tips as to implementing it in place of the current one?
Once again, a magnificent tutorial, and thanks anyway.
Will it be feasible on a low quality SD video (640x480p) which has 12 fps?
I’ve got offline SD videos with only 12fps.
Please reply !!! ?
Give it a try and see! I’m happy to provide this tutorial and code for free, but if you have the data already, the best way to learn is to apply the code. Learn by experimentation 🙂
Hi Adrian,
I want to use this model to detect only people, animals and vehicles. Right now, I have written conditional code to only detect those specific classes based on the idx values. But I understand that this will not have any performance improvements since the model is trained on all the classes. Do I have to retrain the model with the subset of classes I need in order to decrease the processing time? Could you please guide me to some resources on how I could go about doing this?
Thanks!
What you’re referring to is called “fine-tuning”. Fine-tuning, including code, is covered in detail inside Deep Learning for Computer Vision with Python.
Hello Adrian,
thank you so much for this wonderful tutorial
while going through this one and the one with SSD-Caffe model
i was finding it difficult to understand, how to interpret the output array
for YOLOv2 we had 85 parameters in the detection
whereas in SSD we had 7 parameters in the detection
is it that, in YOLOV2, each position corresponds to the classID ( so a value at position 75 corresponds to 70th CLASSID, (75-5 ) for the locations)
please correct me, if i am wrong about this.
and in SSD, the we check the value in the parameter and compare the index to get the class ID?
please throw some light on the same, if you have some free time
thanks,
Martin
hi sir,your tutorials are very useful….
how to code to speedup object detection process for fast moving objects …..with little latency in streaming for realtime
My book, Raspberry Pi for Computer Vision, covers that exact question.
Hi Adrian,
If I need to train(transfer learning) any other object using the same Caffe Model, Can you please suggest me a link/code.
I see this is faster with pi camera.
Thanks
Guru
Thank you for the example program. I have tried the program in Raspberry Pi 4, I got 2.7 fps and 58 fps.
Thanks for sharing!
Good day!
I have been enjoying playing with OpenCV, and using your tutorials has been a great boon. However, I have recently switched from using the Raspberry Pi Camera to a Logitech c270 webcam, and now the object detection (or any script) no longer works.
I receive VIDIOC_QBUF: Invalid argument (about 200 of them in a row).
I have tested the camera with guvcview and the camera itself is fine. The PiCamera still works when I un-comment the line in the python script.
No online resource has been able to help me resolve this issue. Any help or a point in the right direction is apprecated, thank you!
That doesn’t sound like an issue specific to this script, it sounds like an OpenCV-related issue. Rip out all the object detection code and just try to display the frames to your screen (no other video/image processing). That will help you debug.
I couldn’t get anything running at all, going right down to basic frame grabs with Python, just error after error. Frustrated I took the SD card out of my Pi4 and put it into a Pi3 B+ and it loaded immediately. The Open CV build is from the Pi4 w/ Buster tutorial but so far each additional tutorial I’ve followed has worked just fine on the Pi3.
Something to do with the USB I figure. I’ll dive back into it later, now that it’s working on Pi3 I’m having too much fun!
Hello.
I ran the program, but Raspberry Pi shut down itself. In about 20 seconds. I installed a cooler. It happened again.
Why would that be?
Raspberry Pi 4
OpenCV 4.1.0
Check your USB power/plug power. It may be that your RPi isn’t receiving enough power draw. It could also be overheating.
Hi Adrian!
Thank you for this tutorial! I’m trying to detect traffic lights with a raspberry pi 4 and a pi camera for a project graduation. I already found the bvlc googlenet pre trained model with the “traffic light” label and tried to modify the code but didn’t success to make it work. What do I have to change to make it work with an other model ?
Hey Mounir — it’s funny that you mention traffic light detection. That topic is covered inside Raspberry Pi for Computer Vision.
Dear Adrian,
is it possible to recognize and classify traffic lights with the Raspberry Pi in real time?
Do you have any experience with that?
Sincerely
Siggi
It is. That topic is covered inside Raspberry Pi for Computer Vision.
ciao Adrian,
nice to read your beautiful guides (my name is Adrian too lol),
i already reproduced your code and work properly but I would like to use SquizeNet instead of MobileNetSSD… already I tried to modify a little the pthon code but it doesn’t work (maybe because I don’t understand in deep the architecture of the nets).
could you help me ?
the idea behind is to test more than one net to find the “lightest” one.
thanks in advance and happy christmass.
I would suggest reading this guide on object detection first. It discusses the differences between image classification and object detection which I think you may be struggling with. In particular, be sure to read the parts of the post on “backbone” architectures.
Dear Adrian,
Thank you for this info. Do you have tutorials how to create our own dataset. we have a project incomputer vision that needs to classify different vehicles including tricycle.
Yes, see this tutorial.
Dear Adrian
Thank you for sharing your knowledge. As you mentioned above deep learning models are very computationally intensive. I was wondering is it possible to speed up the inference process using pure c code?
Besides that what was the highest performance rate you acquired?(beyond .9 frame per second)
TensorFlow libraries are compiled and optimized. Using “pure C/C++” isn’t going to help improve inference too much.
hello,
I’m working on a self-contained car project. I wanted to know how to use your trafficsignet recognition model instead of the neural network model.
Thank you for your help. I
i’m French, sorry for my English.
Hey Adrian,
Thank you for sharing your knowledge! I need your opinion on something, I am working on a project to have continuous object detection to train a robot hand to pick up an object. We are using openCV. We have a code already but we need to make it better. Any options we should take?
can I use a rpi zero w ?
No, the Pi Zero W is going to be too slow for a deep learning-based object detector.