In this tutorial, you will learn how to apply Holistically-Nested Edge Detection (HED) with OpenCV and Deep Learning. We’ll apply Holistically-Nested Edge Detection to both images and video streams, followed by comparing the results to OpenCV’s standard Canny edge detector.
Edge detection enables us to find the boundaries of objects in images and was one of the first applied use cases of image processing and computer vision.
When it comes to edge detection with OpenCV you’ll most likely utilize the Canny edge detector; however, there are a few problems with the Canny edge detector, namely:
- Setting the lower and upper values to the hysteresis thresholding is a manual process which requires experimentation and visual validation.
- Hysteresis thresholding values that work well for one image may not work well for another (this is nearly always true for images captured in varying lighting conditions).
- The Canny edge detector often requires a number of preprocessing steps (i.e. conversion to grayscale, blurring/smoothing, etc.) in order to obtain a good edge map.
Holistically-Nested Edge Detection (HED) attempts to address the limitations of the Canny edge detector through an end-to-end deep neural network.
This network accepts an RGB image as an input and then produces an edge map as an output. Furthermore, the edge map produced by HED does a better job preserving object boundaries in the image.
To learn more about Holistically-Nested Edge Detection with OpenCV, just keep reading!
Holistically-Nested Edge Detection with OpenCV and Deep Learning
In this tutorial we will learn about Holistically-Nested Edge Detection (HED) using OpenCV and Deep Learning.
We’ll start by discussing the Holistically-Nested Edge Detection algorithm.
From there we’ll review our project structure and then utilize HED for edge detection in both images and video.
Let’s go ahead and get started!
What is Holistically-Nested Edge Detection?
The algorithm we’ll be using here today is from Xie and Tu’s 2015 paper, Holistically-Nested Edge Detection, or simply “HED” for short.
The work of Xie and Tu describes a deep neural network capable of automatically learning rich hierarchical edge maps that are capable of determining the edge/object boundary of objects in images.
This edge detection network is capable of obtaining state-of-the-art results on the Berkely BSDS500 and NYU Depth datasets.
A full review of the network architecture and algorithm outside the scope of this post, so please refer to the official publication for more details.
Project structure
Go ahead and grab today’s “Downloads” and unzip the files.
From there, you can inspect the project directory with the following command:
$ tree --dirsfirst . ├── hed_model │ ├── deploy.prototxt │ └── hed_pretrained_bsds.caffemodel ├── images │ ├── cat.jpg │ ├── guitar.jpg │ └── janie.jpg ├── detect_edges_image.py └── detect_edges_video.py 2 directories, 7 files
Our HED Caffe model is included in the hed_model/ directory.
I’ve provided a number of sample images/ including one of myself, my dog, and a sample cat image I found on the internet.
Today we’re going to review the detect_edges_image.py and detect_edges_video.py scripts. Both scripts share the same edge detection process, so we’ll be spending most of our time on the HED image script.
Holistically-Nested Edge Detection in Images
The Python and OpenCV Holistically-Nested Edge Detection example we are reviewing today is very similar to the HED example in OpenCV’s official repo.
My primary contribution here is to:
- Provide some additional documentation (when appropriate)
- And most importantly, show you how to use Holistically-Nested Edge Detection in your own projects.
Let’s go ahead and get started — open up the detect_edge_image.py file and insert the following code:
# import the necessary packages
import argparse
import cv2
import os
# construct the argument parser and parse the arguments
ap = argparse.ArgumentParser()
ap.add_argument("-d", "--edge-detector", type=str, required=True,
help="path to OpenCV's deep learning edge detector")
ap.add_argument("-i", "--image", type=str, required=True,
help="path to input image")
args = vars(ap.parse_args())
Our imports are handled on Lines 2-4. We’ll be using argparse to parse command line arguments. OpenCV functions and methods are accessed through the cv2 import. Our os import will allow us to build file paths regardless of operating system.
This script requires two command line arguments:
--edge-detector: The path to OpenCV’s deep learning edge detector. The path contains two Caffe files that will be used to initialize our model later.--image: The path to the input image for testing. Like I said previously — I’ve provided a few images in the “Downloads”, but you should try the script on your own images as well.
Let’s define our CropLayer class:
class CropLayer(object): def __init__(self, params, blobs): # initialize our starting and ending (x, y)-coordinates of # the crop self.startX = 0 self.startY = 0 self.endX = 0 self.endY = 0
In order to utilize the Holistically-Nested Edge Detection model with OpenCV, we need to define a custom layer cropping class — we appropriately name this class CropLayer .
In the constructor of this class, we store the starting and ending (x, y)-coordinates of where the crop will start and end, respectively (Lines 15-21).
The next step when applying HED with OpenCV is to define the getMemoryShapes function, the method responsible for computing the volume size of the inputs :
def getMemoryShapes(self, inputs): # the crop layer will receive two inputs -- we need to crop # the first input blob to match the shape of the second one, # keeping the batch size and number of channels (inputShape, targetShape) = (inputs[0], inputs[1]) (batchSize, numChannels) = (inputShape[0], inputShape[1]) (H, W) = (targetShape[2], targetShape[3]) # compute the starting and ending crop coordinates self.startX = int((inputShape[3] - targetShape[3]) / 2) self.startY = int((inputShape[2] - targetShape[2]) / 2) self.endX = self.startX + W self.endY = self.startY + H # return the shape of the volume (we'll perform the actual # crop during the forward pass return [[batchSize, numChannels, H, W]]
Line 27 derives the shape of the input volume as well as the target shape.
Line 28 extracts the batch size and number of channels from the inputs as well.
Finally, Line 29 extracts the height and width of the target shape, respectively.
Given these variables, we can compute the starting and ending crop (x, y)-coordinates on Lines 32-35.
We then return the shape of the volume to the calling function on Line 39.
The final method we need to define is the forward function. This function is responsible for performing the crop during the forward pass (i.e., inference/edge prediction) of the network:
def forward(self, inputs): # use the derived (x, y)-coordinates to perform the crop return [inputs[0][:, :, self.startY:self.endY, self.startX:self.endX]]
Lines 43 and 44 take advantage of Python and NumPy’s convenient list/array slicing syntax.
Given our CropLayer class we can now load our HED model from disk and register CropLayer with the net:
# load our serialized edge detector from disk
print("[INFO] loading edge detector...")
protoPath = os.path.sep.join([args["edge_detector"],
"deploy.prototxt"])
modelPath = os.path.sep.join([args["edge_detector"],
"hed_pretrained_bsds.caffemodel"])
net = cv2.dnn.readNetFromCaffe(protoPath, modelPath)
# register our new layer with the model
cv2.dnn_registerLayer("Crop", CropLayer)
Our prototxt path and model path are built up using the --edge-detector command line argument available via args["edge_detector"] (Lines 48-51).
From there, both the protoPath and modelPath are used to load and initialize our Caffe model on Line 52.
Let’s go ahead and load our input image :
# load the input image and grab its dimensions
image = cv2.imread(args["image"])
(H, W) = image.shape[:2]
# convert the image to grayscale, blur it, and perform Canny
# edge detection
print("[INFO] performing Canny edge detection...")
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
blurred = cv2.GaussianBlur(gray, (5, 5), 0)
canny = cv2.Canny(blurred, 30, 150)
Our original image is loaded and spatial dimensions (width and height) are extracted on Lines 58 and 59.
We also compute the Canny edge map (Lines 64-66) so we can compare our edge detection results to HED.
Finally, we’re ready to apply HED:
# construct a blob out of the input image for the Holistically-Nested
# Edge Detector
blob = cv2.dnn.blobFromImage(image, scalefactor=1.0, size=(W, H),
mean=(104.00698793, 116.66876762, 122.67891434),
swapRB=False, crop=False)
# set the blob as the input to the network and perform a forward pass
# to compute the edges
print("[INFO] performing holistically-nested edge detection...")
net.setInput(blob)
hed = net.forward()
hed = cv2.resize(hed[0, 0], (W, H))
hed = (255 * hed).astype("uint8")
# show the output edge detection results for Canny and
# Holistically-Nested Edge Detection
cv2.imshow("Input", image)
cv2.imshow("Canny", canny)
cv2.imshow("HED", hed)
cv2.waitKey(0)
To apply Holistically-Nested Edge Detection (HED) with OpenCV and deep learning, we:
- Construct a
blobfrom our image (Lines 70-72). - Pass the blob through the HED net, obtaining the
hedoutput (Lines 77 and 78). - Resize the output to our original image dimensions (Line 79).
- Scale our image pixels back to the range [0, 255] and ensure the type is
"uint8"(Line 80).
Finally, we we’ll display:
- The original input image
- The Canny edge detection image
- Our Holistically-Nested Edge detection results
Image and HED Results
To apply Holistically-Nested Edge Detection to your own images with OpenCV, make sure you use the “Downloads” section of this tutorial to grab the source code, trained HED model, and example image files. From there, open up a terminal and execute the following command:
$ python detect_edges_image.py --edge-detector hed_model --image images/cat.jpg [INFO] loading edge detector... [INFO] performing Canny edge detection... [INFO] performing holistically-nested edge detection...
On the left we have our input image.
In the center we have the Canny edge detector.
And on the right is our final output after applying Holistically-Nested Edge Detection.
Notice how the Canny edge detector is not able to preserve the object boundary of the cat, mountains, or the rock the cat is sitting on.
HED, on the other hand, is able to preserve all of those object boundaries.
Let’s try another image:
$ python detect_edges_image.py --edge-detector hed_model --image images/guitar.jpg [INFO] loading edge detector... [INFO] performing Canny edge detection... [INFO] performing holistically-nested edge detection...
In Figure 3 above we can see an example image of myself playing guitar. With the Canny edge detector there is a lot of “noise” caused by the texture and pattern of the carpet — HED, on the other contrary, has no such noise.
Furthermore, HED does a better job of capturing the object boundaries of my shirt, my jeans (including the hole in my jeans), and my guitar.
Let’s do one final example:
$ python detect_edges_image.py --edge-detector hed_model --image images/janie.jpg [INFO] loading edge detector... [INFO] performing Canny edge detection... [INFO] performing holistically-nested edge detection...
There are two objects in this image: (1) Janie, the dog, and (2) the chair behind her.
The Canny edge detector (center) does a reasonable job highlighting the outline of the chair but isn’t able to properly capture the object boundary of the dog, primarily due to the light/dark and dark/light transitions in her coat.
HED (right) is able to capture the entire outline of Janie more easily.
Holistically-Nested Edge Detection in Video
We’ve applied Holistically-Nested Edge Detection to images with OpenCV — is it possible to do the same for videos?
Let’s find out.
Open up the detect_edges_video.py file and insert the following code:
# import the necessary packages
from imutils.video import VideoStream
import argparse
import imutils
import time
import cv2
import os
# construct the argument parser and parse the arguments
ap = argparse.ArgumentParser()
ap.add_argument("-d", "--edge-detector", type=str, required=True,
help="path to OpenCV's deep learning edge detector")
ap.add_argument("-i", "--input", type=str,
help="path to optional input video (webcam will be used otherwise)")
args = vars(ap.parse_args())
Our vide script requires three additional imports:
VideoStream: Reads frames from an input source such as a webcam, video file, or another source.imutils: My package of convenience functions that I’ve made available on GitHub and PyPi. We’re using myresizefunction.time: This module allows us to place a sleep command to allow our video stream to establish and “warm up”.
The two command line arguments on Lines 10-15 are quite similar:
--edge-detector: The path to OpenCV’s HED edge detector.--input: An optional path to an input video file. If a path isn’t provided then the webcam will be used.
Our CropLayer class is identical to the one we defined previously:
class CropLayer(object): def __init__(self, params, blobs): # initialize our starting and ending (x, y)-coordinates of # the crop self.startX = 0 self.startY = 0 self.endX = 0 self.endY = 0 def getMemoryShapes(self, inputs): # the crop layer will receive two inputs -- we need to crop # the first input blob to match the shape of the second one, # keeping the batch size and number of channels (inputShape, targetShape) = (inputs[0], inputs[1]) (batchSize, numChannels) = (inputShape[0], inputShape[1]) (H, W) = (targetShape[2], targetShape[3]) # compute the starting and ending crop coordinates self.startX = int((inputShape[3] - targetShape[3]) / 2) self.startY = int((inputShape[2] - targetShape[2]) / 2) self.endX = self.startX + W self.endY = self.startY + H # return the shape of the volume (we'll perform the actual # crop during the forward pass return [[batchSize, numChannels, H, W]] def forward(self, inputs): # use the derived (x, y)-coordinates to perform the crop return [inputs[0][:, :, self.startY:self.endY, self.startX:self.endX]]
After defining our identical CropLayer class, we’ll go ahead and initialize our video stream and HED model:
# initialize a boolean used to indicate if either a webcam or input
# video is being used
webcam = not args.get("input", False)
# if a video path was not supplied, grab a reference to the webcam
if webcam:
print("[INFO] starting video stream...")
vs = VideoStream(src=0).start()
time.sleep(2.0)
# otherwise, grab a reference to the video file
else:
print("[INFO] opening video file...")
vs = cv2.VideoCapture(args["input"])
# load our serialized edge detector from disk
print("[INFO] loading edge detector...")
protoPath = os.path.sep.join([args["edge_detector"],
"deploy.prototxt"])
modelPath = os.path.sep.join([args["edge_detector"],
"hed_pretrained_bsds.caffemodel"])
net = cv2.dnn.readNetFromCaffe(protoPath, modelPath)
# register our new layer with the model
cv2.dnn_registerLayer("Crop", CropLayer)
Whether we elect to use our webcam or a video file, the script will dynamically work for either (Lines 51-62).
Our HED model is loaded and the CropLayer is registered on Lines 65-73.
Let’s acquire frames in a loop and apply edge detection!
# loop over frames from the video stream while True: # grab the next frame and handle if we are reading from either # VideoCapture or VideoStream frame = vs.read() frame = frame if webcam else frame[1] # if we are viewing a video and we did not grab a frame then we # have reached the end of the video if not webcam and frame is None: break # resize the frame and grab its dimensions frame = imutils.resize(frame, width=500) (H, W) = frame.shape[:2]
We begin looping over frames on Lines 76-80. If we reach the end of a video file (which happens when a frame is None ), we’ll break from the loop (Lines 84 and 85).
Lines 88 and 89 resize our frame so that it has a width of 500 pixels. We then grab the dimensions of the frame after resizing.
Now let’s process the frame exactly as in our previous script:
# convert the frame to grayscale, blur it, and perform Canny
# edge detection
gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
blurred = cv2.GaussianBlur(gray, (5, 5), 0)
canny = cv2.Canny(blurred, 30, 150)
# construct a blob out of the input frame for the Holistically-Nested
# Edge Detector, set the blob, and perform a forward pass to
# compute the edges
blob = cv2.dnn.blobFromImage(frame, scalefactor=1.0, size=(W, H),
mean=(104.00698793, 116.66876762, 122.67891434),
swapRB=False, crop=False)
net.setInput(blob)
hed = net.forward()
hed = cv2.resize(hed[0, 0], (W, H))
hed = (255 * hed).astype("uint8")
Canny edge detection (Lines 93-95) and HED edge detection (Lines 100-106) are computed over the input frame.
From there, we’ll display the edge detection results:
# show the output edge detection results for Canny and
# Holistically-Nested Edge Detection
cv2.imshow("Frame", frame)
cv2.imshow("Canny", canny)
cv2.imshow("HED", hed)
key = cv2.waitKey(1) & 0xFF
# if the `q` key was pressed, break from the loop
if key == ord("q"):
break
# if we are using a webcam, stop the camera video stream
if webcam:
vs.stop()
# otherwise, release the video file pointer
else:
vs.release()
# close any open windows
cv2.destroyAllWindows()
Our three output frames are displayed on Lines 110-112: (1) the original, resized frame, (2) the Canny edge detection result, and (3) the HED result.
Keypresses are captured via Line 113. If "q" is pressed, we’ll break from the loop and cleanup (Lines 116-128).
Video and HED Results
So, how does Holistically-Nested Edge Detection perform in real-time with OpenCV?
Let’s find out.
Be sure to use the “Downloads” section of this blog post to download the source code and HED model.
From there, open up a terminal and execute the following command:
$ python detect_edges_video.py --edge-detector hed_model [INFO] starting video stream... [INFO] loading edge detector...
In the short GIF demo above you can see a demonstration of the HED model in action.
Notice in particular how the boundary of the lamp in the background is completely lost when using the Canny edge detector; however, when using HED the boundary is preserved.
In terms of performance, I was using my 3Ghz Intel Xeon W when gathering the demo above. We are obtaining close to real-time performance on the CPU using the HED model.
To obtain true real-time performance you would need to utilize a GPU; however, keep in mind that GPU support for OpenCV’s “dnn” module is particularly limited (specifically NVIDIA GPUs are not currently supported).
In the meantime, you may want to consider using the Caffe + Python bindings if you need real-time performance.
What's next? We recommend PyImageSearch University.
84 total classes • 114+ hours of on-demand code walkthrough videos • Last updated: February 2024
★★★★★ 4.84 (128 Ratings) • 16,000+ Students Enrolled
I strongly believe that if you had the right teacher you could master computer vision and deep learning.
Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?
That’s not the case.
All you need to master computer vision and deep learning is for someone to explain things to you in simple, intuitive terms. And that’s exactly what I do. My mission is to change education and how complex Artificial Intelligence topics are taught.
If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to successfully and confidently apply computer vision to your work, research, and projects. Join me in computer vision mastery.
Inside PyImageSearch University you'll find:
- ✓ 86 courses on essential computer vision, deep learning, and OpenCV topics
- ✓ 86 Certificates of Completion
- ✓ 115+ hours of on-demand video
- ✓ Brand new courses released regularly, ensuring you can keep up with state-of-the-art techniques
- ✓ Pre-configured Jupyter Notebooks in Google Colab
- ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
- ✓ Access to centralized code repos for all 540+ tutorials on PyImageSearch
- ✓ Easy one-click downloads for code, datasets, pre-trained models, etc.
- ✓ Access on mobile, laptop, desktop, etc.
Summary
In this tutorial, you learned how to perform Holistically-Nested Edge Detection (HED) using OpenCV and Deep Learning.
Unlike the Canny edge detector, which requires preprocessing steps, manual tuning of parameters, and often does not perform well on images captured using varying lighting conditions, Holistically-Nested Edge Detection seeks to create an end-to-end deep learning edge detector.
As our results show, the output edge maps produced by HED do a better job of preserving object boundaries than the simple Canny edge detector. Holistically-Nested Edge Detection can potentially replace Canny edge detection in applications where the environment and lighting conditions are potentially unknown or simply not controllable.
The downside is that HED is significantly more computationally expensive than Canny. The Canny edge detector can run in super real-time on a CPU; however, real-time performance with HED would require a GPU.
I hope you enjoyed today’s post!
To download the source code to this guide, and be notified when future tutorials are published here on PyImageSearch, just enter your email address in the form below!
Download the Source Code and FREE 17-page Resource Guide
Enter your email address below to get a .zip of the code and a FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning. Inside you'll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL!
Hot Damn!
Cool stuff once again Adrian. I need to use something like this in the future to measure the size difference between one edge compared to another. Luckily I can use an image rather than a video.
Adrian, great demo as usual. I have few questions –
1. What are the usecases according to you?
2. How would you do this using custom CNN i.e training our own CNN?
1. Building a document scanner is a great example of how edge detection can be applied.
2. I’m not sure what you’re asking here. Are you asking how to train your own custom HED network? If so, refer to the paper publication.
Many thanks for this new tutorial Adrian and the new beard looks good on you.
Is it possible to get an array of objects that have closed / curved line structure to extract or measure the size in real-time or offline?
You would want to apply contour detection.
An obvious difference I see between Canny and HED is that Canny (by design) has sharper edges, where HED appears to yield burry boundaries. How would you recommend cleaning the HED results up? Run Canny as a second step?
There are a number of steps you could do. Thresholding would be a super simple method to completely binarize the image. I would suggest starting there.
Good day, Dr. Adrian. Thanks for this cool tutorial!
Just have a couple of quick questions:
1. How do I get the bounding box / region of interest (roi) for each object whose edges have been completely detected?
— My idea is to use the startX / endX and startY / endY coordinates; but
— Do I need to get the lowest startX / startY and the highest endX / endY to do this?
2. Would it be likely to work if I pass each roi (will save each roi in an individual image file) to an object detection model like SSD trained on Coco?
Please advise. Thanks in advance! 🙂
1. You would apply contour detection. See this tutorial for an example.
2. You wouldn’t pass the ROI through an object detector. You would just apply the object detector to the entire image and skip edge detection.
Hi Adrian,
Thanks for this tutorials,I have a GTX 1080 TI? Is Opencv’s dnn support of this GPU? if so, for active this capability ,what do i do ?
If currently not supported of Nvidia GPU’s , is that possible to use the capability of OpenCL this GPU?
At the present, OpenCV’s “dnn” module does not support NVIDIA GPUs. I know they are working on it, specifically during this year’s Google Summer of Code, but you cannot use your GPU for the “dnn” module yet.
Nice tutorial allready know well hed maybe you can write a tuto for scanning using hed ( just like the tuto for scan receipt using canny)
🙂
Thanks Rui!
Since the hed does not provide binary output, it would be more correct to compare it wit Sobel operator. More over hed causes headache when you try to apply it on larger image, where there are e.g. a lot of cats and pure resizing of image would lost too much of information. I would be really interested in how hed can be applied in such case.
Great Post! Seems to be a little CPU heavy. Do you have any suggestions on a good GPU implementation? I’m currently using one I found that’s implemented in Tensorflow, https://github.com/moabitcoin/holy-edge. I didn’t know if you had a preference?
The HED model is a Caffe model so you can use “pycaffe” with GPU access enabled to run HED on your GPU.
As usual, Very interesting article Adrian.
Thanks Israfil!
Thank you so much for keeping us updated on new and better CV technologies. A lot of your work has been helpful for me to get into computer vision.
Also, can you recommend forums and blogs to follow like yours that would help me know more about the latest trends in computer vision?
The PyImageSearch Gurus course has dedicated forums. I spend more time in there than in the blog post comments. I would definitely take a look if you’re interested in more targeted advice for your projects.
The model that you have given was run by me on a Macbook Air and Pro. It turned out to be very heavy on processing. Is it expected to behave like this? How can we train the model to make it lighter on processing? My use-case is an edge detector which can crop an identity card from an image. I have already tried purely machine vision based detection (Gray > Blur > Canny > Hough Lines > Find Contours) without any model with poor results across different use-cases. What approach would you recommend for this?
Yes, it is expected to be very computationally heavy. Make sure you are reading the entire blog post as I discuss the benefits and tradeoffs of HED vs. traditional image processing edge detection. The final section of the real-time edge detection and “Summary” section should be required reading to address your question.
Adrian,
I have a failure on
cv2.dnn_registerLayer(“Crop”, CropLayer)
module ‘cv2’ has no attribute ‘dnn_registerLayer’
Python 3.7
It’s not Python that’s causing the issue, it’s OpenCV. What version of OpenCV are you using?
Well, I used your blog to find that out.
It now prints the version just before the offending line.
OpenCV version 3.4.1
Thanks for the rapid response, I knew I would have to wait until you got up as I am on UK time!
Upgrade to OpenCV 3.4.4 or OpenCV 4+. That should resolve the error (v3.4.1 is pretty “old” in terms of deep learning model support).
Thanks Adrian, you are brilliant.
What is the exact string for that as neither conda nor pip can find OpenCV4?
OK I used
pip install opencv-python==
to give a list of versions available
(from versions: 3.4.2.16, 3.4.2.17, 3.4.3.18, 3.4.4.19, 3.4.5.20, 4.0.0.21)
Then copied the version I wanted after the ==
Works fine with 4.0.0.21.
You should use “opencv-python-contrib” for the reasons I suggest in this post. Otherwise, you should be all set.
I also have a failure,
opencv-python 3.3.0.10
Just saw your comment comment. Your OpenCV version is too old. You need OpenCV 3.4 or OpenCV 4+.
issue: Inferences images have these border around it, basically at left and top side.
Hi there,
Thank you for your helpful blog,
However, I can not run the code on the spyder.
I was wondering If you could give me some advice.
Bests,
Hossein.
I suggest you execute the code via the command line instead of an IDE. If you are using an IDE you likely need to set the command line arguments.
Hi Adrian,
thanks for this great post.
As I tested, hed’s output is clearer than canny, specially when the background is complex, like with existing grass, trees,….. I am curious to know how this can be used to implement better object detection (not recognition) algorithms, or can it help to have better background subtraction,….?
Do you have any efficient conventional object detection (ignoring recognition) method using this?
You wouldn’t use HED naturally for object detection. For object detection you would use Faster R-CNN, YOLO, SSD, RetinaNet, etc. For instance segmentation you would use Mask R-CNN. All of these are covered inside Deep Learning for Computer Vision with Python.
Hi adrian,
thanks for great post, I have failure
;Segmentation fault (core dumped)’
Do you know how to solve it?
Can you debug by using “pdb” or “print” to determine exactly which line of code is causing the segfault? Unfortunately without knowing more information I cannot provide any suggestions.
Thank you Adrian…interesting as always.
Can we make this detector detects only vertical or horizontal edges like what Sobel does?
Thank Adrian for this blogp post!
I have some aerial image (video in a second step) of a small river (more like a “stern” in fact) from a dron. In the image I can see booth coast and the water. I need to get the coastline, can I use this method to get a better segmentation?
The coastline will be used to perform some autonomous navigation of a boat.
Hey Leonardo — I would suggest giving it a try and seeing for yourself. Use the source code and run it on the input image/video. What do your results look like?
MasyaAllah,,,your article always great. It’s really helping me for learning deep learning. May God Always Bless You.
Hey, I want to know if there’s any image size limit to this? It seems to be working fine with the default images but as soon as I use a larger image, like 1080, 1920, 3, it seems to throw a ‘Segmentation fault (core dumped)’ error.
Can you tell me why that would happen?
Most likely the system does not have enough memory to process the large image and hold the deep learning model. Check your memory usage during execution and see if it spikes.
Hello Adrian,
Thanks for perfect work share. But I can’t find dnn_registerLayer function alternative for android opencv 3.4.3. Do u have any information about it ?
Thanks.
You need to use OpenCV 3.4.4 or OpenCV 4.
Hi Mozart, I’m also trying to implement the same with Android, Any success implementation from your side?
Thanks.
Hi Adrian,
Thank you for your tutorials! After walking through one I always learn something useful that I can apply in a number of projects.
I do have one question on this post. What is the CropLayer Class doing ? I can print out the inputShape and targetShapes and see its calculating starting and ending points plus returning (N,C,H,W) but can’t quite wrap my mind around what is being cropped and how its impacting the final image ?
Hi Adrian, very appreciate your post.
As mentioned above, HED method can help detect edge in an image. I am wondering whether it is possible to use HED for blur dectection. For example, when I got the hed after this code:
hed = net.forward()
i wanna use it to calculate the std of it as the bluriness score of the image.
Have you tried following my blur detection post here? You could try using the HED as a replacement for the Laplacian of the image. That might require some tuning though.
Hi Adrian how did you know how/what to implement for the custom CropLayer class? If I were to start from this point https://github.com/s9xie/hed how would I able to get to the OpenCV implementation?
Hi Adrian,
Nice work here. But can you tell me how I could use this when I want to detect an object? I am making a project wherein I have to detect a specific leaf out of many different kinds of leaves in the background. Please help. Thank you
That sounds more like an object detection or instance segmentation problem rather than an edge detection one. Is there anything “different” about that one leaf that would make it different than the rest of the background?
Hi Adrian,
If I want to blur out a face in a photo with two faces in close proximity facing each other, could I used a combo of edge detection and skin color detection to designate the area to blur out? Ie. how can I get the x,y coordinates of boundaries of the detected edges? It’s a close up photo so I’m not sure if Mask-RCNN will work.
I would still give Mask R-CNN a try. The results will be more aesthetically pleasing. The other option is to fit a minimum enclosing circle to the detected face bounding box and then only blur the circular region.
Hi Adrian!
I’m starting research on these topics and manage to replicate your results. But there’s another project and I can’t replicate their results using your code and an input figure (composed by a triangle, a circle and a rectangle) that is in the other project:
https://github.com/ashislaha/Shape-Detection-in-AR/blob/master/README.md
I obtain some artifacts on the black region… Should I do a threshold to obtain the black and white effect? Thanks!
Adrian, the algorithm has magic numbers 104.00698793, 116.66876762, 122.67891434 — what are they? you didn’t mention them and I looked at the original paper without seeing them mentioned (perhaps I need more caffeine and re-read?).
Is this the mean RGB values of the image? Should it be calculated on each frame to improve accuracy?
They are the mean subtraction values used to normalize the data. You typically compute the RGB mean over the training set and subtract it prior to training, ultimately leading to higher accuracy. Mean subtraction, using the same values, must be performed on testing images as well. If you’d like to learn more about mean resizing, including training your own deep learning models, I would definitely suggest taking a look at my book, Deep Learning for Computer Vision with Python.
Hi Adrian,
Thank you for your work. I want to detect building’s skyline in the photo, could you give me some advice?
Thanks a lot !
How has HED worked for that task? Have you given it a try?
Suppose that I have two lanes to detect. It seems that HED would do it successfully. But I need to get the coordinates of these lines. How can I do it? You mentioned about contour detection which you demonstrated before. However it was about finding corners. In my situation I will have two main lines. Nothing more. The only need is to find out their x (or) y coordinates. Beginning and ending coordinates will be ok as well.
You mentioned performing lane detection so I assume you’re interested in self-driving cars. In that case semantic segmentation would likely be a better approach for you.
Sir, i have problem
how can i get hed_model?
i mean how can i get :
1. deploy.prototxt
2. hed_pretrained_bsds.caffemodel
i got it, at download section. Thankyou
Hi, how many fps were you able to get please?
Thanks for the code! This is incredibly easy and straightforward to run compared to other HED implementations.
Sorry for the dumb question, how can I use this to batch process all pictures inside a folder and/or frames from a video and save the results? Currently I can only see the results but not save them, but even if I could, it would take a lot of time to do it one by one.
Thanks for your kind attention!
You can use the “list_images” function in the imutils library to loop over all images in a file. The “cv2.VideoCapture” function can be used to access your webcam, then apply the edge detector to each frame of the video. I would also suggest you read Practical Python and OpenCV so you can better learn how to use the functions.
Hi Adrian, thanks for the cool tutorial. I tried combining it with https://pyimagesearch.com/2016/02/22/writing-to-video-with-opencv and although it works fine to output the original frame, if I try writing the canny or head matrix instead, the video ends up 0 seconds long. Thoughts? given that the fourcc and codec option actually worked but not for the processed steam ….
My use case is to process an input video containing human motion for which the background or the actual human is not so interesting, only the motion.
That’s most likely an issue with your video codecs and your install of OpenCV. That’s unfortunately a very hard problem to debug and one that is heavily dependent on how you installed OpenCV. I can’t really support that but I would recommend taking a look at the VM inside my book, Practical Python and OpenCV. That VM will run HED and allow you to write the frames back to a video file at the same time.
Thank you very much for your work and source code . But, in the code , It need the path of “OpenCV’s deep learning edge detector”, I don’t know what that is and where to download it. Thanks again
You can use the “Downloads” section of the post to download the source code and deep learning edge detector.
Hi!
Thanks for a good tutorial.
However, when I try using my own png-image, it starts and after 10-20 seconds it says “KILLED”. Any ideas what goes wrong?
It sounds like your machine is running out of RAM. Check your RAM usage when the script is running to verify.
For some reflective objects, or in the case of exposure, are there any algorithms to remove these bright spots? I now encounter a problem, through the way of word bags to carry out image search, but the image similarity is very high, for this kind of object with high similarity, is there any good way to solve it?
Could HED be used to for barcode detection?
Potentially but may not just use a dedicated barcode detection library like ZBar?
Hi there I am trying to use the Holistic edge detection for get the the contours of objects in an image. But after applying the net. I have an image with tick white borders. How do i get the edges needed for the findcontours. When i apply the Canny i get double borders and the edges it find afterwards are all over the place. I did apply a treshold after the net and before the canny. Does someone have a suggestion? i would like to be able to crop the image to the object in the picture.
Dear Adrian,
Thank you for this great content. I’m new to python and I don’t know to upgrade opencv to newer version. I’m trying to updated it using anaconda prompt. It still 3.4.1 not updated. Can you please help me how to upgrade opencv in a simple way on windows 10 64x. Thank you very much. I’m looking forward to hear from you.
You would want to create a new Python virtual environment and then install OpenCV into that environment via pip. See my OpenCV install guides for more information.
Hi Adrian,
Thanks for this really informative post. I am looking at remote sensing edge detection. I have an image where the true edge pixels =1 (derived from ground truthing) and all other pixels =0. Can I provide this to the CNN for training? I then hope to use this to derive the edge in other satellite images.
Thanks
Martin
Hi,
Thank you for the great post.
I tried the video HED detection, but when run the code, 3 small window open up but no video is shown in any of them.
I am inputting a video from my disk, and not a webcam.
So what might be the error
Hi Adrian,
Thank you for the great tutorial.
I am getting pretty good results using the HED pretrained BSDS model that you show in the tutorial. I wonder if I could get better results by using a different model or even creating my own. I am using images of pills (tablets and capsules) of all shapes where I control the lighting, camera, etc. I need edge detection to re-position, crop, etc.
Do you have any advice on other models I could try? Tutorials or papers for creating a custom model?
Many thanks Adrian for your outstanding efforts
I used HED for my project in detecting the borders of soil grains in different kinds of soil samples images However , it gives very promising results in the large size grains ignoring most of noise within the grain itself but in case of fine grained soil like small sand particles (around 0.4 inch ) I have some difficulty to get the borders of the soil grains in the image .. it shows some grains and the others are black How can I enhance the efficiency of this wonderful algorithm to overcome this step .
I think it’s safe to say that this HED model was not trained on social samples. To improve edge detection results I would suggest training your own HED model on your own soil images. That would likely improve results.