Have you ever had a horrible case of eyelid twitching? One where your eyelid just won’t stop spazzing out, no matter what you do?
That’s currently what’s going on with me — and it’s been going on for over two weeks. As you can imagine, I’m not exactly happy.
However, to combat this persistent, often distracting nuisance, I’ve started sleeping more regularly, drinking substantially less caffeine (I was only drinking two cups of coffee a day to begin with), and consuming next to no alcohol. Which is a real challenge when you enjoy a frothy beer as much as I do.
More sleep has certainly helped. And I don’t miss the occasional beer.
But man, these caffeine headaches are the worst.
Luckily, I woke up this morning to an email that practically cured me of my caffeine free woes and interminable eyelid twitching.
Opening my inbox, I found an excellent question from dedicated PyImageSearch reader Sriharsha Annamaneni, who wrote in and asked:
Hi Adrian,
In one of Jitendra Malik’s YouTube videos, he explains superpixels. I did not understand it completely. Can you explain it on your website?
Thank you,
Sriharsha Annamaneni
Great question Sriharsha. I’d be happy to discuss superpixels, I’m sure a ton of other PyImageSearch readers would be interested too.
But before we dive into some code, let’s discuss what exactly a superpixel is and why it has important applications in the computer vision domain.
OpenCV and Python versions:
This example will run on Python 2.7/Python 3.4+ and OpenCV 2.4.X/OpenCV 3.0+.
So, What’s a Superpixel?
Take a second, clear your mind, and consider how we represent images.
Images are represented as a grid of pixels, in either single or multiple channels.
We take these M x N pixel grids and then apply algorithms to them, such as face detection and recognition, template matching, and even deep learning applied directly to the raw pixel intensities.
The problem is that our pixel grid is by no means a natural representation of an image.
For example, consider the image below:
If I were to take a single pixel from the image on the left (highlighted by the red arrow) and then showed it to you on the right, could you reasonably tell me that (1) that pixel came from a Velociraptor and (2) that single pixel actually holds some sort of semantic meaning?
Unless you really, really want to play devils advocate, the answer is a resounding no — a single pixel, standing alone by itself, is not a natural representation of an image.
Taken as a whole, this grouping of pixels as a pixel grid is simply an artifact of an image — part of the process of capturing and creating digital images.
That leads us to superpixels.
Intuitively, it would make more sense to explore not only perceptual, but semantic meanings of an image formed by locally grouping pixels as well.
When we perform this type of local grouping of pixels on our pixel grid, we arrive at superpixels.
These superpixels carry more perceptual and semantic meaning than their simple pixel grid counterparts.
Benefits of Superpixels
To summarize the work of Dr. Xiaofeng Ren, we can see that a mapping from pixel grids to superpixels would (ideally) hold the desirable properties of:
- Computational efficiency: While it may be (somewhat) computationally expensive to compute the actual superpixel groupings, it allows us to reduce the complexity of the images themselves from hundreds of thousands of pixels to only a few hundred superpixels. Each of these superpixels will then contain some sort of perceptual, and ideally, semantic value.
- Perceptual meaningfulness: Instead of only examining a single pixel in a pixel grid, which caries very little perceptual meaning, pixels that belong to a superpixel group share some sort of commonality, such as similar color or texture distribution.
- Oversegmentation: Most superpixel algorithms oversegment the image. This means that most of important boundaries in the image are found; however, at the expense of generating many insignificant boundaries. While this may sound like a problem or a deterrent to using super pixels, it’s actually a positive — the end product of this oversegmentation is that that very little (or no) pixels are lost from the pixel grid to superpixel mapping.
- Graphs over superpixels: Dr. Ren refers to this concept as “representationally efficient”. To make this more concrete, imagine constructing a graph over a 50,000 x 50,000 pixel grid, where each pixel represents a node in the graph — that leads to very large representation. However, suppose we instead applied superpixels to the pixel grid space, leaving us with a (arbitrary) 200 superpixels. In this representation, constructing a graph over the 200 superpixels is substantially more efficient.
Example: Simple Linear Iterative Clustering (SLIC)
As always, a PyImageSearch blog post wouldn’t be complete without an example and some code.
Ready?
Open up your favorite editor, create slic.py , and let’s get coding:
# import the necessary packages
from skimage.segmentation import slic
from skimage.segmentation import mark_boundaries
from skimage.util import img_as_float
from skimage import io
import matplotlib.pyplot as plt
import argparse
# construct the argument parser and parse the arguments
ap = argparse.ArgumentParser()
ap.add_argument("-i", "--image", required = True, help = "Path to the image")
args = vars(ap.parse_args())
# load the image and convert it to a floating point data type
image = img_as_float(io.imread(args["image"]))
# loop over the number of segments
for numSegments in (100, 200, 300):
# apply SLIC and extract (approximately) the supplied number
# of segments
segments = slic(image, n_segments = numSegments, sigma = 5)
# show the output of SLIC
fig = plt.figure("Superpixels -- %d segments" % (numSegments))
ax = fig.add_subplot(1, 1, 1)
ax.imshow(mark_boundaries(image, segments))
plt.axis("off")
# show the plots
plt.show()
On Lines 2-7 we import the packages we’ll be using for this example. To perform the SLIC superpixel segmentation, we will be using the sckit-image implementation, which we import on Line 2. To draw the actual superpixel segmentations, scikit-image provides us with a mark_boundaries function which we import on Line 3.
From there, we import a utility function, img_as_float on Line 4, which as the name suggests, converts an image from an unsigned 8-bit integer, to a floating point data with, with all pixel values called to the range [0, 1].
Line 5 imports the io sub-package of scikit-image which is used for loading and saving images.
We’ll also make use of matplotlib to plot our results and argparse to parse our command line arguments.
Lines 10-12 handle parsing our command line arguments. We need only a single switch, --image , which is the path to where our image resides on disk.
We then load this image and convert it from an unsigned 8-bit integer to a floating point data type on Line 15.
Now the interesting stuff happens.
We start looping over our number of superpixel segments on Line 18. In this case, we’ll be examining three increasing sizes of segments: 100, 200, and 300, respectively.
We perform the SLIC superpixel segmentation on Line 21. The slic function takes only a single required parameter, which is the image we want to perform superpixel segmentation on.
However, the slic function also provides many optional parameters, which I’ll only cover a sample of here.
The first is the is the n_segments argument which defines how many superpixel segments we want to generate. This value defaults to 100 segments.
We then supply sigma , which is the smoothing Gaussian kernel applied prior to segmentation.
Other optional parameters can be utilized in the function, such as max_iter , which the maximum number of iterations for k-means, compactness , which balances the color-space proximity with image space-proximity, and convert2lab which determines whether the input image should be converted to the L*a*b* color space prior to forming superpixels (in nearly all cases, having convert2lab set to True is a good idea).
Now that we have our segments, we display them using matplotlib in Lines 24-27.
In order to draw the segmentations, we make use of the mark_boundaries function which simply takes our original image and overlays our superpixel segments.
Finally, our results are displayed on Line 30.
Now that our code is done, let’s see what our results look like.
Fire up a shell and execute the following command:
$ python superpixel.py --image raptors.png
If all goes well, you should see the following image:
In this image, we have found (approximately) 100 superpixel segmentations. Notice how locally similar regions of the image, such as the scales of the Velociraptor and the shrubbery are grouped in similar superpixels.
Now, let’s increase to 200 superpixel segments:
Same story here — local regions with similar color and texture distributions are part of the same superpixel group.
Finally, let’s generate a very dramatic oversegmentation of the image using 300 super pixels:
Notice how as the number of segments increases, the segments also become more rectangular and grid like. This is not a coincidence, and it can be further controlled by the optional compactness parameter of slic .
What's next? We recommend PyImageSearch University.
86 total classes • 115+ hours of on-demand code walkthrough videos • Last updated: October 2024
★★★★★ 4.84 (128 Ratings) • 16,000+ Students Enrolled
I strongly believe that if you had the right teacher you could master computer vision and deep learning.
Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?
That’s not the case.
All you need to master computer vision and deep learning is for someone to explain things to you in simple, intuitive terms. And that’s exactly what I do. My mission is to change education and how complex Artificial Intelligence topics are taught.
If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to successfully and confidently apply computer vision to your work, research, and projects. Join me in computer vision mastery.
Inside PyImageSearch University you'll find:
- ✓ 86 courses on essential computer vision, deep learning, and OpenCV topics
- ✓ 86 Certificates of Completion
- ✓ 115+ hours of on-demand video
- ✓ Brand new courses released regularly, ensuring you can keep up with state-of-the-art techniques
- ✓ Pre-configured Jupyter Notebooks in Google Colab
- ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
- ✓ Access to centralized code repos for all 540+ tutorials on PyImageSearch
- ✓ Easy one-click downloads for code, datasets, pre-trained models, etc.
- ✓ Access on mobile, laptop, desktop, etc.
Summary
In this blog post I explained what superpixel segmentation is and how it has many benefits in the computer vision world.
For example, working with superpixels instead of the standard pixel grid space yields us computational efficiency, perceptual meaningfulness, oversegmentation, and efficient graph representations across regions of the image.
Finally, I showed you how to utilize the Simple Linear Iterative Clustering (SLIC) algorithm to apply superpixel segmentation to your own images.
This certainly won’t be the last time we discuss superpixels on this blog, so look forward to more posts in the future!
Download the Source Code and FREE 17-page Resource Guide
Enter your email address below to get a .zip of the code and a FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning. Inside you'll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL!
Neatly explained. Wonderful!
Thanks for the great explanation!
I have a question. Is there an easy way to get the neighbouring super pixels (pixels that share a border) for a super pixel of a given ID returned by slic?
Yes, but not easily. Each of the superpixels is represented by a unique integer in the “mask” returned by SLIC. I’ll write a post soon on how to obtain each of the individual superpixel segmentations
gee!! this is really great stuff. i just found that superpixel will be available in opencv 3.0. Really looking forward to learn more about them from you.
Thanks a lot, and keep up the good work!
I have a new post on superpixel algorithms and accessing each individual segment almost ready to go. It should go live towards the end of November/early December.
How do you install skimage or the module skimage.segmentation.
I get the error when I run:
(cv)pi@raspberrypi ~/mycode $ python superpixel.py -i ../mycode/images/trex.png
Traceback (most recent call last):
File “superpixel.py”, line 3, in
from skimage.segmentation import slic
ImportError: No module named skimage.segmentation
Hi Chris, take a look at the scikit-image install instructions: http://scikit-image.org/docs/0.10.x/install.html
Hello Adrian:
I´ve trying to use your code.
In this code:
ax.imshow( mark_boundaries(image, segments) )
I got this error:
RuntimeError: array type not supported
Related to:
_min_or_max_filter
I fixed it by adding this line:
# Cast segments to int8!!!!
segments = segments.astype(np.int8)
It works fine, however computation time is >5 seconds for a 500×300 image.
Is this a normal time?
Regards.
Hi Ruben, 5 seconds seems pretty slow for a 500×300 image, I’m not sure why that would be.
Hello there!
Very helpful tutorial. I’m just running into a small issue when I copy your example…
At ax.imshow(mark_boundaries(image, segments)), I’m getting that the array is not broadcastable to the correct shape. The segments array appears to be a 10x4x512 array, whereas my original image is a 10x4x512x512 array. Is there something I’m missing when finding segments with slic?
Thanks!
UPDATE:
Ergh. Never mind. Some digging found this OTHER link of yours: https://pyimagesearch.com/2014/12/29/accessing-individual-superpixel-segmentations-python/
Using cv2’s imread and then converting to float seems to work just fine. Not sure why.
Awesome, I’m glad to hear that it worked for you!
Thanks for a great tutorial!
How would you isolate the brightest superpixel? The darkest?
Just loop over each individual superpixel, construct a mask for each, then use the mask to compute the mean of the region. Keep track of the darkest and light regions, respectively.
I am getting an error at this line:
ap.add_argument(“-i”, “–image”, required = True, help = “Path to the image”)
How it should look like if I have a path to, let’s say, “C:\user\image.jpg” ?
Thanks!
The code itself does not have to change. You need to open a terminal, navigate to the code, and execute the following command (like I do in the blog post):
$ python superpixel.py --image raptors.pngNotice how the
--imageswitch is supplied via command line and then passed into thecv2.imreadmethod. I would suggest reading up on command line arguments prior to make sure you have a good grasp on them.have the same problem and cant really grasp what you are explaining please
No worries. Be sure to read up on command line arguments. I would also suggest Googling “terminal” and read a few tutorials to familiarize yourself with the terminal and how to execute programs via the command line.
Hello, Does this method works on grayscale images?
Thank you.
Yes, superpixels can be computed for grayscale images using SLIC.
thanks,
But the algo above don’t work on grayscale images.
In your opinion, i must change in the slic function parameters or what?
Thank you Adrian.
A quick hack is to stack grayscale images into a 3D representation, like this:
image = np.dstack([gray, gray, gray])And then passing
imageas an input to SLIC.Hello Adrian. Is there a way to get the centers of the superpixels?
Sure, absolutely. If you treat each superpixel as a contour, you can then compute the centroid (i.e., center) of the superpixel using Hu moments. You can read more about this technique here.
The dstack hack did the trick. Thank you Adrian 🙂
Hello,
Great tutorial. I installed and ran your code perfectly on a stock image of a Ferrari. When I switched to a different image, I get the following error:
Segmentation fault (core dumped)
I’m running on Ubuntu 14.04, python 2.7.11, ipython 3.2.0, scikit-image 0.12.3
Do you have any advice?
It’s hard to say without seeing the exact image, but I would suggest inserting print statements in the code to debug and determine exactly where the seg-fault is caused.
Hi Adrian,
Such a neat tutorial! quick question, is there a way to to define the minimum area of each superpixel?
Thank you!
You can define the number of superpixels that can be extracted which in turn influences the size of the superpixel (i.e., the larger the superpixels are, the smaller their area will be). But you cannot define a minimum size directly with SLIC.
Hi,
Great post ! Can you also make a tutorial about connecting this to CRFs?
OpenCV also has SLIC algorithm. Is there any reason why you use scikit-image instead?
OpenCV’s SLIC implementation is still in development (hence being in the “contrib” repo). I’m also not sure if there are Python bindings for it? The scikit-image SLIC is also very easy to use and well documented.
Hello,
I just downloaded your code and tried to run it on anaconda 3.5.But just could not , do I have to make any changes to the code as I am trying to put the path in the “help=Path to image” line 11 of your code. But it is giving error.
Also I tried through command line of IPython ,it gives syntax error with python superpixel.py –image raptors.png.
Also, can I work on a satellite image for SLIC, as it is a grayscale image, if possible .Kindly clarify how?
You do not need to put the path to the image into the
helpattribute. I would suggest taking a step back and reading up on how command line arguments work before you proceed — I am quite confident this is your issue.As for working with grayscale images please see my reply to “rabih” above.
Awesome
I am having a numpy array of a grayscale image, I just want to convert it to 3 channel image, i just tried to convert it like this:
m = np.dstack(gray,gray,gray)) # here m is my original image 2d array
now when I pass it to slic and compile :
I get error name gray is not defined .
Kindly clarify how can I convert this image to grayscale.
If you are getting an error related to “gray not defined” then you do not have a variable named
gray. You should double check your code that you have correctly named your variable.Secondly, you need to wrap your items as a list before calling
np.dstack. This is the correct way to do it:m = np.dstack([gray, gray, gray])It looks like you tried to do this but I see an extra closing parenthesis in your code. You might want to check that.
Dear Adrian,
You made my day…Thanks a lot…this was a petty mistake I was doing in np.dstack. Thanks a lot.
No problem, I’m happy I could help 🙂
Hi Adrian,
I am trying to work this out in my code right now.
What should I be declaring gray as?
The “gray” variable would be the grayscale version of the image. For example<
gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)Dear Adrian,
Is there any way to work on a particular superpixel i.e. I just want to retrieve all the pixel values in a particular super pixel.
Absolutely. Please see this post on accessing individual superpixels.
i want to track object by performing segmentation using superpixel in a video. Seeing your above post, i am bit clear with segemtation part,. Can you please explain the tracking part??
Thanx in advance
hi Adran,
this topic is really practical and usefull. Congratulations.
I have some questions,
how can I pass the image with the drawn segmentes as an open cv object image.
how can I save to disk the image?
are the segments contours?
thank you 🙂
I’m not sure what you mean by passing an image with drawn segments, but you can access each individual superpixel via this post.
Hey Adrian, Will this code help in Segmenting and accessing only the nucleus part of tissue image ? I tried using active contours but applying the texture feature and checking exactly which texture dimension is best for isolating nucleus is tough. So will the superpixel help in isolating only the nucleus of the cells in tissue image ?.. if yes, please mail me some directions. Thanks Adrian.
Do you have any example images of what you’re working with, Prateeth? That would be helpful in determining if superpixel segmentation would help with your given project.
Thanks , Simply Great .. it helped me a lot
Awesome, glad to hear it Ankit!
hi, I am a new man to python could you please tell me how to modify in line 11?
Awesome tutorial! Helped me a lot!
BTW, right before the code example you wrote “Open up your favorite editor, create slic.py , and let’s get coding:” and later in running explanation (and in the zip file) the filename was “superpixel.py”. So you might want to change that. 🙂
Besides that, once again, greate tutorial and thanks a lot!
Thank you for letting me know about the filename issue, Amir! 🙂
Thanks for the explanation. I wanted to ask how would one extract some particular segment(s)?
Please see this tutorial on accessing individual superpixels.
should the image whose segmentation is to be done have some specific size or resolution? I am making a project on image tampering so i need to know what exact input specifications are to be kept for the same.
Typically we only process images that have are a maximum of 600-800 pixels along their largest dimension (i.e., width or height). Often we try to resize our images as small as possible without sacrificing accuracy (in the 200-500 pixel range). The smaller the image, the less data there is to process, therefore the algorithm can run faster.
thanks for the info! 🙂
I wanted to know if there is a way if I can implement this into a GUI application. I managed to make a button define it for segmentation but I need to be able to perform the segmentation on an already opened image in the application. There are different buttons for different operation on the images, one of them is to be superpixel segmentation.
I would suggest you take a look at this blog post where I use TKinter as a GUI. Other GUI frameworks exist such as Kivy and QT.
Hi Adrian, looking forideas to use Superpixel I found my matplotlib does not work well, even a simple plot in a 5 lines with simple or randomized images, the script stuck with no errors, warnings, nothing. The process must be stopped to run another script or anything. Could you give me a hint about what can help this works right?
Regards Thank you.
If I understand your question correctly matplotlib is not displaying the output image?
Hi Adrian!
I am trying to divide an image into 4 superpixels. The console window displays that the image has been divided into four superpixels and when i try to access them individually, I am able to access four superpixels too! But the issue is that the image on which the boundaries have been marked of superpixels (mark_boundaries function) only three lines are printed. That is, the image is shown as if it is divided into three superpixels only. One mid line and a line dividing right half into two parts is displayed but the line dividing left half into two is not marked while it is not a whole segment but two of them.
I have kept a large compactness factor to get square superpixels. Maybe you can help me out in getting proper boundaries of the superpixels.
Hi Ambika — is there a particular reason why you only want to generate four superpixels? Secondly, have you followed this tutorial on accessing individual superpixels?
Hello Adrian!
Thanks for the quick reply.
The reason why I want only four superpixels is because in my project I need to embed another image into 4 superpixels of the host image in a way that the image information does not degrade much (PSNR i mean). Thus i need exact 4 superpixels and that too in square shape (compactness is kept large).
I have referred to your tutorial addressing about how to access indivdual superpixels and that too is a great tutorial and I have used it extensively inside the code. But, the step of marking the boundaries of superpixels comes before accessing individual superpixels. Also I would like to prefer to use the labelled image (returned by slic) to be used for marking the boundaries as it contains the image’s segmented array.
Can you please help me out with that? If that’s complicated, then can you tell me any alternative through which we can mark the boundaries to display the segments formed on the whole image?
You don’t need the labeled image returned by SLIC. You can simply extract each superpixel. The boundary of each superpixel (given by computing contours) will give you the (x, y)-coordinates. Or you could simply compute the bounding rectangle for the superpixel.
hello adrian,
thank you for your tutorial,
i am a total noob super beginner in python ( this my first interaction with python), i am a graphic designer more familiar with java and processing.
i tried everything in my possible but i can’t make it work, i’am using IDLE on mac, i think i followed every step but i am missing something.
or can you guide me to a basic tutorial just to make this script work,
sorry for my stupid question!
Did you use one of my tutorials to install OpenCV on your Mac? If so, virtual environments cannot be used in the IDLE. Please use the command line and the script will execute.
Hello Adrian,
I couldn’t find the explanation on why do you load the image as a float?
Is there a specific reason for that?
The SLIC algorithm assumes we are using floats as we’ll be scaling the data during the clustering process. Images are normally represented by 8-bit unsigned integers. Using floats fixes this.
Hi
Is it expected for the output image to have the colour balance significantly shifted? My input image is rather warm, but I get a blueish output image.
Can you please link to your raptor.jpg just for testing?
Thanks
It sounds like your image channels are reversed. Are you displaying the image via matplotlib or OpenCV? Keep in mind that OpenCV stores images in BGR order rather than RGB. I believe this is likely the issue here.
Thank you for your description and explain but I have one question can I use this code to get superpixels in the hyperspectral image like done on Matlab by a superpixels3 script. thanks in advance
Sorry, I don’t have any experience with hyperspectral images.
Hello Adrian
I did not got what you taught in argument parsing. I went through your another blog on command line arguments but i think i don’t need command line arguments. I am working on Spyder installed under Anaconda. I just to read multiple images from a folder on my desktop and apply superpixeling to it. Can you suggest me something how can i proceed to execute my work? The multiple images contain both RGB and Gray Scale Images.
If you are working with computer vision, deep learning, or even computer science in general, you should take the time to learn command line arguments. You will encounter them at some point in career. Take the time now to invest in that knowledge. Walk before you run.
Hi Adrian
How can I save each segment separately? I am using in the final part of the code cv2.imwrite(‘hola.jpg’, cv2.bitwise_and(image, image, mask = mask)) but it only saves the last segment
Apart from this, How can I save the segments for a t-shirt for example? I need to segment the clothes for a person separately
Are you trying to save each individual superpixel to disk? If so, it looks like you’re overwriting the “hola.png” file at each iteration of the loop. Give each segment a unique filename.