g

·
image
-
-
OpenCV
-
-
Introduction to OpenCV
§ OpenCV is a programming library with real-time computer vision capabilities
§ Officially it was launched in 1999, OpenCV from an intel initiative
§ It is written in C++
§ First major release 1.0 was in 2006, second in 2009 and third in 2015
§ It has many algorithms provided for processing
§ It is supported in many languages like Python, C, C++, Java etc
§ It is a free and open source library
OpenCV applications

§ 2D and 3D feature toolkits § Medical image analysis

§ Street view image stitching § Structure from motion
§ Egomotion estimation
§ Motion tracking
§ Facial-recognition system
§ Augmented reality
§ Gesture recognition
§ Human-computer interaction § Video/image search and retrieval
§ Mobile robotics § Robot and driverless car navigation and control
§ Motion understanding § Driver drowsiness and distraction detection
§ Object identification
§ Automated inspection and surveillance
§ Segmentation and recognition
§ Stereopsis stereo vision
OpenCV modules

§ OpenCV is divided into modules to provide OpenCV

image processing capabilities
§ Core core imgproc imgcodecs highgui videoio
§ Core functionality is a module defining basic data
structures and also basic functions used by all calib3d features2d objdetect dnn ml
other modules in the library
§ Imgproc flann photo stiching shape superres
§ An image-processing module that includes image
filtering, geometrical image transformations, color
videostab viz videoio
space conversion, and histograms
§ Imgcodecs
§ Image codecs. Image file reading and writing
§ Videoio
§ Interface to video capturing and video codecs
Image Fundamentals
§ An image is an artifact that depicts visual perception, such as a photograph or other two-dimensional
picture, that resembles a subject—usually a physical object—and thus provides a depiction of it
§ It can be seen as a two-dimensional (2D) view of a 3D world
§ A digital image is a numeric representation of a 2D image
§ It is a finite set of digital values, which are called pixels
§ The goal of OpenCV is to transform the 2D data into
§ A new representation (for example, a new image)
§ A decision (for example, perform a concrete task)
§ A new result (for example, correct classification of the image)
§ Some useful information extraction (for example, object detection)
Color Space
§ There are several different color models (also known as color spaces)
§ It is used to explain how the image looks like
§ RGB Color Space
§ The most common one is RGB model which has three basic colors Red, Green and Blue
§ These colors are mixed together to produce broad range of colors
§ Each color (R, G and B) is usually called as a channel, which is commonly represented as an integer value in the range
of 0-255
§ Which means each channel produces 256 levels
§ Since there are 3 channels, this is called as 24-bit color depth
Image file formats
-

§ Image file formats are standardized means of organizing and storing digital images
§ An image file format may store data in an uncompressed format, a compressed format (which may be
lossless or lossy), or a vector format
§ E.g.
§ Bitmap image file (BMP)
-

§ Device independent bitmap (DIB)

compressed
-

§ Joint Photographic Experts Group (JPEG)

-
-

§ JPEG 2000
§ Graphics Interchange Format (GIF)
-

§ Portable Network Graphics (PNG)

-

§ Portable pixmap format (PPM)

-

§ Portable bitmap format (PBM

-

§ Portable graymap format (PGM)

-

§ Tagged Image File Format (TIFF)

-
-

raw image
How image is stored on computer?
§ Every image is stored as binary data (pixel)
§ OpenCV uses RGB color space by default
§ Each pixel coordinate (x, y) contains 3 values ranging for intensities of 0 to 255 (8bit)
§ Red
§ Green
§ Blue
§ Mixing different intensities of each color gives us the full color spectrum
§ Yellow
§ Red - 255
§ Green - 255
§ Blue - 0
How image is stored on computer?
How image is stored on computer? [I Be255]
§ Image is stored in multi-dimensional arrays ↓ ,

255]
[255, 255 ,
,

i Three dimensional array

-
- -

S One dimensional array

-
,

I ..
-
-
2010
·
e)

-
I
3

S
,

Two dimensional array !

S
OpenCV Image Representation
§ OpenCV uses BGR model instead of RGB model
§ The basic colors remain same but they are read in different order

§ The pixel structure can be visualized as: Pixel

Coordinate System in OpenCV
§ Image is collection of 2D non-zero binary data (pixels)
§ The left top corner starts (0, 0)
Image Processing steps

Get the image Process the image Show result

Image Processing Levels

Low Level Processing Mid Level Processing High Level Processing

§ Noise removal § Getting rectangle information § Face recognition

§ Image sharpening § Getting shapes information § Emotion recognition

§ Illumination normalization § Drowsiness and distraction

detection
§ Perspective correction § Remote heart rate measurement
from the face
Image/File handling in OpenCV

Image Image

Files Files
OpenCV
Video Video

URL Output

cv2.imread() cv2.imshow()
cv2.VideoCapture() cv2.VideoWriter()
cv2.imwrite()
Reading and Writing files
import cv2

# read the image

image = cv2.imread("logo.png")

# show the image and wait for user’s key input

cv2.imshow('image', image)
cv2.waitKey(0)
cv2.destroyAllWindows()

# write image to disk

cv2.imwrite(“/tmp/newfile.png”, image)
Reading Video

import cv2

capture = cv2.VideoCapture(0)

while capture.isOpened():
ret, frame = capture.read()
cv2.imshow('output', frame)
if cv2.waitKey(20) & 0xFF == ord('q'):
break

capture.release()
cv2.destroyAllWindows()
Drawing Shapes
Introduction to shapes
§ OpenCV provides many functions to draw basic shapes
§ Common basic shapes include
§ Lines
§ Rectangles
§ Circles
§ Texts
§ It is useful in the scenario where the result needs to be highlighted
Creating empty image
§ Image is a collection of 2D binary data (pixel)
§ To create an empty image, just create an empty array

§ image = np.zeros((400, 400, 3), dtype=np.uint8)

Terminology
§ img
§ It is the image where the shape will be drawn.
§ color
§ It is the color (BGR triplet) used to draw the shape.
§ thickness
§ If this value is positive, it is the thickness of the shape outline. Otherwise, a filled shape will be drawn.
§ lineType
§ It is the type of the shape boundary. OpenCV provides three types of line:
§ cv2.LINE_4: This means four-connected lines
§ cv2.LINE_8: This means eight-connected lines
§ cv2.LINE_AA: This means an anti-aliased line
§ shift
§ This indicates the number of fractional bits in connection with the coordinates of some points defining the shape
Drawing Lines
§ To draw line, call line function

cv2.line(img, pt1, pt2, color, thickness=1, lineType=8, shift=0)

§ E.g.

cv2.line(image, (10, 10), (100, 10), (0, 255, 255), 1)

cv2.line(image, (10, 30), (100, 30), (0, 255, 255), 5)

cv2.line(image, (10, 60), (100, 60), (0, 255, 255), 10)

Drawing Arrows
§ To draw line, call line function

cv2.arrowedLine(img, pt1, pt2, color, thickness=1, lineType=8, shift=0, tipLength=0.1)

§ E.g.

cv2.arrowedLine(image, (50, 50), (200, 50), (0, 255, 0), 5, 8, 0)

cv2.arrowedLine(image, (50, 100), (200, 100), (0, 255, 255), 1, 4, 0)

cv2.arrowedLine(image, (50, 200), (200, 200), (0, 0, 255), 10, 8, 0)

Drawing Rectangles
§ To draw rectangle use following function:

cv2.rectangle(img, pt1, pt2, color, thickness=1, lineType=8, shift=0)

§ E.g.

cv2.rectangle(image, (10, 50), (60, 300), (255, 0, 0), 3)

cv2.rectangle(image, (80, 50), (130, 300), (0, 255, 0), -1)

cv2.rectangle(image, (150, 150), (350, 300), (0, 0, 255), 10)

Drawing Circles
§ To draw circle use following function:

§ cv2.circle(img, center, radius, color, thickness=1, lineType=8, shift=0)

§ E.g.

cv2.circle(image, (50, 50), 20, (255, 0, 0), 3)

cv2.circle(image, (100, 100), 30, (0, 255, 0), -1)

cv2.circle(image, (200, 200), 40, (0, 0, 255), -1)

Drawing Texts

§ To draw text in OpenCV, use the following function

§ cv2.putText( img, text, org, fontFace, fontScale, color, thickness=1, lineType= 8)

§ E.g.

cv2.putText(image, 'OpenCV', (10, 30), cv2.FONT_HERSHEY_SIMPLEX, 0.9, (0,

255, 0), 2, cv2.LINE_4)

cv2.putText(image, 'OpenCV', (10, 70), cv2.FONT_HERSHEY_DUPLEX, 0.9, (0,

255, 255), 2, cv2.LINE_8)

cv2.putText(image, 'OpenCV', (10, 110),

cv2.FONT_HERSHEY_SCRIPT_COMPLEX, 0.9, (0, 0, 255), 2,
cv2.LINE_AA)
Image Processing Techniques
Introduction
§ Image processing is the core of OpenCV
§ OpenCV provides various algorithms for image processing
§ These algorithms include
§ Splitting and merging channels
§ Geometric transformations of images
§ translation, rotation, scaling, affine transformation, perspective transformation, and cropping
§ Arithmetic with images—bitwise operations (AND, OR, XOR, and NOT) and masking
§ Smoothing and sharpening techniques
§ Morphological operations
§ Color spaces
§ Color maps
Splitting and merging channels
§ Sometimes, you have to work with specific channels on multichannel images
§ To do this, you have to split the multichannel image into several single-channel images
§ E.g.
§ (b, g, r) = cv2.split(image)

§ After processing each channel, you can merge them back using merge function
§ E.g.
§ Image = cv2.merge((b, g, r))
Image Cropping
§ Extracting a segment of an image
§ Syntax:

§ img [ start_row : end_row, start_col : end_col ]

img = cv2.imread('messi5.jpg')
cropped = img[50:155, 200:271]
cv2.imshow('cropped', cropped)
cv2.waitKey(0)
cv2.destroyAllWindows()
Resizing the image
§ Use cv2.resize() to resize the image

img = cv2.imread('messi5.jpg')
h, w = img.shape[:2]
new = cv2.resize(img, (w * 2, h * 2))
cv2.imshow('new image', new)
cv2.waitKey(0)
cv2.destroyAllWindows()
Rotations
§ Use cv2.warpAffine to implement the translations
§ Matrix

#$%& −%()&
!=
%()& #$%&

img = cv2.imread('messi5.jpg')
§ Use cv2.getRotationMatrix2D() to create the matrix h, w = img.shape[:2]
center = (w//2, h//2)
t = cv2.getRotationMatrix2D(center, -90, 1)
new = cv2.warpAffine(img, t, (w, h))
cv2.imshow('new image', new)
cv2.waitKey(0)
cv2.destroyAllWindows()
Translation
§ Use cv2.warpAffine to implement the translations
§ Matrix

S !=
1 0 !,
0 1 !-
img = cv2.imread('messi5.jpg')
h, w = img.shape[:2]
t = np.float32([[1, 0, ⑧
--
... 10], [0, 1, 10]])
-

new = cv2.warpAffine(img, t, (w, h))

-

cv2.imshow('new image', new)

cv2.waitKey(0)
cv2.destroyAllWindows()
Arithmetic operations
§ These are simple operations that allow us to directly add or subtract to the color intensity

img = cv2.imread('messi5.jpg')
m = np.ones(img.shape, dtype='uint8') * 105
added = cv2.add(img, m)
removed = cv2.subtract(img, m)
cv2.imshow('added', added)
cv2.imshow('removed', removed)
cv2.waitKey(0)
cv2.destroyAllWindows()
Edge Detection
§ Edge detection is a very important area in OpenCV, especially when dealing with contours
§ Edge can be defined as sudden changes (discontinuities) in an image and they can encode just as
much information as pixels
§ Types
§ Sobel: to emphasize vertical or horizontal edges
§ Laplacian: gets all orientations
§ Canny: optimal due to low error rate, well defined edges and accurate detection
Edge Detection - Canny Edge
-
-

§ Developed by John F. Canny in 1986

§ Applied Gaussian blurring
§ Find intensity gradient of the image
§ Applied non-maximum suppression (i.e. removes pixels that are not edges)
§ Hysteresis – Applies thresholds (i.e. if pixel is within the upper or lower thresholds, it is considered on
the edge)
Contours
Image Segmentation

§ Segmentation is partitioning images into different regions

Introduction to Contours
§ Contours can be explained simply as a curve joining all the continuous points (along the boundary),
having same color or intensity
§ The contours are a useful tool for shape analysis and object detection and recognition.
§ For better accuracy, use binary images. So before finding contours, apply threshold or canny edge
detection.
§ In OpenCV, finding contours is like finding white object from black background. So remember, object to
be found should be white and background should be black.
Finding contours
§ See, there are three arguments in cv.findContours() function, first one is source image, second is
contour retrieval mode, third is contour approximation method
§ And it outputs a modified image, the contours and hierarchy
§ contours is a Python list of all the contours in the image
§ Each individual contour is a Numpy array of (x,y) coordinates of boundary points of the object

im = cv.imread('test.jpg')
imgray = cv.cvtColor(im, cv.COLOR_BGR2GRAY)
ret, thresh = cv.threshold(imgray, 127, 255, 0)
im2, contours, hierarchy = cv.findContours(thresh, cv.RETR_TREE, cv.CHAIN_APPROX_SIMPLE)
Draw the contours
§ To draw the contours, cv.drawContours function is used
§ It can also be used to draw any shape provided you have its boundary points
§ Its first argument is source image, second argument is the contours which should be passed as a
Python list, third argument is index of contours (useful when drawing individual contour)
§ To draw all contours, pass -1 and remaining arguments are color, thickness etc.

§ To draw all the contours in an image:

§ cv.drawContours(img, contours, -1, (0,255,0), 3)
§ To draw an individual contour, say 4th contour:
§ cv.drawContours(img, contours, 3, (0,255,0), 3)
§ But most of the time, below method will be
§ cv.drawContours(img, [cnt], 0, (0,255,0), 3)
Shape Detection
§ Use approxPolyDP() to detect the shape

approx = cv2.approxPolyDP(c, 0.01 * cv2.arcLength(c, True), True)

§ Use boundingRect(c) to detect the bounding rectangle of the contour

(x, y, w, h) = cv2.boundingRect(c)
Feature Detection
Cascading classifiers
§ Cascading is a particular case of ensemble learning based on the concatenation of several classifiers,
using all information collected from the output from a given classifier as additional information for the
next classifier in the cascade
§ Unlike voting or stacking ensembles, which are multiexpert systems, cascading is a multistage one
§ Cascading classifiers are trained with several hundred "positive" sample views of a particular object
and arbitrary "negative" images of the same size
§ After the classifier is trained it can be applied to a region of an image and detect the object in question
§ To search for the object in the entire frame, the search window can be moved across the image and
check every location for the classifier
§ This process is most commonly used in image processing for object detection and tracking, primarily
facial detection and recognition
Cascading classifiers in OpenCV
§ Object Detection using Haar feature-based cascade classifiers is an effective object detection method
proposed by Paul Viola and Michael Jones in their paper, "Rapid Object Detection using a Boosted
Cascade of Simple Features" in 2001
§ It is a machine learning based approach where a cascade function is trained from a lot of positive and
negative images. It is then used to detect objects in other images

eyeCascade = cv2.CascadeClassifier’(/haarcascade_eye.xml’)
gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
eyes = eyeCascade.detectMultiScale(gray, scaleFactor=1.1, minNeighbors=5, minSize=(30, 30))

for (x, y, w, h) in eyes :

cv2.rectangle(frame, (x, y), (x + w, y + h), (0, 255, 0), 2)