IMAGE COLORIZATION USING DEEP LEARNING

A MINI PROJECT REPORT

submitted in partial fulfilment of the requirements
for the award of the degree of
BACHELOR OF TECHNOLOGY
In
ELECTRONICS AND COMMUNICATION ENGINEERING
Submitted by
D. Chinni Vaishnavi (20B95A0403 ) D. Eureka(19B91A0450)
B. Jhansi Lakshmi Devi (19B91A042 ) A. Kamal Sai Kumar(19B91A0401)

Under the esteemed guidance of

DR.S.S.MOHAN REDDY ,
M. Tech,Ph.D,MIEEE,MIE,MIETE,MISTE,FLAEME

Professor
DEPARTMENT
OF
ELECTRONICS AND COMMUNICATION ENGINEERING
S.R.K.R. ENGINEERING COLLEGE (A)
(Affiliated to JNTU, KAKINADA)
(Recognized by A.I.C.T.E., Accredited by N.B.A.,
& Accredited by N.A.A.C. with ‘A’ grade, NEW DELHI)
CHINNA AMIRAM, BHIMAVARAM-534204
(2019-2023)
S.R.K.R. ENGINEERING COLLEGE
(Affiliated to JNTU, KAKINADA)
DEPARTMENT
OF
ELECTRONICS AND COMMUNICATION ENGINEERING

BONAFIDE CERTIFICATE
This is to certify that this mini project work entitled

IMAGE COLORIZATION USING DEEP LEARNING

is the bonafide work by
Mr/Miss ……………………………………………………………………………….

Regd.no ……………………………. of third year B. Tech along with his/her batch mates
submitted in partial fulfilment of the requirements for the award of Degree in Technology
of Engineering in Electronics and Communication Engineering during the academic year
2019-2023.

Guide: Head of the Department:

Dr S.S.MOHAN REDDY Dr N. UDAYA KUMAR

M. Tech,Ph.D,MIEEE,MIE,MIETE,MISTE,FLAEME M. Tech , Ph.D., S.M.I.E.E.E., F. I.E.T.E.,F.I.E

Department of ECE Department of ECE

CERTIFICATE OF EXAMINATION

This is to certify that we had examined the thesis and here by accord our approval of it as a study carried out
and presented in a manned required for its acceptance in a partial fulfilment for the award of BACHELOR
OF TECHNOLOGY in ELECTRONICS AND COMMUNICATION ENGINEERING for which it has
been submitted.This approval does not endorse or accept every statement made, opinion expressed or
conclusion drawn as in report. It signifies acceptance of report for the purpose for which it is submitted.

External Examiner Internal Examiner

ACKNOWLEDGEMENTS

We express our sincere thanks to our beloved and esteemed guide Dr . S.S. Mohan Reddy, for his
excellent guidance and constant encouragement throughout the project work. His timely guidance
and motivation helped me to complete the project work.

We would like to express our sincere thanks to the H.O.D of E.C.E Department, Dr . N.UDAYA
KUMAR for his cooperation, encouragement and constant review during the project.

We would like to express my sincere thanks to Dr.M.Jagapathi Raju, Principal, S.R.K.R.

Engineering College, Bhimavaram for giving me this opportunity for the successful completion of
our project.

We also thank other teaching and non-teaching staff for their assistance and help extended. We
thank one and all that have contributed directly or indirectly to our project
ABSTRACT
Image colorization is the process of estimating RGB colours for grayscale images or video
frames to improve their aesthetic and perceptual quality. Deep learning techniques for image colorization have
progressed notably over the last decade, calling the need for a systematic survey and benchmarking of these
techniques. Enhancement of Cultural Heritage such as historical images is very crucial to safeguard the
diversity of cultures. Automated colorization of black and white images has been subject to extensive research
through computer vision and machine learning techniques. Our research addresses the problem of generating a
plausible coloured photograph of ancient, historically black, and white images using deep learning techniques
without direct human intervention.

Motivated by the recent success of deep learning techniques in image processing, a feed-forward, deep
Convolutional Neural Network (CNN) our model is trained by sets of sample images using back-propagation to
recognize the pattern in RGB and grayscale values. The trained neural network is then used to predict two a*
and b* chroma channels given grayscale, L channel of test images. CNN vividly colorizes images with the help
of the fusion layer accounting for local features as well as global features.

The model is trained on the dataset with 1.2 K historical images comprised of old and ancient photographs,
each having 256 × 256 resolution. Other than presenting the training results, the public acceptance or subjective
validation of the generated images is assessed by means of a user study.
DECLARATION
This is to certify that the project entitled “IMAGE COLOURIZATION
USING DEEP LEARNING” which is submitted by D. Chinni Vaishnavi (20B95A0403), B. Jhansi
Lakshmi Devi(19B91A0423),D. Eureka(19B91A0450), A.Kamal Sai Kumar(19B91A0401) in partial
fulfillment of the requirement for the award of degree in B. Tech in Electronics and Communication
Engineering to S.RK.R. Engineering college, affiliated to JNTU KAKINADA comprises only our
original work and due acknowledgement has been made in text to all other material used.

Date: D. Chinni Vaishnavi

(20B95A0403)

B. Jhansi Lakshmi Devi

(19B91A0423)

D. Eureka

(19B91A0450)

A. Kamal Sai Kumar

(19B91A0401)
CONTENTS
Title Page

Certification of Examination

Acknowledgements

Abstract
CHAPTER 1: Introduction
1.1. Problem Statement
1.2. Previous Work
1.3 Literature Survey
CHAPTER 2: Deep Learning
2.1. Deep Learning
2.2. Neural Networks
CHAPTER 3: Convolutional Neural Networks
3.1. Overview of CNN
3.2. Working of CNN
CHAPTER 4: Methodology
4.1. Key Terms
4.2. Operation
4.3. Software Tools Used
CHAPTER 5: Source Code
CHAPTER 6: Simulation Results
CHAPTER 7: Conclusion and Future work
7.1. Conclusion
7.2. Future Word
7.3.References
CHAPTER 1

INTRODUCTION

Historical black-and-white images are regarded as irreplaceable, with exceptional artistic value.
However, by looking at them it is impossible to fully imagine the actual scene, as colour is a very important
component of visual representation. The colorization of black-and-white images greatly alters the perspective of
the viewer. The time gap between the past and the present fades away while making the scene more
conceivable. However, insight into the authentic colours of early photographs is often unavailable, making
satisfactory reconstruction difficult. Main applications of colorization include the revival of historical black-
and-white images, movie restoration and colouring astronomy photographs.

1.1.Problem Statement:

This project investigates the problem of colorizing grayscale images. The main challenge arises as
various colours can give rise to same grayscale values. In this project an attempt has been made to come with
methods to colorize images without human assistance. The algorithm works by a training a model on a large
corpus of images and then using the developed model to colorize grayscale images. This project formulates
colorization as a regression problem and neural networks are employed to solve regression. A large image
database is used for training the model.

1.2.Previous Work:

Colorization term was invented for the purpose of colouring black and white movies and photograph to
make it more appealing to the viewers. There are two main approaches that are popularly followed :

 First method requires users to give a reference image along with the target image. Then people apply
various colour transfer mechanism which have been improved over the years.
 This is the most popular approach in this field. In second method, people try to build a collection of
various categories of images into a database and then applying various image searching methods to find
the most similar image for transferring colours.

1.3. Literature Survey:

In this paper grey-scale images have been coloured using various deep learning approaches. A model is
proposed which is based on a neural network. This model starts from scratch and various high-quality features
are extracted. There is a pre-trained model by the name of "Inception-ResNet-v2”. In this paper, a particular
encoder-decoder model is present which can work on images that can be of any size and ratio. After calculating
the results "public acceptance" of the model is carried out. A separate user study is carried out for this. Then a
separate menu of applications that has different types of images is presented. So, the steps which are carried out
in this model can be summarized as follows:

 High-level features are extracted using the pre-trained model specified above.
 Analysis of the architecture is carried out using CNN.
 A separate user study is carried out to check whether that model is publicly accepted or not.
 A separate set of old pictures are presented and the model is tested on that.
CHAPTER 2

DEEP LEARNING

2.1. Deep Learning:

Deep learning is a machine learning technique that teaches computers to do what comes naturally to
humans.. Deep learning is getting lots of attention lately and for good reason. It’s achieving results that were not
possible before. In deep learning, a computer model learns to perform classification tasks directly from images,
text, or sound. Deep learning models can achieve state-of-the-art accuracy, sometimes exceeding human-level
performance. Models are trained by using a large set of labelled data and neural network architectures that
contain many layers. Most deep learning methods use neural network architectures, which is why deep learning
models are often referred to as deep neural networks. The term “deep” usually refers to the number of hidden
layers in the neural network. Traditional neural networks only contain 2-3 hidden layers, while deep networks
can have as many as 150.Deep learning models are trained by using large sets of labelled data and neural
network architectures that learn features directly from the data without the need for manual feature extraction.
Deep learning is a key technology behind driverless cars, enabling them to recognize a stop sign, or to
distinguish a pedestrian from a lamppost. It is the key to voice control in consumer devices like phones, tablets,
TVs, and hands-free speakers.

2.2. Neural Network:

A neural network is a method in artificial intelligence that teaches computers to process data in a way
that is inspired by the human brain. It is a type of machine learning process, called deep learning, that uses
interconnected nodes or neurons in a layered structure that resembles the human brain. It creates an adaptive
system that computers use to learn from their mistakes and improve continuously. Thus, artificial neural
networks attempt to solve complicated problems, like summarizing documents or recognizing faces, with
greater accuracy. Neural networks can help computers make intelligent decisions with limited human
assistance. This is because they can learn and model the relationships between input and output data that are
nonlinear and complex. For instance, they can do the following tasks. A neural network works similarly to
the human brain’s neural network. A “neuron” in a neural network is a mathematical function that collects
and classifies information according to a specific architecture. The different types of neural networks are
Perceptron ,Feed Forward Networks, Multilayer Perceptron, Radial Based Networks, Convolutional Neural
Networks ,Recurrent Neural Networks, Long Short-Term Memory Networks.

Figure. 1
CHAPTER 3

CONVOLUTIONAL NEURAL NETWORKS

3.1. Overview of CNN:

A convolutional neural network (CNN) is a type of artificial neural network used in image
recognition and processing that is specifically designed to process pixel data. When it comes to image
classification, the most used neural networks are Convolution Neural Networks (CNN). CNN contain
multiple convolution layers which are responsible for the extraction of important features from the image.
A digital image is a binary representation of visual data. It contains a series of pixels arranged in a grid-
like fashion that contains pixel values to denote how bright and what colour each pixel should be .The

earlier layers are responsible for low-level details and the later layers are responsible for more high-level

features.

Figure. 2

3.2. Working of CNN:

The Convolution operation uses a custom matrix, also called as filters, to convolute over the input
image and produce maps. These filters are initialized randomly and then are updated via backpropagation. One
example of such a filter is the Canny Edge Detector, which is used to find the edges in any image.

After the convolution layer, there is a pooling layer which is responsible for the aggregation of the maps
produced from the convolutional layer. It can be Max Pooling, Min Pooling, etc. For regularization, CNNs
also include an option for adding dropout layers which drop or make certain neurons inactive to reduce
overfitting and quicker convergence.

CNNs use ReLU (Rectified Linear Unit) as activation functions in the hidden layers. As the last

layer, the CNNs have a fully connected dense layer and the activation function mostly as Softmax for

classification, and mostly ReLU for regression.

Figure.3
Figure.4
Sometimes padding is also used in between these operations when edges of an image constitute important
aspects that might be helpful while training the model. It can also be used to contain the shrinkage along height

and width due to operations like pooling. Generally, the depth of hidden layers are high as the number of
filters used in a convolution operation are of high number. Number of filters are maintained high as each filter
learns new features or new shapes.

The weight of every convolution operation are updated & used.

backpropagation. Backpropagation involves calculation of gradients which in turn helps w to reach an ideal
state where the error rate (or any other loss metric) is very less. In Pytorch, backpropagation is performed
using torch. backward function. Mathematically, we can say that this function does something similar to
operation J.v, where J is Jacobian matrix and v, is the gradient of loss w.r.t next layer. Jacobian matrix
consists of partial derivatives and can be considered as a local gradient . For a function, y = f(x) where y and x
are vectors, J.v is

Where l is the loss function.

Batch Normalization (BN) is performed to address the issues of Internal Covariate Shift (ICF). Because of
different distributions of weights in each operation, hidden layers take more time to adjust to these
changes. Pytorch, during training a batch of hidden layers keeps running estimates of its computed mean and

variance, which are later used for normalization during evaluation/testing in that particular layer. By default,
the elements of γ are set to 1 and the elements of β are set to 0
CHAPTER 4
METHODOLOGY
4.1. Key Terms:
 RGB Colour Space:
An RGB colour space is any additive colour space based on the RGB colour model.[1][2] An RGB colour
space is defined by chromaticity coordinates of the red, green, and blue additive primaries, the white point
which is usually a standard illuminate, and the transfer function which is also known as the tone response
curve (TRC) or gamma.

 LAB Colour Space:

The CIELAB colour space, also referred to as L*a*b* , is a colour space defined by the International
Commission on Illumination (abbreviated CIE) in 1976. It expresses colour as three values: L* for
perceptual lightness, and a* and b* for the four unique colours of human vision: red, green, blue, and
yellow. CIELAB was intended as a perceptually uniform space, where a given numerical change
corresponds to a similar perceived change in colour.

4.2. Operation:
1. Convert all training images from the RGB colour space to the Lab colour space

2.Use the L channel as the input to the network and train the network to predict the ab channels.

3.Combine the input channel with the predicted ab channels.

4.Convert the Lab image back to RGB.

 During Training Time:Figure.5

Initially a number of RGB input images are taken and pre-processed. During pre-processing the RGB
image is converted to LAB colour space. After pre-processing the lightness from the LAB image is given to
a Convolutional Network(CNN) to train it . After training it with a number of images the model will be able
to predict the A*( red-green),B*(blue-yellow) channels of the LAB image . Then the model is trained.

 During Test Time:

The grey scaled test images are applied as input to the trained CNN . The CNN will extract the A*,B*
channels and concatenate them with the L channel of the grey scaled image. Now the LAB image is formed.
To make it visually perceptual to humans the LAB image is converted to the RGB image. Thus the output is
obtained.

Figure.6
4.3. SOFTWARE TOOLS USED:

 Programming Language:

Python is a general-purpose high level programming language that is widely used in data science and for
producing deep learning algorithms. This brief tutorial introduces Python and its libraries like Numpy, Scipy
,Pandas, Matplotlib; frameworks like Theano, TensorFlow, Keras.

 Libraries:

NUMPy: NumPy is a library for the Python programming language, adding support for large, multi-
dimensional arrays and matrices, along with a large collection of high-level mathematical functions to
operate on these arrays.

OPEN CV: OpenCV is a library of programming functions mainly aimed at real-time computer
vision. It is a great tool for image processing and performing computer vision tasks.

 IDE:
The Jupyter Notebook is an open source web application that you can use to create and share documents that
contain live code, equations, visualizations, and text. Jupyter Notebook is maintained by the people at
Project Jupyter.
CHAPTER 5
SOURCE CODE
import numpy as np
import cv2
image_name = input("enter name of image along with extension:")
print("loading models.....")
net = cv2.dnn.readNetFromCaffe('colorization_deploy_v2.prototxt','colorization_release_v2.caffemodel')
pts = np.load('pts_in_hull.npy')
class8 = net.getLayerId("class8_ab")
conv8 = net.getLayerId("conv8_313_rh")
pts = pts.transpose().reshape(2,313,1,1)
net.getLayer(class8).blobs = [pts.astype("float32")]
net.getLayer(conv8).blobs = [np.full([1,313],2.606,dtype='float32')]
image = cv2.imread(image_name)
scaled = image.astype("float32")/255.0
lab = cv2.cvtColor(scaled,cv2.COLOR_BGR2LAB)
resized = cv2.resize(lab,(224,224))
L = cv2.split(resized)[0]
L -= 50
net.setInput(cv2.dnn.blobFromImage(L))
ab = net.forward()[0, :, :, :].transpose((1,2,0))
ab = cv2.resize(ab,( image.shape[1],image.shape[0]))
L = cv2.split(lab)[0]
colorized = np.concatenate((L[:,:,np.newaxis], ab), axis=2)
colorized = cv2.cvtColor(colorized,cv2.COLOR_LAB2BGR)
colorized = np.clip(colorized,0,1)
colorized = (255 * colorized).astype("uint8")
cv2.imshow("Original",image)
cv2.imshow("Colorized",colorized)
cv2.waitKey(0)
CHAPTER 6
SIMULATION RESULTS

Figure .7

Figure .8

Figure.9
Figure.10

Figure .11

Figure .12
CHAPTER 7

CONCLUSION AND FUTURE SCOPE

7.1. CONCLUSION:
Through this project, we have demonstrated the efficiency and potential of using deep
convolutional neural net-works to colorize black and white images. This project was an attempt towards
achieving full automated image colorization. Baseline implementation consisted of training a deep neural
network model using pre-defined features. The performance of this model was evaluated and
shortcomings of the model were analysed .This proposition works well for the images of nature and
different living species like animals, birds, human beings etc. We have achieved 90% accuracy for pure
black and white images . The end result of colorised image was differing in least ,which go unnoticed
when compared with the original. Our results indicate that the presented model can be used as a creativity
tool to assist human knowledge in nearby future. As our project is based on “Deep learning based
Convolutional Neural Networks and Computer Vision” ,it is completely automated. Thus, this saves a lot
of time and energy which is wasted by the older versions.

7.2.FUTURE SCOPE:
This idea is a foundation for future research. Several extensions and improvements like image
quality enhancement, image resizing, image cartooning etc can be added to improvise this model. Video
colourization would be one of the most wonderful extensions for this project. In order to implement this
project for any video, the method would currently require further refinement, if, performed manually by
an artist. If trained on a larger dataset, the predictive power of the model would increase and is likely to
produce more consistent colorization. For future work, it would be interesting to compare colorization
produced by ResNet models with significantly more depth (which require more computational resources
to train) and models based on conditional generative adversarial networks, as the results they have been
able to put together when applied to natural images are quite impressive and allow the user to have more
control over the result by adjusting the latent space variable. Additionally, the CNN model could be
adjusted to generate scribbles to use in conjunction with the algorithms we mentioned in Chapter 3,
instead of full colorization. This could lead to results that more closely match the currently used
colorization methods that apply color to cartoon movies.
7.3. REFERENCES:

1. Z. Cheng, Q. Yang, and B. Sheng, “Deep colorization,” in IEEE International Conference on Computer
Vision, 2015, pp. 415–423.

2. S. Yoo, H. Bahng, S. Chung, J. Lee, J. Chang, and J. Choo, “Coloring with limited data: Few-shot
colorization via memory augmented networks,” in IEEE Conference on Computer Vision and Pattern
Recognition, 2019, pp. 11 283–11 292.

3. T. Welsh, M. Ashikhmin, and K. Mueller, “Transferring color to greyscale images,” in 29th annual
conference on Computer graphics and interactive techniques, 2002, pp. 277–280.

4. A. Levin, D. Lischinski, and Y. Weiss, “Colorization using optimization,” in Proceedings of

International Conference on Computer Graphics and Interactive Techniques’ ACM, 2004, pp. 689–694.

5. Y.-C. Huang, Y.-S. Tung, J.-C. Chen, S.-W. Wang, and J.-L. Wu, “An adaptive edge detection based
colorization algorithm and its applications,” in 13th annual ACM international conference on Multimedia,
2005, pp. 351–354.

6. https://ieeexplore.ieee.org/document/8278079

7. https://ieeexplore.ieee.org/document/9262377

8. https://learnopencv.com/convolutional-neural-network-based-image-colorization-using-opencv/

9. https://ieeexplore.ieee.org/document/7900208

10. https://ieeexplore.ieee.org/document/9390626

11. https://ieeexplore.ieee.org/document/9219019