Preprints (www.preprints.org) | NOT PEER-REVIEWED | Posted: 11 July 2020 doi:10.20944/preprints202007.0230.

v1

Machine Learning: A Review of Learning Types

Shagan Sah 1

Abstract The availability of large scale labeled data samples makes it

an expensive approach for tasks where data is scarce. These
In this paper, various machine learning techniques approaches can be broadly divided into two main categories-
are discussed. These algorithms are used for many
applications which include data classification, pre- 2.1.1. C LASSIFICATION
diction, or pattern recognition. The primary goal
of machine learning is to automate human assis- The output variable is one of some known number of cat-
tance by training an algorithm on relevant data. egories. For example, ”cat” or ”dog”, ”positive” or ”nega-
This paper should also serve as a collection of tive”.
various machine learning terminology for easy
reference. 2.1.2. R EGRESSION
The output variable is a real or a continuous value. For
example, ”price”, ”geographical location”.
1. Introduction
Further reading (Kotsiantis et al., 2007).
Machine learning is the study of computer algorithms that
provides systems the ability to automatically learn and im-
prove from experience. It is generally seen as a sub-field
of artificial intelligence. Machine learning algorithms al-
low the systems to make decisions autonomously without
any external support. Such decisions are made by finding
valuable underlying patterns within complex data.
Based on the learning approach, the type of data they input
and output, and the type of problem that they solve, there
are few primary categories of machine learning algorithms-
supervised, unsupervised and reinforcement learning. There Figure 1. Overview of supervised learning. Input examples are
are a few hybrid approaches and other common methods categorized into a known set of classes.
that offer natural extrapolation of machine learning problem
forms.
In the following sections, all the methods are briefly de-
scribed. Recommended literature for further reading is also
listed. This paper should also serve as a collection of various 2.2. Unsupervised Learning
machine learning terminology for easy reference.
Unsupervised learning is applied when the data is available
only in the form of an input and there is no correspond-
2. Primary Approaches ing output variable. Such algorithms model the underlying
2.1. Supervised Learning patterns in the data in order to learn more about its charac-
teristics.
Supervised learning is applied when the data is in the form
of input variables and output target values. The algorithm One of the main types of unsupervised algorithms is clus-
learns the mapping function from the input to the output. tering. In this technique, inherent groups in the data are
discovered and then used to predict output for unseen inputs.
1
California, USA. Correspondence to: Shagan Sah An example of this technique would be to predict customer
<[email protected]>. purchasing behavior.
Further reading (Bengio et al., 2012).

© 2020 by the author(s). Distributed under a Creative Commons CC BY license.

Preprints (www.preprints.org) | NOT PEER-REVIEWED | Posted: 11 July 2020 doi:10.20944/preprints202007.0230.v1

Machine Learning: A Review of Learning Types

Figure 4. Overview of semi-supervised learning. The clusters

Figure 2. Overview of unsupervised learning. Input samples are formed by a large amount of unlabeled data are used to classify a
grouped into clusters based on the underlying patterns. limited amount of labeled data.

2.3. Reinforcement Learning 3.2. Self-supervised Learning

Reinforcement learning is applied when the task at hand Self-supervised learning is a form of unsupervised learning
is to make a sequence of decisions towards a final reward. where the training data is autonomously (or automatically)
During the learning process, an artificial agent gets either labeled. The data is not required to be manually labelled
rewards or penalties for the actions it performs. Its goal is but is labeled by finding and exploiting the relations (or
to maximize the total reward. Examples include learning correlations) between different input features. This is done
agents to play computer games or performing robotics tasks in an unsupervised manner by forcing the network to learn
with end goal. semantic representation about the data. Knowledge is then
transferred to the model for the main task. It is sometimes
Further reading (François-Lavet et al., 2018). referred to as pretext learning.
Further reading (Jing & Tian, 2020).

Figure 3. Overview of reinforcement learning. An agent observes

the environment state and performs actions to maximize an overall
reward.
Figure 5. Overview of self-supervised learning. A model is learned
on unlabeled data (data is similar to the labeled data) using a
dummy task and then the learned model is used for the main task.

3. Hybrid Approaches
3.1. Semi-supervised Learning
3.3. Self-taught Learning
As the name suggests, this is an intermediate between su-
pervised and unsupervised learning techniques. These algo- Self-taught learning is applicable in solving a supervised
rithms are trained using a combination of labeled and unla- learning task given both labeled and unlabeled data, where
beled data. In a common setting, there is a small amount of the unlabeled data does not share the class labels or the
labeled data and a very large amount of unlabeled data. A generative distribution of the labeled data. In simple words,
basic procedure involved is that first similar data is clustered it applies transfer learning from unlabeled data. Once the
using an unsupervised learning algorithm and then existing representation has been learned in the first stage, it can then
labeled data is used to label the rest of the unlabeled data. be applied repeatedly to different classification tasks.
Further reading (Chapelle et al., 2009). Further reading (Raina et al., 2007).
Preprints (www.preprints.org) | NOT PEER-REVIEWED | Posted: 11 July 2020 doi:10.20944/preprints202007.0230.v1

Machine Learning: A Review of Learning Types

Figure 8. Overview of active learning. From a large pool of unla-

beled data, a model selects the samples that it can learn most from
Figure 6. Overview of self-taught learning. A model is learned on for a required task. The selected data is labeled and then used to
unlabeled data (data may be from a dissimilar domain as the data train the model.
used in main task) and then trained with small amounts of labeled
data.
4.3. Online Learning
Online learning involves training using data that becomes
available in a sequential order. This technique contrasts with
batch sampling based learning where the complete training
4. Other Common Approaches data is always available. It is useful in scenarios where
4.1. Multi-task Learning algorithms are required to dynamically adapt to novel data
patterns from all incoming data.
Multi-task learning refers to a training paradigm where mul-
tiple tasks are learned at the same time by a single model. 4.3.1. I NCREMENTAL L EARNING
This allows leveraging of useful relationships contained in
related tasks. They improve generalization across all the Incremental learning strategy is very similar to (or at times
tasks and hence improve prediction accuracy for specific same as) online learning. The main difference is that in
tasks compared to models trained individually. online learning a training sample is used only once from an
incoming data stream. In incremental learning, samples are
Further reading (Zhang & Yang, 2017). usually picked from a finite dataset and the same samples
can be processed multiple times.

4.3.2. S EQUENTIAL L EARNING

Sequential learning is a term widely used for learning with
data that has a temporal ordering to it. Under certain condi-
tions, it can be also interpreted as a type of online learning.
Further reading (Losing et al., 2018).
Figure 7. Overview of multi-task learning. Model learning is
achieved through multiple tasks that represent properties of the
data.

4.2. Active Learning

This algorithm proactively selects a subset of data samples
Figure 9. Overview of online learning. The model learns from a
that is wants to learn from. The samples are selected from continuous incoming stream of data.
a large pool of unlabeled samples and are then labeled.
This allows the algorithm to perform better than traditional
methods with substantially less labeled data for training. 4.4. Transfer Learning
Such methods are highly useful where unlabeled data may
Transfer learning refers to training (or fine-tuning) a devel-
be abundant but labels are difficult, time-consuming, or
oped algorithm on a different yet related task. The main idea
expensive to obtain.
is about transferring knowledge from one supervised learn-
Further reading (Settles, 2009). ing task to another and hence it generally requires further
Preprints (www.preprints.org) | NOT PEER-REVIEWED | Posted: 11 July 2020 doi:10.20944/preprints202007.0230.v1

Machine Learning: A Review of Learning Types

labeled data from a different but related task. One limitation

of this approach is the requirement of additional labeled
data, rather than unlabeled data, for the new supervised
learning tasks.
Further reading (Weiss et al., 2016).

Figure 12. Overview of ensemble learning. Multiple models are

learned for the same task and their individual predictions are used
to obtain the best result.

4.7. Adversarial Learning

In adversarial machine learning, a model is explicitly trained
on a lot of adversarial data such that it is not fooled by those
Figure 10. Overview of transfer learning. A model is learned on a
dataset and the knowledge is transferred to a new task by retaining examples. When a standard machine learning model is de-
a part of the model. ployed in the real world, it is susceptible to failures due
to presence of intelligent and adaptive adversaries. This
is because common machine learning techniques are de-
4.5. Federated Learning signed for stationary environments where the training and
test data are assumed to be generated from the same statisti-
Federated learning enables training in a distributed man-
cal distribution. Adversarial learning enhances the model
ner using a large corpus of data residing on independent
capability against a malicious adversary by surreptitiously
devices. It de-centralizes model training without sharing
manipulating the input data.
data samples among individual entities. This addresses the
fundamental problems of privacy, ownership, and locality Further reading (Lowd & Meek, 2005).
of data.
Further reading (Bonawitz et al., 2019).

Figure 13. Overview of adversarial learning. The model is trained

to discriminate between real and synthetic data samples.
Figure 11. Overview of federated learning. The data resides with
individual entities which provides model updates to a centralized
server without sharing their data. 4.8. Meta Learning
In a meta-learning paradigm, the machine learning model
4.6. Ensemble Learning gains experience over multiple learning episodes that often
cover a distribution of related tasks and then uses this ex-
Ensemble learning is a machine learning paradigm where
perience to improve any future learning performance. The
multiple learners are trained to solve the same problem.
goal is to solve new tasks with only a small number of train-
They obtain better predictive performance than could be ob-
ing samples. In contrast to conventional machine learning
tained from any one of the constituent learning algorithms.
approaches where a given task is learned from scratch using
An ensemble contains a number of learners which are usu-
a fixed learning algorithm, meta-learning aims to improve
ally called base learners. The generalization ability of an
the learning algorithm itself, given the experience of multi-
ensemble is usually much stronger than that of the base
ple learning episodes, hence is also referred to as learn the
learners.
learning process. Examples include few-shot learning and
Further reading (Polikar, 2012). metric learning.
Preprints (www.preprints.org) | NOT PEER-REVIEWED | Posted: 11 July 2020 doi:10.20944/preprints202007.0230.v1

Machine Learning: A Review of Learning Types

Further reading (Hospedales et al., 2020). is done by processing the training data to find a hypothesis
(or a function) that best fits the training examples. The
4.8.1. M ETRIC L EARNING goal is to classify a data point as either belonging to or not
belonging to a particular concept or idea. In this context, a
Metric learning is a form of machine learning that utilizes
concept can be viewed as a boolean-valued function defined
distances between data samples. It learns from similarity
over a large data set. A common approach is using the
or dis-similarity among the examples. It is often used for
Find-S Algorithm.
dimensionality reduction, recommendation systems, identity
verification etc. Further reading (Mitchell et al., 1997).
Further reading (Suárez et al., 2018).

Figure 14. Overview of meta learning. The model gains experience

by learning over multiple learning episodes on related tasks before
using the knowledge on the main task. Figure 16. Overview of concept learning. The model finds the best
hypothesis that satisfies all the boolean concepts in the data.

4.9. Targeted Learning

4.11. Bayesian Learning
Targeted learning methods build machine learning based
models that estimate the features of the probability distribu- Bayesian learning uses Bayes’ theorem to determine the
tion of the data. In simple words, they target the learning conditional probability of a hypotheses given some evidence
towards a certain parameter of interest. These methods are or observations. In contrast to maximum likelihood learning,
also used to obtain influence statistics about the model pa- Bayesian learning explicitly models uncertainty over both
rameters. They are popular since the estimated parameter the input data and the model parameters. The initial or prior
selection allows the subjective choices made by machines knowledge is incorporated though a distribution over the
to mimic human behavior. parameters.
Further reading (van der Laan & Petersen, 2012). Further reading (Bernardo & Smith, 2009).

Figure 17. Overview of Bayesian learning. The model used ini-

Figure 15. Overview of targeted learning. The model makes tar- tial knowledge (prior) and the data observations to determine the
geted updates to specific parameters that minimize the statistical conditional probability of for data using Bayes’ theorem.
goal.

4.12. Analytical Learning

4.10. Concept Learning
The goal is to use logical reasoning to identify features that
This approach involves learning from concepts to identify can distinguish among different input examples. It is a non-
whether a sample belongs to a specific category or not. This statistical learning approach that allows a learner to process
Preprints (www.preprints.org) | NOT PEER-REVIEWED | Posted: 11 July 2020 doi:10.20944/preprints202007.0230.v1

Machine Learning: A Review of Learning Types

information, break it into component parts (features), and 4.14. Deep Learning
generate hypotheses by using critical and logical thinking
Deep learning is a technique to implement various machine
skills. These approaches analyze each problem instance
learning algorithms using multi-layers neural networks.
individually, rather than a set of problem instances. Such
These multiple processing layers learn representations of
approaches do not require large amounts of training data to
data with multiple levels of abstraction for understanding
work well.
the input data.
4.12.1. I NDUCTIVE L EARNING Further reading (LeCun et al., 2015).
The goal is to use statistical reasoning to identify features
that empirically distinguish different input examples. The
performance is highly dependent on the number of training
samples.
Further reading (Kawaguchi et al., 2019; Ruiz, 2012).

Figure 20. Overview of deep learning. A term used for a multi-

layered neural network that learns feature extraction and classifica-
tion (or other discrimination task) in an end-to-end manner.

4.15. Curriculum Learning

In the curriculum learning paradigm, the training data is
organized in a meaningful order which gradually illustrates
more complex concepts. The idea is analogous to human
Figure 18. Overview of analytical and inductive learning. This learning in an organized education system that introduces
terminology is used to distinguish between models learning using
different concepts at different times. This technique allows
logical or statistical reasoning.
exploitation of previously learned concepts to ease the learn-
ing of new abstractions.
Further reading (Bengio et al., 2009).
4.13. Multi-modal Learning
These are types of algorithms that learn features over multi-
ple modalities. Examples of modalities include visual, audi-
tory, kinesthetic among other sensory data. By combining
such modes, learners are able to combine information from
different sources and hence yield better feature extraction
and predictions at a large scale.
Further reading (Baltrušaitis et al., 2018).
Figure 21. Overview of curriculum learning. The model is learned
in stages where data is organized in an meaningful order such that
the complexity gradually increases.

References
Baltrušaitis, T., Ahuja, C., and Morency, L.-P. Multimodal
machine learning: A survey and taxonomy. IEEE trans-
actions on pattern analysis and machine intelligence, 41
(2):423–443, 2018.

Bengio, Y., Louradour, J., Collobert, R., and Weston, J.

Figure 19. Overview of multi-modal learning. The model is Curriculum learning. In Proceedings of the 26th annual
learned using data from multiple modalities to exploit their re- international conference on machine learning, pp. 41–48,
lationships. 2009.
Preprints (www.preprints.org) | NOT PEER-REVIEWED | Posted: 11 July 2020 doi:10.20944/preprints202007.0230.v1

Machine Learning: A Review of Learning Types

Bengio, Y., Courville, A. C., and Vincent, P. Unsupervised Raina, R., Battle, A., Lee, H., Packer, B., and Ng, A. Y.
feature learning and deep learning: A review and new Self-taught learning: transfer learning from unlabeled
perspectives. CoRR, abs/1206.5538, 1:2012, 2012. data. In Proceedings of the 24th international conference
on Machine learning, pp. 759–766, 2007.
Bernardo, J. M. and Smith, A. F. Bayesian theory, volume
405. John Wiley & Sons, 2009. Ruiz, C. Using domain knowledge in machine learn-
ing inductive vs analytical learning, 2012. URL
Bonawitz, K., Eichner, H., Grieskamp, W., Huba, D., Inger- http://web.cs.wpi.edu/˜cs539/f12/
man, A., Ivanov, V., Kiddon, C., Konecny, J., Mazzocchi, Slides/ilp_ebl.pdf.
S., McMahan, H. B., et al. Towards federated learning at
scale: System design. arXiv preprint arXiv:1902.01046, Settles, B. Active learning literature survey. Technical
2019. report, University of Wisconsin-Madison Department of
Computer Sciences, 2009.
Chapelle, O., Scholkopf, B., and Zien, A. Semi-supervised Suárez, J. L., Garcı́a, S., and Herrera, F. A tutorial on
learning (chapelle, o. et al., eds.; 2006)[book reviews]. distance metric learning: mathematical foundations, algo-
IEEE Transactions on Neural Networks, 20(3):542–542, rithms and software. arXiv preprint arXiv:1812.05944,
2009. 2018.
François-Lavet, V., Henderson, P., Islam, R., Bellemare, van der Laan, M. J. and Petersen, M. L. Targeted learning.
M. G., and Pineau, J. An introduction to deep reinforce- In Ensemble Machine Learning, pp. 117–156. Springer,
ment learning. arXiv preprint arXiv:1811.12560, 2018. 2012.

Hospedales, T., Antoniou, A., Micaelli, P., and Storkey, Weiss, K., Khoshgoftaar, T. M., and Wang, D. A survey of
A. Meta-learning in neural networks: A survey. arXiv transfer learning. Journal of Big data, 3(1):9, 2016.
preprint arXiv:2004.05439, 2020.
Zhang, Y. and Yang, Q. A survey on multi-task learning.
Jing, L. and Tian, Y. Self-supervised visual feature learning arXiv preprint arXiv:1707.08114, 2017.
with deep neural networks: A survey. IEEE Transactions
on Pattern Analysis and Machine Intelligence, 2020.

Kawaguchi, K., Bengio, Y., Verma, V., and Kaelbling, L. P.

Generalization in machine learning via analytical learning
theory. stat, 1050:6, 2019.

Kotsiantis, S. B., Zaharakis, I., and Pintelas, P. Supervised

machine learning: A review of classification techniques.
Emerging artificial intelligence applications in computer
engineering, 160:3–24, 2007.

LeCun, Y., Bengio, Y., and Hinton, G. Deep learning. nature,

521(7553):436–444, 2015.

Losing, V., Hammer, B., and Wersing, H. Incremental on-

line learning: A review and comparison of state of the art
algorithms. Neurocomputing, 275:1261–1274, 2018.

Lowd, D. and Meek, C. Adversarial learning. In Pro-

ceedings of the eleventh ACM SIGKDD international
conference on Knowledge discovery in data mining, pp.
641–647, 2005.

Mitchell, T. M. et al. Machine learning. 1997. Burr Ridge,

IL: McGraw Hill, 45(37):870–877, 1997.

Polikar, R. Ensemble learning. In Ensemble machine learn-

ing, pp. 1–34. Springer, 2012.