Parallel & Distributed Computing

Lecture NO: 01
Introduction

Farhad M. Riaz
[email protected]

Department of Computer Science

NUML, Islamabad
Course Pre-requisites

 Programming Experience (preferably

Python/C++/Java)
 Understanding of Computer Organization
and Architecture
 Understanding of Operating System
Requirements & Grading

 Roughly
– 50 % Final Exam
– 25% Internal Evaluation
 Quiz 8 Marks
 Assignments 8 Marks
 Project 9 Marks
– 25% Mid term exam
Books

 Some good books are:

– Distributed Systems Third edition
– PRINCIPLES OF PARALLEL PROGRAMMING
– Designing and Building Parallel Programs
– Distributed and Cloud Computing
Course Project

 At the end of the semester students needs to

submit a semester project like
– Distributed computing & smart city services
– Large scale convolutional neural networks
– Distributed computing with delay tolerant network
Course Overview

 This course covers following main concepts

– Concepts of parallel and distributed computing
– Analysis and profiling of applications
– Shared memory concepts
– Distributed memory concepts
– Parallel and distributed programming (OpenMP, MPI)
– GPU based computing and programming (CUDA)
– Virtualization
– Cloud Computing, MapReduce
– Grid Computing
– Peer-to-Peer Computing
– Future trends in computing
Recommended Material
 Distributed Systems, Maarten van Steen & Andrew S. Tanenbaum, 3rd
Edition (2020), Pearson.
 Parallel Programming: Concepts and Practice, Bertil Schmidt, Jorge
Gonzalez-Dominguez, Christian Hundt, Moritz Schlarb, 1st Edition
(2018), Elsevier.
 Parallel and High-Performance Computing, Robert Robey and Yuliana
Zamora, 1st Edition (2021).
 Distributed and Cloud Computing: From Parallel Processing to the
Internet of Things, Kai Hwang, Jack Dongarra, Geoffrey Fox, 1st
Edition (2012), Elsevier.
 Multicore and GPU Programming: An Integrated Approach,
Gerassimos Barlas, 2nd Edition (2015), Elsevier.
 Parallel programming: For multicore and cluster systems. Rauber,
Thomas, and Gudula Rünger. Springer Science & Business Media,
2013.
Recent Jobs
Jobs
Research In Parallel & Distributed
Computing
Single Processor Architecture
Memory Hierarchy
5 years of Technology Advance
Productivity Gap
Pipelining
Pipelining
Multicore Trend
Application Partitioning
High-Performance Computing
(HPC)

 HPC is the use of parallel processing for running

advanced application programs efficiently, reliably
and quickly.
 It applies especially to systems that function above a
tera FLOPs (floating-point operations per second)
processing speed.
 The term HPC is occasionally used as a synonym for
supercomputing, although technically a
supercomputer is a system that performs at or near
the currently highest operational rate for computers.
High Performance Computing
GPU-accelerated Computing

 GPU-accelerated computing is the use of a graphics

processing unit (GPU) together with a CPU to
accelerate deep learning, analytics, and engineering
applications.
 Pioneered in 2007 by NVIDIA, GPU accelerators now
power energy-efficient data centers in government labs,
universities, enterprises, and small-and-medium
businesses around the world.
 They play a huge role in accelerating applications in
platforms ranging from artificial intelligence to cars,
drones, and robots.
What is GPU?

 It is a processor optimized for 2D/3D graphics, video,

visual computing, and display.
 It is highly parallel, highly multithreaded
multiprocessor optimized for visual computing.
 It provide real-time visual interaction with computed
objects via graphics images, and video.
 It serves as both a programmable graphics
processor and a scalable parallel computing
platform.
 Heterogeneous Systems: combine a GPU with a
CPU
SGI Altix Supercomputer 2300 processors
HPC System composition
Parallel Computers

 Virtually all stand-alone computers

today are parallel from hardware
perspective:
– Multiple functional units (L1 cache,
L2 cache, branch, pre-fetch,
decode, floating-point, graphics
processing (GPU), integer, etc.)
– Multiple execution units/cores
– Multiple hardware threads

IBM BG/Q Compute Chip with 18 cores (PU) and 16 L2 Cache units (L2)
Parallel Computers
 Networks connect multiple
stand-alone computers
(nodes) to make larger
parallel computer clusters.
 Parallel computer cluster
– Each compute node is a
multi-processor parallel
computer in itself
– Multiple compute nodes are
networked together with an
Infiniband network
– Special purpose nodes, also
multi-processor, are used for
other purposes
Types of Parallel and Distributed
Computing

 Parallel Computing
– Shared Memory
– Distributed Memory

 Distributed Computing
– Cluster Computing
– Grid Computing
– Cloud Computing
– Distributed Pervasive Systems
Parallel Computing
Distributed (Cluster) Computing

 Essentially a group of high-end

systems connected through a
LAN
 Homogeneous: same OS, near-
identical hardware
 Single managing node
Distributed (Grid) Computing

 Lots of nodes from everywhere

– Heterogeneous
– Dispersed across several organizations
– Can easily span a wide-area network

 To allow for collaborations, grids generally use virtual

organizations.
 In essence, this is a grouping of users (or their IDs) that will
allow for authorization on resource allocation.
Distributed (Cloud) Computing
Distributed (Pervasive) Computing

 Emerging next-generation of distributed systems in which

nodes are small, mobile, and often embedded in a larger
system, characterized by the fact that the system naturally
blends into the user’s environment.
 Three subtypes
– Ubiquitous computing systems: pervasive and
continuously present, i.e., there is a continuous
interaction between system and user.
– Mobile computing systems: pervasive, but emphasis is
on the fact that devices are inherently mobile.
– Sensor (and actuator) networks: pervasive, with
emphasis on the actual (collaborative) sensing and
actuation of the environment.
Why Use Parallel Computing?
The Real World is Massively
Parallel
 In the natural world, many
complex, interrelated events
are happening at the same
time, yet within a temporal
sequence.
 Compared to serial computing,
parallel computing is much
better suited for modeling,
simulating and understanding
complex, real world
phenomena.
 For example, imagine modeling
these serially =>
SAVE TIME AND/OR MONEY
(Main Reasons)

 In theory, throwing
more resources at a
task will shorten its
time to completion,
with potential cost
savings.
 Parallel computers
can be built from
cheap, commodity
components.
SOLVE LARGER / MORE COMPLEX
PROBLEMS (Main Reasons)
 Many problems are so large and/or complex
that it is impractical or impossible to solve
them on a single computer, especially given
limited computer memory.
 Example: Web search engines/databases
processing millions of transactions every
second
PROVIDE CONCURRENCY
(Main Reasons)

 A single compute resource can only do one

thing at a time. Multiple compute resources
can do many things simultaneously.
 Example: Collaborative Networks provide a
global venue where people from around the
world can meet and conduct work "virtually".
MAKE BETTER USE OF UNDERLYING
PARALLEL HARDWARE
(Main Reasons)

 Modern computers, even

laptops, are parallel in
architecture with multiple
processors/cores.
 Parallel software is
specifically intended for
parallel hardware with
multiple cores, threads, etc.
 In most cases, serial
programs run on modern
computers "waste" potential
computing power.

Intel Xeon processor with 6 cores and 6

L3 cache units
The Future
(Main Reasons)
 During the past 20+ years, the trends
indicated by ever faster networks,
distributed systems, and multi-
processor computer architectures
(even at the desktop level) clearly
show that parallelism is the future of
computing.
 In this same time period, there has
been a greater than 500,000x
increase in supercomputer
performance, with no end currently in
sight.
 The race is already on for Exascale
Computing!
 Exaflop = 1018 calculations per
second
That’s all for today!!