Contents

Preface ii

List of Symbols vi

Contents ix

I Fundamentals 1

1 Introduction 2
1.1 Specifications . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2 Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.3 Proving Algorithm Correctness . . . . . . . . . . . . . . . . 7
1.4 Algorithm Analysis . . . . . . . . . . . . . . . . . . . . . . . 8
1.5 Data Structures . . . . . . . . . . . . . . . . . . . . . . . . . 12
1.6 A Case Study: Maximum Subsequence Sum . . . . . . . . . 12
1.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
1.8 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
1.9 Chapter Notes . . . . . . . . . . . . . . . . . . . . . . . . . . 24

2 Proving Algorithm Correctness 25

2.1 The Basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
2.2 Handling Recursion . . . . . . . . . . . . . . . . . . . . . . . 27
2.3 Handling Iteration . . . . . . . . . . . . . . . . . . . . . . . . 29
2.4 Combining Recursion and Iteration . . . . . . . . . . . . . . 37
2.5 Mutual Recursion . . . . . . . . . . . . . . . . . . . . . . . . 45
2.6 Finding Errors . . . . . . . . . . . . . . . . . . . . . . . . . . 47
2.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
2.8 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
2.9 Chapter Notes . . . . . . . . . . . . . . . . . . . . . . . . . . 55

ix
CONTENTS x

3 Analyzing Algorithms 56
3.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
3.2 Big-O Notation . . . . . . . . . . . . . . . . . . . . . . . . . 58
3.3 Big-Ω and Big-Θ . . . . . . . . . . . . . . . . . . . . . . . . 61
3.4 Operations on Sets . . . . . . . . . . . . . . . . . . . . . . . 65
3.5 Smooth Functions and Summations . . . . . . . . . . . . . . 67
3.6 Analyzing while Loops . . . . . . . . . . . . . . . . . . . . . 71
3.7 Analyzing Recursion . . . . . . . . . . . . . . . . . . . . . . 72
3.8 Analyzing Space Usage . . . . . . . . . . . . . . . . . . . . . 80
3.9 Multiple Variables . . . . . . . . . . . . . . . . . . . . . . . . 81
3.10 Little-o and Little-ω . . . . . . . . . . . . . . . . . . . . . . . 88
3.11 * Use of Limits in Asymptotic Analysis . . . . . . . . . . . . 91
3.12 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
3.13 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
3.14 Chapter Notes . . . . . . . . . . . . . . . . . . . . . . . . . . 103

II Data Structures 105

4 Basic Techniques 106

4.1 Stacks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
4.2 A Simple Stack Implementation . . . . . . . . . . . . . . . . 109
4.3 Expandable Arrays . . . . . . . . . . . . . . . . . . . . . . . 119
4.4 The ConsList ADT . . . . . . . . . . . . . . . . . . . . . . 123
4.5 Amortized Analysis Using Potential Functions . . . . . . . . 129
4.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
4.7 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
4.8 Chapter Notes . . . . . . . . . . . . . . . . . . . . . . . . . . 145

5 Priority Queues 148

5.1 Sorted Arrays . . . . . . . . . . . . . . . . . . . . . . . . . . 149
5.2 Heaps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152
5.3 Leftist Heaps . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
5.4 Skew Heaps . . . . . . . . . . . . . . . . . . . . . . . . . . . 165
5.5 Randomized Heaps . . . . . . . . . . . . . . . . . . . . . . . 169
5.6 Sorting and Binary Heaps . . . . . . . . . . . . . . . . . . . 178
5.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186
5.8 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189
5.9 Chapter Notes . . . . . . . . . . . . . . . . . . . . . . . . . . 194
CONTENTS xi

6 Storage/Retrieval I: Ordered Keys 195

6.1 Binary Search Trees . . . . . . . . . . . . . . . . . . . . . . . 198
6.2 AVL Trees . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205
6.3 Splay Trees . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214
6.4 Skip Lists . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223
6.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237
6.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 238
6.7 Chapter Notes . . . . . . . . . . . . . . . . . . . . . . . . . . 242

7 Storage/Retrieval II: Unordered Keys 244

7.1 Arrays with Virtual Initialization . . . . . . . . . . . . . . . 244
7.2 Hashing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 248
7.3 Deterministic Hash Functions . . . . . . . . . . . . . . . . . 254
7.4 Universal Hashing . . . . . . . . . . . . . . . . . . . . . . . . 260
7.5 Number Theoretic Universal Hash Families . . . . . . . . . . 265
7.6 Perfect Hashing . . . . . . . . . . . . . . . . . . . . . . . . . 274
7.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 284
7.8 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 285
7.9 Chapter Notes . . . . . . . . . . . . . . . . . . . . . . . . . . 288

8 Disjoint Sets 289

8.1 Using DisjointSets in Scheduling . . . . . . . . . . . . . . 290
8.2 A Tree-Based Implementation . . . . . . . . . . . . . . . . . 291
8.3 A Short Tree Implementation . . . . . . . . . . . . . . . . . 293
8.4 * Path Compression . . . . . . . . . . . . . . . . . . . . . . . 295
8.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 304
8.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305
8.7 Chapter Notes . . . . . . . . . . . . . . . . . . . . . . . . . . 308

9 Graphs 309
9.1 Universal Sink Detection . . . . . . . . . . . . . . . . . . . . 311
9.2 Topological Sort . . . . . . . . . . . . . . . . . . . . . . . . . 313
9.3 Adjacency Matrix Implementation . . . . . . . . . . . . . . . 316
9.4 Adjacency List Implementation . . . . . . . . . . . . . . . . 319
9.5 Multigraphs . . . . . . . . . . . . . . . . . . . . . . . . . . . 323
9.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 327
9.7 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 327
9.8 Chapter Notes . . . . . . . . . . . . . . . . . . . . . . . . . . 330
CONTENTS xii

III Algorithm Design Techniques 331

10 Divide and Conquer 332

10.1 Polynomial Multiplication . . . . . . . . . . . . . . . . . . . 333
10.2 Merge Sort . . . . . . . . . . . . . . . . . . . . . . . . . . . . 336
10.3 Quick Sort . . . . . . . . . . . . . . . . . . . . . . . . . . . . 337
10.4 Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 343
10.5 Integer Division . . . . . . . . . . . . . . . . . . . . . . . . . 350
10.6 * Newton’s Method . . . . . . . . . . . . . . . . . . . . . . . 357
10.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 365
10.8 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 365
10.9 Chapter Notes . . . . . . . . . . . . . . . . . . . . . . . . . . 372

11 Optimization I: Greedy Algorithms 374

11.1 Job Scheduling . . . . . . . . . . . . . . . . . . . . . . . . . . 375
11.2 Minimum-Cost Spanning Trees . . . . . . . . . . . . . . . . . 377
11.3 Single-Source Shortest Paths . . . . . . . . . . . . . . . . . . 381
11.4 Huffman Codes . . . . . . . . . . . . . . . . . . . . . . . . . 383
11.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 388
11.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 388
11.7 Chapter Notes . . . . . . . . . . . . . . . . . . . . . . . . . . 391

12 Optimization II: Dynamic Programming 392

12.1 Making Change . . . . . . . . . . . . . . . . . . . . . . . . . 392
12.2 Chained Matrix Multiplication . . . . . . . . . . . . . . . . . 397
12.3 All-Pairs Shortest Paths . . . . . . . . . . . . . . . . . . . . 398
12.4 The Knapsack Problem . . . . . . . . . . . . . . . . . . . . . 401
12.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 403
12.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 404
12.7 Chapter Notes . . . . . . . . . . . . . . . . . . . . . . . . . . 412

IV Common Reduction Targets 413

13 Depth-First Search 414

13.1 Ancestry in Rooted Trees . . . . . . . . . . . . . . . . . . . . 415
13.2 Reachability in a Graph . . . . . . . . . . . . . . . . . . . . 420
13.3 A Generic Depth-First Search . . . . . . . . . . . . . . . . . 424
13.4 Articulation Points . . . . . . . . . . . . . . . . . . . . . . . 429
13.5 Topological Sort Revisited . . . . . . . . . . . . . . . . . . . 434
CONTENTS xiii

13.6 Strongly Connected Components . . . . . . . . . . . . . . . 437

13.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 440
13.8 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 440
13.9 Chapter Notes . . . . . . . . . . . . . . . . . . . . . . . . . . 444

14 Network Flow and Matching 445

14.1 The Ford-Fulkerson Algorithm . . . . . . . . . . . . . . . . . 447
14.2 The Edmonds-Karp Algorithm . . . . . . . . . . . . . . . . . 453
14.3 Bipartite Matching . . . . . . . . . . . . . . . . . . . . . . . 455
14.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 465
14.5 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 465
14.6 Chapter Notes . . . . . . . . . . . . . . . . . . . . . . . . . . 468

15 * The Fast Fourier Transform 469

15.1 Convolutions . . . . . . . . . . . . . . . . . . . . . . . . . . . 469
15.2 Commutative Rings . . . . . . . . . . . . . . . . . . . . . . . 481
15.3 Integer Multiplication . . . . . . . . . . . . . . . . . . . . . . 485
15.4 The Schönhage-Strassen Algorithm . . . . . . . . . . . . . . 497
15.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 504
15.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 504
15.7 Chapter Notes . . . . . . . . . . . . . . . . . . . . . . . . . . 506

V Intractable Problems 507

16 N P-Completeness 508
16.1 Boolean Satisfiability . . . . . . . . . . . . . . . . . . . . . . 508
16.2 The Set P . . . . . . . . . . . . . . . . . . . . . . . . . . . . 511
16.3 The Set N P . . . . . . . . . . . . . . . . . . . . . . . . . . . 513
16.4 Restricted Satisfiability Problems . . . . . . . . . . . . . . . 517
16.5 Vertex Cover and Independent Set . . . . . . . . . . . . . . . 523
16.6 3-Dimensional Matching . . . . . . . . . . . . . . . . . . . . 527
16.7 Partitioning and Strong N P-Completeness . . . . . . . . . . 532
16.8 Proof of Cook’s Theorem . . . . . . . . . . . . . . . . . . . . 539
16.9 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 553
16.10 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 554
16.11 Chapter Notes . . . . . . . . . . . . . . . . . . . . . . . . . . 559
CONTENTS xiv

17 Approximation Algorithms 561

17.1 Polynomial Turing Reducibility . . . . . . . . . . . . . . . . 561
17.2 Knapsack . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 563
17.3 Bin Packing . . . . . . . . . . . . . . . . . . . . . . . . . . . 570
17.4 The Traveling Salesperson Problem . . . . . . . . . . . . . . 574
17.5 The Maximum Cut and Minimum Cluster Problems . . . . . 579
17.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 584
17.7 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 585
17.8 Chapter Notes . . . . . . . . . . . . . . . . . . . . . . . . . . 586

Bibliography 588

Index 597
Part I

Fundamentals

1
Chapter 1

Introduction

A large software system consists of many components, each designed to

perform a specific task. The overall quality of a given system is determined
largely by how well these components function individually and interact
with each other and the external environment. The focus of this book is
on individual components — algorithms and data structures — that can be
used in a wide variety of software systems. In this chapter, we will present
an overview of what these components are and how we will study them
throughout the remainder of the book. In the course of this overview, we
will illustrate a top-down approach for thinking about both algorithms and
data structures. This approach will provide an essential framework for both
designing and understanding algorithms and data structures.

1.1 Specifications
Before we can design or analyze any software component — including an
algorithm or a data structure — we must first know what it is supposed
to accomplish. A formal statement of what a software component is meant
to accomplish is called a specification. Here, we will discuss specifically the
specification of an algorithm. The specification of a data structure is similar,
but a bit more involved. For this reason, we will wait until Chapter 4 to
discuss the specification of data structures in detail.
Suppose, for example, that we wish to find the kth smallest element
of an array of n numbers. Thus, if k = 1, we are looking for the smallest
element in the array, or if k = n, we are looking for the largest. We will refer
to this problem as the selection problem. Even for such a seemingly simple
problem, there is a potential for ambiguity if we are not careful to state the

2
CHAPTER 1. INTRODUCTION 3

problem precisely. For example, what do we expect from the algorithm if k

is larger than n? Do we allow the algorithm to change the array? Do all of
the elements in the array need to be different? If not, what exactly do we
mean by the kth smallest?
Specifically, we need to state the following:

• a precondition, which is a statement of the assumptions we make about

the input to the algorithm and the environment in which it will be
executed; and

• a postcondition, which is a statement of the required result of executing

the algorithm, assuming the precondition is satisfied.

The precondition and postcondition, together with a function header giving

a name and parameter list, constitute the specification of the algorithm. We
say that the algorithm meets its specification if the postcondition is satisfied
whenever the precondition is satisfied. Note that we guarantee nothing
about the algorithm if its precondition is not satisfied.
The selection problem will have two parameters, an array A and a posi-
tive integer k. The precondition should specify what we are assuming about
these parameters. For this problem, we will assume that the first element of
A is indexed by 1 and the last element is indexed by some natural number
(i.e., nonnegative integer) n; hence, we describe the array more precisely
using the notation A[1..n]. We will assume that each element of A[1..n] is We adopt the
convention that
a number; hence, our precondition must include this requirement. Further- if the last index
more, we will not require the algorithm to verify that k is within a proper is smaller than
range; hence, the precondition must require that 1 ≤ k ≤ n, where k ∈ N the first (e.g.,
A[1..0]), then
(N is the mathematical notation for the set of natural numbers). the array has no
Now let us consider how we might specify a postcondition. We need elements.
to come up with a precise definition of the kth smallest element of A[1..n].
Consider first the simpler case in which all elements of A[1..n] are distinct.
In this case, we would need to return the element A[i] such that exactly k
elements of A[1..n] are less than or equal to A[i]. However, this definition
of the kth smallest element might be meaningless when A[1..n] contains
duplicate entries. Suppose, for example, that A[1..2] contains two 0s and
that k = 1. There is then no element A[i] such that exactly 1 element of
A[1..2] is less than or equal to A[i].
To better understand the case in which elements of A might be dupli-
cated, let us consider a specific example. Let A[1..8] = h1, 5, 6, 9, 9, 9, 9, 10i,
and let k = 5. An argument could be made that 10 is the kth smallest be-
cause there are exactly k distinct values less than or equal to 10. However,
CHAPTER 1. INTRODUCTION 4

Figure 1.1 Specification for the selection problem

Precondition: A[1..n] is an array of Numbers, 1 ≤ k ≤ n, k and n are

Nats.
Postcondition: Returns the value x in A[1..n] such that fewer than k
elements of A[1..n] are strictly less than x, and at least k elements of A[1..n]
are less than or equal to x. The elements of A[1..n] may be permuted.
Select(A[1..n], k)

if we were to adopt this definition, there would be no kth smallest element

for k = 6. Because A is sorted, it would be better to conclude that the kth
smallest is A[k] = 9. Note that all elements strictly less than 9 are in A[1..4];
i.e., there are strictly fewer than k elements less than the kth smallest. Fur-
thermore, rearranging A would not change this fact. Likewise, observe that
all the elements in A[1..5] are less than or equal to 9; i.e., there are at least k
elements less than or equal to the kth smallest. Again, rearranging A would
not change this fact.
The above example suggests that the proper definition of the kth smallest
element of an array A[1..n] is the value x such that
• there are fewer than k elements A[i] < x; and

• there are at least k elements A[i] ≤ x.

It is possible to show, though we will not do so here, that for any array A[1..n]
and any positive integer k ≤ n, there is exactly one value x satisfying both
of the above conditions. We will therefore adopt this definition of the kth
smallest element.
The complete specification for the selection problem is shown in Figure
1.1. To express the data types of n and the elements of A, we use Nat to
denote the natural number type and Number to denote the number type.
Note that Nat is a subtype of Number — every Nat is also a Number.
In order to place fewer constraints on the algorithm, we have included in
the postcondition a statement that the elements of A[1..n] may be permuted
(i.e., rearranged). In order for a specification to be precise, the postcondi-
tion must state when side-effects such as this may occur. In order to keep
specifications from becoming overly wordy, we will adopt the convention
that no values may be changed unless the postcondition explicitly allows
the change.
CHAPTER 1. INTRODUCTION 5

1.2 Algorithms
Once we have a specification, we need to produce an algorithm to implement
that specification. This algorithm is a precise statement of the computa-
tional steps taken to produce the results required by the specification. An
algorithm differs from a program in that it is usually not specified in a pro-
gramming language. In this book, we describe algorithms using a notation
that is precise enough to be implemented as a programming language, but
which is designed to be read by humans.
A straightforward approach to solving the selection problem is as follows:

1. Sort the array in nondecreasing order.

2. Return the kth element.

If we already know how to sort, then we have solved our problem; otherwise,
we must come up with a sorting algorithm. By using sorting to solve the
selection problem, we say that we have reduced the selection problem to the
sorting problem.
Solving a problem by reducing it to one or more simpler problems is the
essence of the top-down approach to designing algorithms. One advantage
to this approach is that it allows us to abstract away certain details so that
we can focus on the main steps of the algorithm. In this case, we have a
selection algorithm, but our algorithm requires a sorting algorithm before
it can be fully implemented. This abstraction facilitates understanding of
the algorithm at a high level. Specifically, if we know what is accomplished
by sorting — but not necessarily how it is accomplished — then because
the selection algorithm consists of very little else, we can readily understand
what it does.
When we reduce the selection problem to the sorting problem, we need
a specification for sorting as well. For this problem, the precondition will be
that A[1..n] is an array of Numbers, where n ∈ N. Our postcondition will be
that A[1..n] is a permutation of its initial values such that for 1 ≤ i < j ≤ n,
A[i] ≤ A[j] — i.e., that A[1..n] contains its initial values in nondecreasing
order. Our selection algorithm is given in Figure 1.2. Note that Sort is
only specified — its algorithm is not provided.
Let us now refine the SimpleSelect algorithm of Figure 1.2 by design-
ing a sorting algorithm. We will reduce the sorting problem to the problem
of inserting an element into a sorted array. In order to complete the re-
duction, we need to have a sorted array in which to insert. We have thus
returned to our original problem. We can break this circularity, however,
CHAPTER 1. INTRODUCTION 6

Figure 1.2 An algorithm implementing the specification of Figure 1.1

SimpleSelect(A[1..n], k)
Sort(A[1..n])
return A[k]

Precondition: A[1..n] is an array of Numbers, n is a Nat.

Postcondition: A[1..n] is a permutation of its initial values such that for
1 ≤ i < j ≤ n, A[i] ≤ A[j].
Sort(A[1..n])

by using the top-down approach in a different way. Specifically, we reduce

larger instances of sorting to smaller instances. In this application of the
top-down approach, the simpler problem is actually a smaller instance of
the same problem.
The algorithm is given in Figure 1.3. Though this application of the top-
down approach may at first seem harder to understand, we can think about
it in the same way as we did for SimpleSelect. If n ≤ 1, the postcondition
for Sort(A[1..n]) (given in Figure 1.2) is clearly met. For n > 1, we use
the specification of Sort to understand that InsertSort([1..n − 1]) sorts
A[1..n − 1]. Thus, the precondition for Insert(A[1..n]) is satisfied. We
then use the specification of Insert(A[1..n]) to understand that when the
algorithm completes, A[1..n] is sorted.
The thought process outlined above might seem mysterious because it
doesn’t follow the sequence of steps in an execution of the algorithm. How-
ever, it is a much more powerful way to think about algorithms.
To complete the implementations of SimpleSelect and InsertSort,
we need an algorithm for Insert. We can again use the the top-down
approach to reduce an instance of Insert to a smaller instance of the same
problem. According to the precondition (see Figure 1.3), A[1..n−1] must be
sorted in nondecreasing order; hence, if either n = 1 or A[n] ≥ A[n − 1], the
postcondition is already satisfied. Otherwise, A[n − 1] must be the largest
element in A[1..n]. Therefore, if we swap A[n] with A[n − 1], A[1..n − 2] is
sorted in nondecreasing order and A[n] is the largest element in A[1..n]. If we
then solve the smaller instance A[1..n − 1], we will satisfy the postcondition.
The algorithm is shown in Figure 1.4. In this book, we will assume
CHAPTER 1. INTRODUCTION 7

Figure 1.3 An algorithm implementing the specification of Sort, given in

Figure 1.2

InsertSort(A[1..n])
if n > 1
InsertSort(A[1..n − 1])
Insert(A[1..n])

Precondition: A[1..n] is an array of Numbers such that n is a Nat, and

for 1 ≤ i < j ≤ n − 1, A[i] ≤ A[j].
Postcondition: A[1..n] is a permutation of its initial values such that for
1 ≤ i < j ≤ n, A[i] ≤ A[j].
Insert(A[1..n])

Figure 1.4 An algorithm implementing the specification of Insert, given

in Figure 1.3

RecursiveInsert(A[1..n])
if n > 1 and A[n] < A[n − 1]
A[n] ↔ A[n − 1]
RecursiveInsert(A[1..n − 1])

that a logical and or or is evaluated by first evaluating its first operand,

then if necessary, evaluating its second operand. Thus, in evaluating the if
statement in RecursiveInsert, if n ≤ 1, the second operand will not be
evaluated, as the value of the expression must be false. We use the notation
x ↔ y to swap the values of variables x and y.

1.3 Proving Algorithm Correctness

Once we have an algorithm, we would like some assurance that it meets
its specification. We have argued somewhat informally that the three algo-
rithms, SimpleSelect, InsertSort, and RecursiveInsert, meet their
CHAPTER 1. INTRODUCTION 8

respective specifications; however, these arguments may not convince every-

one. For example, it might not be clear that it is valid to assume that the
recursive call to InsertSort in Figure 1.3 meets its specification. After all,
the whole point of that discussion was to argue that InsertSort meets its
specification. That argument might therefore seem circular.
In the next chapter, we will show how to prove formally that an algo-
rithm meets its specification. These proofs will rest on solid mathematical
foundations, so that a careful application of the techniques should give us
confidence that a given algorithm is, indeed, correct. In particular, we will
formally justify the reasoning used above — namely, that we can assume
that a recursive call meets its specification, provided its input is of a smaller
size than that of the original call, where the size is some natural number.
The ability to prove algorithm correctness is quite possibly the most
underrated skill in the entire discipline of computing. First, knowing how
to prove algorithm correctness also helps us in the design of algorithms.
Specifically, once we understand the mechanics of correctness proofs, we can
design the algorithm with a proof of correctness in mind. This approach
makes designing correct algorithms much easier. Second, the exercise of
working through a correctness proof — or even sketching such a proof —
often uncovers subtle errors that would be difficult to find with testing alone.
Third, this ability brings with it a capacity to understand specific algorithms
on a much deeper level. Thus, the ability to prove algorithm correctness is
a powerful tool for designing and understanding algorithms. Because these
activities are closely related to programming, this ability greatly enhances
programming abilities as well.
The proof techniques we will introduce fit nicely with the top-down
approach to algorithm design. As a result, the top-down approach itself
becomes an even more powerful tool for designing and understanding algo-
rithms.

1.4 Algorithm Analysis

Once we have a correct algorithm, we need to be able to evaluate how well
it does its job. This evaluation essentially boils down to an analysis of
the resource usage of the algorithm, where the resources might be time or
memory, for example. In Chapter 3, we will present tools for mathematically
analyzing such resource usage. For now, let us discuss this resource usage
less formally.
Consider the InsertSort algorithm given in Figures 1.3 and 1.4, for ex-
CHAPTER 1. INTRODUCTION 9

ample. If we were to implement this algorithm in a programming language

and run it on a variety of examples, we would discover that for sufficiently
large inputs — typically a few thousand elements — the program will ter-
minate due to a stack overflow. This problem is not, strictly speaking, an
error in the algorithm, but instead is a reflection of the fact that it uses the
machine’s runtime stack inefficiently. Using the analysis tools of Chapter 3,
it is possible to show that this algorithm’s stack usage is linear in the size of
the array. Because the runtime stack is usually much smaller than the total
memory available, we usually would like the stack usage to be bounded by
a slow-growing function of the size of the input — a logarithmic function,
for example. Thus, if we were to do the analysis prior to implementing the
algorithm, we could uncover the inefficiency earlier in the process.
Having uncovered an inefficiency, we would like to eliminate it. The
runtime stack is used when performing function calls. Specifically, each
time a function call is made, information is placed onto the stack. When
the function returns, this information is removed. As a result, when an
algorithm uses the runtime stack inefficiently, the culprit is almost always
recursion. This in not to say that recursion always uses the stack inefficiently
— indeed, we will see many efficient recursive algorithms in this book. The
analysis tools of Chapter 3 will help us to determine when recursion is or
is not used efficiently. For now, however, we will focus on removing the
recursion.
It turns out that while recursion is the most obvious way to implement an
algorithm designed using the top-down approach, it is by no means the only
way. One alternative is to implement the top-down solution in a bottom-
up way. The top-down solution for InsertSort is to reduce a large in-
stance to a smaller instance and an instance of Insert by first sorting the
smaller instance, then applying Insert. We can apply this same idea in
a bottom-up fashion by observing that the first element of any nonempty
array is necessarily sorted, then extending the sorted portion of the ar-
ray by applying Insert. In other words, we repeatedly apply Insert to
A[1..2], A[1..3], . . . , A[1..n]. This can be done using a loop.
The bottom-up implementation works well for InsertSort because the
recursive call is essentially the first step of the computation. However, the
recursive call in RecursiveInsert is necessarily the last step. Fortunately,
there is a fairly straightforward way of removing the recursion from this
type of algorithm, as well. When a reduction has the form that the solution
to the simpler problem solves the original problem, we call this reduction a
transformation. When we transform a large instance to a smaller instance of
the same problem, the natural recursive algorithm is tail recursive, meaning
CHAPTER 1. INTRODUCTION 10

Figure 1.5 Translation of a typical tail recursive algorithm to an iterative

algorithm

TailRecursiveAlgorithm(p1 , . . . , pk )
if hBooleanConditioni
hBaseCasei
else
hRecursiveCaseComputationi
TailRecursiveAlgorithm(q1 , . . . , qk )

IterativeAlgorithm(p1 , . . . , pk )
while not hBooleanConditioni
hRecursiveCaseComputationi
p 1 ← q 1 ; . . . ; pk ← q k
hBaseCasei

that the last step is a recursive call.

Figure 1.5 illustrates how a typical tail recursive algorithm can be con-
verted to an iterative algorithm. The loop iterates as long as the condition
indicating the base case (i.e., the end of the recursion) is false. Each itera-
tion performs all the computation in the recursive case except the recursive
call. To simulate the recursive call, the formal parameters p1 , . . . , pk are
given the values of their corresponding actual parameters, q1 , . . . , qk , and
the loop continues (a statement of the form “x ← y” assigns the value of y
to the variable x). Once the condition indicating the base case is true, the
loop terminates, the base case is executed, and the algorithm terminates.
Figure 1.6 shows the result of eliminating the tail recursion from Recur-
siveInsert. Because RecursiveInsert is structured in a slightly different
way from what is shown in Figure 1.5, we could not apply this translation
verbatim. The tail recursion occurs, not in the else part, but in the if
part, of the if statement in RecursiveInsert. For this reason, we did not
negate the condition when forming the while loop. The base case is then
the empty else part of the if statement. Because there is no code in the
base case, there is nothing to place after the loop. In order to avoid chang-
ing the value of n, we have copied its value to j, and used j in place of n
throughout the algorithm. Furthermore, because the meaning of the state-
CHAPTER 1. INTRODUCTION 11

Figure 1.6 Iterative algorithm implementing the specification of Insert,

given in Figure 1.3

IterativeInsert(A[1..n])
j←n
// Invariant: A[1..n] is a permutation of its original values such that
// for 1 ≤ k < k ′ ≤ n, if k ′ 6= j, then A[k] ≤ A[k ′ ].
while j > 1 and A[j] < A[j − 1]
A[j] ↔ A[j − 1]; j ← j − 1

Figure 1.7 Insertion sort implementation of Sort, specified in Figure 1.1

InsertionSort(A[1..n])
// Invariant: A[1..n] is a permutation of its original values
// such that A[1..i − 1] is in nondecreasing order.
for i ← 1 to n
j←i
// Invariant: A[1..n] is a permutation of its original values
// such that for 1 ≤ k < k ′ ≤ i, if k ′ 6= j, then A[k] ≤ A[k ′ ].
while j > 1 and A[j] < A[j − 1]
A[j] ↔ A[j − 1]; j ← j − 1

ment “A[1..j] ← A[1..j − 1]” is not clear, we instead simulated the recursion
by the statement “j ← j − 1”.
Prior to the loop in IterativeInsert, we have given a loop invariant,
which is a statement that must be true at the beginning and end of each We consider each
test of a loop
iteration of the loop. Loop invariants will be an essential part of the cor- exit condition to
rectness proof techniques that we will introduce in the next chapter. For mark the
now, however, we will use them to help us to keep track of what the loop is beginning/end of
an iteration.
doing.
The resulting sorting algorithm, commonly known as insertion sort, is
shown in Figure 1.7. Besides removing the two recursive calls, we have also
combined the two functions into one. Also, we have started the for loop at
1, rather than at 2, as our earlier discussion suggested. As far as correctness
CHAPTER 1. INTRODUCTION 12

goes, there is no difference which starting point we use, as the inner loop
will not iterate when i = 1; however, it turns out that the correctness proof,
which we will present in the next chapter, is simpler if we begin the loop at
1. Furthermore, the impact on performance is minimal.
While analysis techniques can be applied to analyze stack usage, a far
more common application of these techniques is to analyze running time. For
example, we can use these techniques to show that while InsertionSort is
not, in general, a very efficient sorting algorithm, there are important cases
in which it is a very good choice. In Section 1.6, we will present a case study
that demonstrates the practical utility of running time analysis.

1.5 Data Structures

One of the major factors influencing the performance of an algorithm is
the way the data items it manipulates are organized. For this reason, it is
essential that we include data structures in our study of algorithms. The
themes that we have discussed up to this point all apply to data structures,
but in a somewhat different way.
For example, a specification of a data structure must certainly include
preconditions and postconditions for the operations on the structure. In
addition, the specification must in some way describe the information rep-
resented by the structure, and how operations on the structure affect that
information. As a result, proofs of correctness must take into account this
additional detail.
While the analysis techniques of Chapter 3 apply directly to operations
on data structures, it will be necessary to introduce a new technique that
analyzes sequences of operations on a data structure. Furthermore, even
the language we are using to express algorithms will need to be enriched in
order to allow design and analysis of data structures. We will present all of
these extensions in Chapter 4.

1.6 A Case Study: Maximum Subsequence Sum

We conclude this chapter by presenting a case study that illustrates the
practical impact of algorithm efficiency. The inefficiencies that we will ad-
dress all can be discovered using the analysis techniques of Chapter 3. This
case study will also demonstrate the usefulness of the top-down approach in
designing significantly more efficient algorithms.
CHAPTER 1. INTRODUCTION 13

Figure 1.8 The subsequence with maximum sum may begin and end any-
where in the array, but must be contiguous

0 n−1

Figure 1.9 Specification for the maximum subsequence sum problem

Precondition: A[0..n − 1] is an array of Numbers, n is a Nat.

Postcondition: Returns the maximum subsequence sum of A.
MaxSum(A[0..n − 1])

Let us consider the problem of computing the maximum sum of any

contiguous sequence in an array of numbers (see Figure 1.8). The numbers
in the array may be positive, negative, or zero, and we consider the empty
sequence to have a sum of 0. More formally, given an array A[0..n − 1] of
numbers, where n ∈ N, the maximum subsequence sum of A is defined to be
( j−1 )
X
max A[k] | 0 ≤ i ≤ j ≤ n .
k=i

Note that when i = j, the sum has a beginning index of i and an ending
index of i − 1. By convention, we always write summations so that the
index (k in this case) increases from its initial value (i) to its final value
(j − 1). As a result of this convention, whenever the final value is less than
the initial value, the summation contains no elements. Again by convention,
such an empty summation is defined to have a value of 0. Thus, in the above Similar
conventions hold
definition, we are including the empty sequence and assuming its sum is 0. for products,
The specification for this problem is given in Figure 1.9. Note that according except that an
to this specification, the values in A[0..n − 1] may not be modified. empty product is
assumed to have
a value of 1.
Example 1.1 Suppose A[0..5] = h−1, 3, −2, 7, −9, 7i. Then the subsequence
A[1..3] = h3, −2, 7i has a sum of 8. By exhaustively checking all other con-
CHAPTER 1. INTRODUCTION 14

Figure 1.10 A simple algorithm implementing the specification given in

Figure 1.9

MaxSumIter(A[0..n − 1])
m←0
for i ← 0 to n
for j ← i to n
sum ← 0
for k ← i to j − 1
sum ← sum + A[k]
m ← Max(m, sum)
return m

tiguous subsequences, we can verify that this is, in fact, the maximum. For
example, the subsequence A[1..5] has a sum of 6.

Example 1.2 Suppose A[0..3] = h−3, −4, −1, −5i. Then all nonempty
subsequences have negative sums. However, any empty subsequence (e.g.,
A[0..−1]) by definition has a sum of 0. The maximum subsequence sum of
this array is therefore 0.
We can easily obtain an algorithm for this problem by translating the
definition of a maximum subsequence sum directly into an iterative solution.
The result is shown in Figure 1.10. By applying the analysis techniques
of Chapter 3, it can be shown that the running time of this algorithm is
proportional to n3 , where n is the size of the array.
In order to illustrate the practical ramifications of this analysis, we im-
plemented this algorithm in the JavaTM programming language and ran it
on a personal computer using randomly generated data sets of size 2k for
various values of k. On a data set of size 210 = 1024, MaxSumIter required
less than half a second, which seems reasonably fast. However, as the size
of the array increased, the running time degraded quickly:

• 211 = 2048 elements: 2.6 seconds

• 212 = 4096 elements: 20 seconds

• 213 = 8192 elements: 2 minutes, 40 seconds

CHAPTER 1. INTRODUCTION 15

Figure 1.11 An algorithm for maximum subsequence sum (specified in

Figure 1.9), optimized by removing an unnecessary loop from MaxSumIter
(Figure 1.10)

MaxSumOpt(A[0..n − 1])
m←0
for i ← 0 to n − 1
sum ← 0
for k ← i to n − 1
sum ← sum + A[k]
m ← Max(m, sum)
return m

• 214 = 16,384 elements: over 21 minutes

Notice that as the size of the array doubles, the running time increases
by roughly a factor of 8. This is not surprising if we realize that the running
time should be cn3 for some c, and c(2n)3 = 8cn3 . We can therefore estimate
that for a data set of size 217 = 131,072, the running time should be 83 = 512
times as long as for 214 . This running time is over a week!
If we want to solve this problem on large data sets, we clearly would
like to improve the running time. A careful examination of Figure 1.10
reveals some redundant computation. Specifically, the inner loop computes
sums of successively longer subsequences. Much work can be saved if we
compute sums of successive sequences by simply adding the next element
to the preceding sum. Furthermore, a small optimization can be made by
running the outer loop to n − 1, as the inner loop would not execute on this
iteration. The result of this optimization is shown in Figure 1.11.
It turns out that this algorithm has a running time proportional to n2 ,
which is an improvement over n3 . To show how significant this improvement
is, we again coded this algorithm and timed it for arrays of various sizes.
The difference was dramatic — for an input of size 217 , which we estimated
would require a week for MaxSumIter to process, the running time of
MaxSumOpt was only 33 seconds. However, because c(2n)2 = 4cn2 , we
would expect the running time to increase by a factor of 4 each time the
array size is doubled. This behavior proved to be true, as a data set of size
220 = 1,048,576 required almost 40 minutes. Extrapolating this behavior,
CHAPTER 1. INTRODUCTION 16

Figure 1.12 The suffix with maximum sum may begin anywhere in the
array, but must end at the end of the array

0 n−1

we would expect a data set of size 224 = 16,777,216 to require over a week.
(MaxSumIter would require over 43,000 years to process a data set of this
size.)
While MaxSumOpt gives a dramatic speedup over MaxSumIter, we
would like further improvement if we wish to solve very large problem in-
stances. Note that neither MaxSumIter nor MaxSumOpt was designed
using the top-down approach. Let us therefore consider how we might solve
the problem in a top-down way. For ease of presentation let us refer to the
maximum subsequence sum of A[0..n−1] as sn . Suppose we can obtain sn−1
(i.e., the maximum subsequence sum of A[0..n − 2]) for n > 0. Then in order
to compute the overall maximum subsequence sum we need the maximum
of sn−1 and all of the sums of subsequences A[i..n − 1] for 0 ≤ i ≤ n. Thus,
we need to solve another problem, that of finding the maximum suffix sum
(see Figure 1.12), which we define to be
(n−1 )
X
max A[k] | 0 ≤ i ≤ n .
k=i

In other words, the maximum suffix sum is the maximum sum that we
can obtain by starting at any index i, where 0 ≤ i ≤ n, and adding together
all elements from index i up to index n − 1. (Note that by taking i = n, we
include the empty sequence in this maximum.) We then have a top-down
solution for computing the maximum subsequence sum:
(
0 if n = 0
sn = (1.1)
max(sn−1 , tn ) if n > 0,

where tn is the maximum suffix sum of A[0..n − 1].

CHAPTER 1. INTRODUCTION 17

Figure 1.13 A top-down solution for maximum subsequence sum, specified

in Figure 1.9

MaxSumTD(A[0..n − 1])
if n = 0
return 0
else
return Max(MaxSumTD(A[0..n−2]), MaxSuffixTD(A[0..n−1]))

Precondition: A[0..n − 1] is an array of Numbers, n is a Nat.

Postcondition: Returns the maximum suffix sum of A.
MaxSuffixTD(A[0..n − 1])
if n = 0
return 0
else
return Max(0, A[n − 1] + MaxSuffixTD(A[0..n − 2]))

Let us consider how to compute the maximum suffix sum tn using the
top-down approach. Observe that every suffix of A[0..n − 1] — except the
empty suffix — ends with A[n − 1]. If we remove A[n − 1] from all of these
suffixes, we obtain all of the suffixes of A[0..n−2]. Thus, tn−1 +A[n−1] = tn
unless tn = 0. We therefore have

tn = max(0, A[n − 1] + tn−1 ). (1.2)

Using (1.1) and (1.2), we obtain the recursive solution given in Figure 1.13.
Note that we have combined the algorithm for MaxSuffixTD with its
specification.
Unfortunately, an analysis of this algorithm shows that it also has a run-
ning time proportional to n2 . What is worse, however, is that an analysis
of its stack usage reveals that it is linear in n. Indeed, the program im-
plementing this algorithm threw a StackOverflowError on an input of
size 212 .
While these results are disappointing, we at least have some techniques
for improving the stack usage. Note that in both MaxSumTD and Max-
SuffixTD, the recursive calls don’t depend on any of the rest of the compu-
tation; hence, we should be able to implement both algorithms in a bottom-
CHAPTER 1. INTRODUCTION 18

Figure 1.14 A bottom-up calculation of the maximum subsequence sum,

specified in Figure 1.9

MaxSumBU(A[0..n − 1])
m ← 0; msuf ← 0
// Invariant: m is the maximum subsequence sum of A[0..i − 1],
// msuf is the maximum suffix sum for A[0..i − 1]
for i ← 0 to n − 1
msuf ← Max(0, msuf + A[i])
m ← Max(m, msuf )
return m

up fashion in order to remove the recursion. Furthermore, we can simplify

the resulting code with the realization that once ti is computed (using (1.2)),
we can immediately compute si using (1.1). Thus, we can compute both val-
ues within a single loop. The result is shown in Figure 1.14.
Because this algorithm uses no recursion, its stack usage is fixed (i.e., it
does not grow as n increases). Furthermore, an analysis of its running time
reveals that it runs in a time linear in n. The program implementing this
algorithm bears out that this is a significant improvement — this program
used less than a tenth of a second on an array of size 223 = 8,388,608.
Furthermore, using the fact that the running time is linear, we estimate
that it would require less than 12 seconds to process an array of size 230 =
1,073,741,824. (Unfortunately, such an array would require more memory
than was available on the PC we used.) By comparison, we estimate that
on an array of this size, MaxSumOpt would require nearly 80 years, and
that MaxSumIter would require over 11 billion years!
The running times of the JavaTM implementations of all four algorithms
are plotted on the graph in Figure 1.15. The code used to generate these
results can be found on the textbook’s web page.
Note that the performance improvement of MaxSumBU over either
MaxSumIter or MaxSumOpt is better than what we can expect to obtain
simply by using faster hardware. Specifically, to achieve the running time
we measured for MaxSumBU with MaxSumOpt would require hardware
over 100,000 times faster for an array of size 220 . Even then, we would be
able to process much larger inputs with MaxSumBU. MaxSumBU also
has the advantage of being simpler than the other algorithms. Furthermore,
CHAPTER 1. INTRODUCTION 19

Figure 1.15 Experimental comparison of maximum subsequence sum algo-

rithms

104

103
Time in seconds

102 MaxSumIter
MaxSumOpt
MaxSumTD
101 MaxSumBU

100

10−1

10−2 10
2 212 214 216 218 220 222 224
Array size

because it makes only one pass through the input, it does not require that
the data be stored in an array. Rather, it can simply process the data as it
reads each element. As a result, it can be used for very large data sets that
might not fit into main memory.
Thus, we can see the importance of efficiency in the design of algorithms.
Furthermore, we don’t have to code the algorithm and test it to see how effi-
cient it is. Instead, we can get a fairly good idea of its efficiency by analyzing
it using the techniques presented in Chapter 3. Finally, understanding these
analysis techniques will help us to know where to look to improve algorithm
efficiency.

1.7 Summary
The study of algorithms encompasses several facets. First, before an algo-
rithm or data structure can be considered, a specification of the requirements
must be made. Having a specification, we can then design the algorithm or
data structure with a proof of correctness in mind. Once we have convinced
CHAPTER 1. INTRODUCTION 20

ourselves that our solution is correct, we can then apply mathematical tech-
niques to analyze its resource usage. Such an analysis gives us insight into
how useful our solution might be, including cases in which it may or may
not be useful. This analysis may also point to shortcomings upon which we
might try to improve.
The top-down approach is a useful framework for designing correct, effi-
cient algorithms. Furthermore, algorithms presented in a top-down fashion
can be more easily understood. Together with the top-down approach, tech-
niques such as bottom-up implementation and elimination of tail recursion
— along with others that we will present later — give us a rich collection of
tools for algorithm design. We can think of these techniques as algorithmic
design patterns, as we use each of them in the design of a wide variety of
algorithms.
In Chapters 2 and 3, we will provide the foundations for proving al-
gorithm correctness and analyzing algorithms, respectively. In Part II, we
will examine several of the most commonly-used data structures, including
those that are frequently used by efficient algorithms. In Part III, we ex-
amine the most common approaches to algorithm design. In Part IV, we
will study several specific algorithms to which many other problems can be
reduced. Finally, in Part V, we will consider a class of problems believed
to be computationally intractable and introduce some techniques for coping
with them.

1.8 Exercises
Exercise 1.1 We wish to design an algorithm that takes an array A[0..n−1]
of numbers in nondecreasing order and a number x, and returns the location
of the first occurrence of x in A[0..n − 1], or the location at which x could
be inserted without violating the ordering if x does not occur in the array.
Give a formal specification for this problem. The algorithm shown in Figure
1.16 should meet your specification.

Exercise 1.2 Give an iterative algorithm that results from removing the
tail recursion from the algorithm shown in Figure 1.16. Your algorithm
should meet the specification described in Exercise 1.1.

Exercise 1.3 Figure 1.17 gives a recursive algorithm for computing the dot
product of two vectors, represented as arrays. Give a bottom-up implemen-
tation of this algorithm.
CHAPTER 1. INTRODUCTION 21

Figure 1.16 An algorithm satisfying the specification described in Exercise

1.1

Find(A[0..n − 1], x)
if n = 0 or A[n − 1] < x
return n
else
return Find(A[0..n − 2], x)

Figure 1.17 Algorithm for Exercise 1.3

Precondition: A[1..n] and B[1..n] are Arrays of Numbers, and n is a

Nat.
Postcondition: Returns
Xn
A[i]B[i].
i=1

DotProduct(A[1..n], B[1..n])
if n = 0
return 0
else
return DotProduct(A[1..n − 1], B[1..n − 1]) + A[n]B[n]

Exercise 1.4 Figure 1.18 gives a specification for Copy.

a. Show how to reduce Copy to a smaller instance of itself by giving Corrected
1/27/10.
a recursive algorithm. You may assume that A and B are distinct,
non-overlapping arrays. Your algorithm should contain exactly one
recursive call and no loops.

b. Convert your algorithm from part a to an iterative algorithm by either

removing tail recursion or implementing it bottom-up, whichever is
more appropriate. Explain how you did your conversion.

c. The specification of Copy does not prohibit the two arrays from shar-
ing elements, for example, Copy(A[1..n − 1], A[2..n]). Modify your
CHAPTER 1. INTRODUCTION 22

Figure 1.18 Specification of Copy

Precondition: n is a Nat.
Postcondition: For 1 ≤ i ≤ n, B[i] is modified to equal A[i].
Copy(A[1..n], B[1..n])

Figure 1.19 Specification for FindMax

Precondition: A[0..n − 1] is an array of numbers, and n is a positive Nat.

Postcondition: Returns i such that 0 ≤ i < n and for any j, 0 ≤ j < n,
A[i] ≥ A[j].
FindMax(A[0..n − 1])

algorithm from part a to handle any two arrays of the same size.
Specifically, you cannot assume that the recursive call does not change
A[1..n]. Your algorithm should contain exactly one recursive call and
no loops.

Exercise 1.5 A palindrome is a sequence of characters that is the same

when read from left to right as when read from right to left. We wish
to design an algorithm that recognizes whether an array of characters is a
palindrome.
a. Give a formal specification for this problem. Use Char to denote the
data type for a character.

b. Using the top-down approach, give an algorithm to solve this problem.

Your algorithm should contain a single recursive call.

c. Give an iterative version of your algorithm from part b by either im-

plementing it bottom-up or eliminating tail recursion, whichever is
appropriate.

Exercise 1.6 FindMax is specified in Figure 1.19.

a. Using the top-down approach, give an algorithm for FindMax. Note
that according to the specification, your algorithm may not change
CHAPTER 1. INTRODUCTION 23

Figure 1.20 Specifications of AppendToAll and SizeOf

Precondition: A[1..n] is an array of arrays and n is a Nat.

Postcondition: For 1 ≤ i ≤ n, modifies A[i] to contain an array whose
size is 1 larger than its original size, and whose elements are the same as its
original elements, with x added as its last element (i.e., x is appended to
the end of each A[i]).
AppendToAll(A[1..n], x)
Precondition: A is an array.
Postcondition: Returns the number of elements in A.
SizeOf(A)

the values in A. Your algorithm should contain exactly one recursive

call.

b. Give an iterative version of your algorithm from part a by either im-

plementing it bottom-up or eliminating tail recursion, whichever is
appropriate.

* c. Show how to reduce the sorting problem to FindMax (as specified

in part a) and a smaller instance of sorting. Use the technique of
transformation, and remove the resulting tail recursion, so that your
algorithm is iterative.

* Exercise 1.7 AppendToAll and SizeOf are specified in Figure 1.20.

a. Show how to reduce AppendToAll to Copy (specified in Figure

1.18), SizeOf, and a smaller instance of itself, by giving a recursive
algorithm. Your algorithm should contain no loops and a single re-
cursive call.

b. Convert your algorithm from part a to an iterative algorithm by either

removing tail recursion or implementing it bottom-up, whichever is
more appropriate. You do not need to provide algorithms for either
Copy or SizeOf.
CHAPTER 1. INTRODUCTION 24

Figure 1.21 Specification for AllSubsets

Precondition: A[1..n] is an Array of distinct elements, and n is a Nat.

Postcondition: Returns an Array P [1..2n ] such that for 1 ≤ i ≤ 2n ,
P [i] is an Array containing exactly one occurrence of each element of some
subset of the elements in A[1..n], where each subset of A[1..n] is represented
exactly once in P [1..2n ].
AllSubsets(A[1..n])

* Exercise 1.8 AllSubsets is specified in Figure 1.21.

a. Show how to reduce to AllSubsets to Copy (specified in Figure

1.18), AppendToAll (specified in Figure 1.20), SizeOf (specified in
Figure 1.20), and a smaller instance of itself. Note that according to
the specification, your algorithm may not change A[1..n]. Your algo-
rithm should contain exactly one recursive call and no loops. (When
calling Copy, your first array index does not need to be 1.)

b. Give an iterative version of your algorithm from part a by either im-

plementing it bottom-up or eliminating tail recursion, whichever is
appropriate. You do not need to provide algorithms for Copy, Ap-
pendToAll, or SizeOf.

1.9 Chapter Notes

The elements of top-down software design were introduced by Dijkstra [28]
and Wirth [114] in the late 1960s and early 1970s. As software systems
grew, however, these notions were found to be insufficient to cope with the
sheer size of large projects. As a result, they were eventually superseded by
object-oriented design and programming. The study of algorithms, however,
does not focus on large software systems, but on small components. Con-
sequently, a top-down approach provides an ideal framework for designing
and understanding algorithms.
The maximum subsequence sum problem, as well as the algorithms Max-
SumIter, MaxSumOpt, and MaxSumBU, was introduced by Bentley [12].
The sorting algorithm suggested by Exercise 1.6 is selection sort.
Java is a registered trademark of Sun Microsystems, Inc.
Chapter 2

Proving Algorithm
Correctness

In Chapter 1, we specified several problems and presented various algo-

rithms for solving these problems. For each algorithm, we argued somewhat
informally that it met its specification. In this chapter, we introduce a
mathematical foundation for more rigorous proofs of algorithm correctness.
Such proofs not only give us more confidence in the correctness of our al-
gorithms, but also help us to find subtle errors. Furthermore, some of the
tools we introduce here are also used in the context of analyzing algorithm
performance.

2.1 The Basics

First consider the algorithm SimpleSelect, shown in Figure 1.2 on page
6. This algorithm is simple enough that ordinarily we would not bother to
give a formal proof of its correctness; however, such a proof serves to illus-
trate the basic approach. Recall that in order for an algorithm to meet its
specification, it must be the case that whenever the precondition is satisfied
initially, the postcondition is also satisfied when the algorithm finishes.

Theorem 2.1 SimpleSelect meets the specification given in Figure 1.1.

Proof: First, we assume that the precondition for SimpleSelect(A[1..n])

is satisfied initially; i.e., we assume that initially A[1..n] is an array of num-
bers, 1 ≤ k ≤ n, and both k and n are natural numbers. This assump-
tion immediately implies the precondition for Sort(A[1..n]), namely, that

25
CHAPTER 2. PROVING ALGORITHM CORRECTNESS 26

A[1..n] is an array of numbers, and that n is a natural number. In order

to talk about both the initial and final values of A without confusion, let
us represent the final value of A by A′ and use A only to denote its initial
value. Because Sort permutes A (we know this from its postcondition), A′
contains the same collection of values as does A.
Suppose A′ [i] < A′ [k] for some i, 1 ≤ i ≤ n. Then i < k, for if k < i,
A [k] > A′ [i] violates the postcondition of Sort. Hence, there are fewer
′

than k elements of A′ , and hence of A, with value less than A′ [k].

Now suppose 1 ≤ i ≤ k. From the postcondition of Sort, A′ [i] ≤ A′ [k].
Hence, there are at least k elements of A with value less than or equal
to A′ [k]. The returned value A′ [k] therefore satisfies the postcondition of
Select.

Because of what it means for an algorithm to meet its specification,

any proof of correctness will begin by assuming the precondition. The goal If the
precondition is
of the proof is then to prove that the postcondition is satisfied when the defined to be
algorithm finishes. In order to reach this goal, we reason about the effect true, we don’t
of each statement in turn. If a statement is a call to another algorithm, we need to assume
it, because we
use that algorithm’s specification. Specifically, we must first make sure its know that true is
precondition is satisfied (for otherwise, we cannot know what the algorithm true.
does), then we conclude that its postcondition is satisfied when it finishes.
We can then use this postcondition to continue our reasoning.
The proof of Theorem 2.1 illustrates a common difficulty with correct-
ness proofs. In algorithms, variables typically change their values as the
algorithm progresses. However, in proofs, a variable must maintain a single
value in order to maintain consistent reasoning. In order to avoid confusion,
we introduce different names for the value of the same variable at different
points in the algorithm. It is customary to use the variable name from the
algorithm to denote the initial value, and to add a “prime” (′ ) to denote its
final value. If intermediate value must be considered, “double-primes” (′′ ),
superscripts, or subscripts can be used.
Although a variable should not change its value over the course of a proof,
we do sometimes use the same variable with different values when proving
different facts. For example, the variable i in the proof of Theorem 2.1 can
have different values in the second and third paragraphs, respectively. It
might be helpful to think of such usage as multiple local variables, each
having the same name. Within each scope, the value of the variable does
not change.
The proof of Theorem 2.1 is straightforward because the algorithm is
straight-line — the flow of control proceeds sequentially from the first state-
CHAPTER 2. PROVING ALGORITHM CORRECTNESS 27

ment in the algorithm to the last. The addition of if statements complicates

matters only slightly — we must simply apply case analysis. However, the
addition of recursion and/or loops requires some additional machinery, which
we will develop over the remainder of this chapter.

2.2 Handling Recursion

Let us now consider InsertSort from Figure 1.3 on page 7. To handle the
if statement, we can consider two cases, depending on whether n > 1. In
case n > 1, however, there is a recursive call to InsertSort. As is suggested
in Section 1.2, we might simply use the specification of Sort just like we
would for a call to any other algorithm. However, this type of reasoning
is logically suspect — we are assuming the correctness of InsertSort in
order to prove the correctness of InsertSort.
In order to break this circularity, we use the principle of mathematical
induction. We will be using this principle throughout this text, so we will
now take the time to present and prove it. The version we present is a
technique for proving properties of the natural numbers. The property that
we wish to prove in this case is that InsertSort(A[1..n]) satisfies its spec-
ification for every natural number n. Let us now formally define what we
mean by a property of the natural numbers, so that our statement of the
induction principle will be clear.

Definition 2.2 A property of the natural numbers is a mapping

P : N → {true, false};

i.e., for a natural number n, P (n) is either true or false.

Theorem 2.3 (Mathematical Induction Principle) Let P (n) be a property

of the natural numbers. Suppose that every natural number n satisfies the
following Induction Property:
• whenever P (i) is true for every natural number i < n, P (n) is also
true.
Then P (n) is true for every natural number n.
Before we prove this theorem, let us consider how we can use it. Suppose
we wish to prove a property P (n) for every natural number n. Theorem 2.3
tells us that it is sufficient to prove the Induction Property. We can break
this proof into two parts:
CHAPTER 2. PROVING ALGORITHM CORRECTNESS 28

1. The induction hypothesis. We assume that for some arbitrary n ∈ N,

P (i) is true for every natural number i < n.

2. The induction step. Using the induction hypothesis, we prove that

P (n) is true.

Some readers may be familiar with another version of induction consist-

ing of three steps:

1. The base case. P (0) is shown to be true.

2. The induction hypothesis. P (n) is assumed to be true for some arbi-

trary natural number n.

3. The induction step. Using the induction hypothesis, P (n+1) is proved.

Though this technique is also valid, the version given by Theorem 2.3 is
more appropriate for the study of algorithms. To see why, consider how we
are using the top-down approach. We associate with each input a natural
number that in some way describes the size of the input. We then recursively
apply the algorithm to one or more inputs of strictly smaller size. Theorem
2.3 tells us that in order to prove that this algorithm is correct for inputs of
all sizes, we may assume that for arbitrary n, the algorithm is correct for all
inputs of size less than n. Thus, we may reason about a recursive algorithm
in the same way we reason about an algorithm that calls other algorithms,
provided the size of the parameters is smaller for the recursive calls.
Now let us turn to the proof of Theorem 2.3.

Proof of Theorem 2.3: Suppose every natural number n satisfies the

Induction Property given in the statement of the theorem. In order to derive
a contradiction, assume that for some n ∈ N, P (n) = false. Specifically, let
n be the smallest such value. Then for every i < n, P (i) = true. By the
Induction Property, P (n) = true — a contradiction. We conclude that our
assumption was invalid, so that P (n) = true for every natural number n.

Before we illustrate the use of this principle by proving the correctness

of InsertSort, let us briefly discuss an element of style regarding induc-
tion proofs. One of the distinguishing features of this particular induction
principle is the absence of a base case. However, in most proofs, there are
special cases in which the induction hypothesis is not used. These are typi-
cally the smallest cases, where the induction hypothesis gives us little or no
information. It would be a bit of a misnomer to use the term, “induction
CHAPTER 2. PROVING ALGORITHM CORRECTNESS 29

step”, for such cases. It is stylistically better to separate these cases into
one or more base cases. Hence, even though a base case is not required, we
usually include one (or more).
Now we will illustrate the principle of induction by proving the correct-
ness of InsertSort.

Theorem 2.4 InsertSort, given in Figure 1.3, satisfies the specification

of Sort, given in Figure 1.2.

Proof: By induction on n.

Base: n ≤ 1. In this case the algorithm does nothing, but its postcondition
is vacuously satisfied (i.e., there are no i, j such that 1 ≤ i < j ≤ n).

Induction Hypothesis: Assume that for some n > 1, for every k < n,
InsertSort(A[1..k]) satisfies its specification.

Induction Step: We first assume that initially, the precondition for Insert-
Sort(A[1..n]) is satisfied. Then the precondition for InsertSort(A[1..n −
1]) is also initially satisfied. By the Induction Hypothesis, we conclude
that InsertSort(A[1..n − 1]) satisfies its specification; hence, its postcon-
dition holds when it finishes. Let A′′ denote the value of A after Insert-
Sort(A[1..n − 1]) finishes. Then A′′ [1..n − 1] is a permutation of A[1..n − 1]
in nondecreasing order, and A′′ [n] = A[n]. Thus, A′′ satisfies the precon-
dition of Insert. Let A′ denote the value of A after Insert(A[1..n]) is
called. By the postcondition of Insert, A′ [1..n] is a permutation of A[1..n]
in nondecreasing order. InsertSort therefore satisfies its specification.

Because the algorithm contains an if statement, the proof requires a

case analysis. The two cases are handled in the Base and the Induction
Step, respectively. Note that the way we prove the Induction Step is very
similar to how we proved Theorem 2.1. The only difference is that we have
the Induction Hypothesis available to allow us to reason about the recursive
call.

2.3 Handling Iteration

Let us now consider MaxSumBU, shown in Figure 1.14 on page 18. This
algorithm contains a for loop. As we did with recursion, we would like to
CHAPTER 2. PROVING ALGORITHM CORRECTNESS 30

be able to apply straight-line reasoning techniques. In Chapter 1, we used

invariants in order to focus on a single loop iteration. Because we already
know how to handle code without loops, focusing on a single iteration allows
us to apply techniques we have already developed. Loop invariants therefore
give us the power to reason formally about loops.
Suppose we wish to show that property P holds upon completion of a
given loop. Suppose further that we can show each of the following:

1. Initialization: The invariant holds prior to the first loop iteration.

2. Maintenance: If the invariant holds at the beginning of an arbitrary

loop iteration, then it must also hold at the end of that iteration.

3. Termination: The loop always terminates.

4. Correctness: Whenever the loop invariant and the loop exit condi-
tion both hold, then P must hold.

It is not hard to show by induction on the number of loop iterations that

if both the initialization and maintenance steps hold, then the invariant
holds at the end of each iteration. If the loop always terminates, then the
invariant and the loop exit condition will both hold when this happens. The
correctness step then guarantees that property P will hold after the loop
completes. The above four steps are therefore sufficient to prove that P
holds upon completion of the loop.
We now illustrate this proof technique by showing correctness of Max-
SumBU. In our informal justification of this algorithm, we used equations
(1.1) and (1.2); however, because we did not prove these equations, our proof
of correctness should not use them.

Theorem 2.5 MaxSumBU satisfies its specification.

Proof: Suppose the precondition holds initially. We will show that when
the loop finishes, m contains the maximum subsequence sum of A[0..n − 1],
so that the postcondition is satisfied. Note that the loop invariant states
that m is the maximum subsequence sum of A[0..i − 1].

Initialization: Before the loop iterates the first time, i has a value of 0.
The maximum subsequence sum of A[0..−1] is defined to be 0. m is initially
assigned this value. Likewise, the maximum suffix sum of A[0..−1] is defined
to be 0, and msuf is initially assigned this value. Therefore, the invariant
initially holds.
CHAPTER 2. PROVING ALGORITHM CORRECTNESS 31

Maintenance: Suppose the invariant holds at the beginning of some iter-

ation. We first observe that because the iteration occurs, 0 ≤ i ≤ n − 1;
hence, A[i] is a valid array location. Let msuf ′ , m′ and i′ denote the values
of msuf , m and i, respectively, at the end of this iteration. Then

• msuf ′ = max(0, msuf + A[i]);

• m′ = max(m, msuf ′ ); and

• i′ = i + 1.

For 0 ≤ j ≤ n, let sj denote the maximum subsequence sum of A[0..j − 1],

and let tj denote the maximum suffix sum of A[0..j − 1]. From the invariant, Corrected
msuf = ti , and m = si . We need to show that msuf ′ = ti′ and m′ = si′ . 1/27/12.

Using the definition of a maximum suffix sum, we have

msuf ′ = max(0, msuf + A[i])

= max(0, ti + A[i])
( i−1 ) !
X
= max 0, max A[k] | 0 ≤ l ≤ i + A[i]
k=l
i−1
( )
X
= max 0, A[i] + A[k] | 0 ≤ l ≤ i
k=l
i
( )
X
= max 0, A[k] | 0 ≤ l ≤ i
k=l
(i′ −1 )
X
′
= max A[k] | 0 ≤ l ≤ i
k=l
= ti′ .

Likewise, using the definitions of a maximum subsequence sum and a

CHAPTER 2. PROVING ALGORITHM CORRECTNESS 32

maximum suffix sum, we have

m′ = max(m, msuf ′ )
= max(si , ti′ )
(h−1 ) (i′ −1 )!
X X
= max max A[k] | 0 ≤ l ≤ h ≤ i , max A[k] | 0 ≤ l ≤ i′
k=l k=l
(h−1 )
X
= max A[k] | 0 ≤ l ≤ h ≤ i′
k=l
= si′ .

Therefore, the invariant holds at the end of the iteration.

Termination: Because the loop is a for loop, it clearly terminates. In this textbook,
a for loop always
contains a single
Correctness: The loop exits when i = n. Thus, from the invariant, m is index variable,
the maximum subsequence sum of A[0..n − 1] when the loop terminates. which either is
incremented by a
fixed positive
As can be seen from the above proof, initialization and maintenance can amount each
iteration until it
be shown using techniques we have already developed. Furthermore, the exceeds a fixed
correctness step is simply logical inference. In the case of Theorem 2.5, value or is
decremented by
termination is trivial, because for loops always terminate. Note, however, a fixed positive
that in order for such a proof to be completed, it is essential that a proper amount each
loop invariant be chosen. Specifically, the invariant must be chosen so that: iteration until it
is less than a
fixed value. The
• it is true every time the loop condition is tested; index cannot be
changed
• it is possible to prove that if it is true at the beginning of an arbitrary otherwise. Such
loops will always
iteration, it must also be true at the end of that iteration; and terminate.
Correctness case
• when coupled with the loop exit condition, it is strong enough to prove for Theorem 2.5
the desired correctness property. corrected
1/27/12.
Thus, if we choose an invariant that is too strong, it may not be true each
time the loop condition is tested. On the other hand, if we choose an invari-
ant that is too weak, we may not be able to prove the correctness property.
Furthermore, even if the invariant is true on each iteration and is strong
enough to prove the correctness property, it may still be impossible to prove
the maintenance step. We will discuss this issue in more detail shortly.
CHAPTER 2. PROVING ALGORITHM CORRECTNESS 33

Figure 2.1 A loop whose termination for all n is unknown

while n > 1
if n mod 2 = 0
n ← n/2
else
n ← 3n + 1

For while loops, the proof of termination is usually nontrivial and in

some cases quite difficult. An example that is not too difficult is Iter-
ativeInsert in Figure 1.6 (page 11). To prove termination of this loop,
we need to show that each iteration makes progress toward satisfying the
loop exit condition. The exit condition for this loop is that j ≤ 1 or
A[j] ≥ A[j − 1]. Usually, the way in which a loop will make progress toward
meeting such a condition is that each iteration will decrease the difference
between the two sides of an inequality. In this case, j is decreased by each it-
eration, and therefore becomes closer to 1. (The other inequality in the exit
condition is not needed to prove termination — if it becomes true, the loop
just terminates that much sooner.) We can therefore prove the following
theorem.

Theorem 2.6 The while loop in IterativeInsert always terminates.

Proof: We first observe that each iteration of the while loop decreases j
by 1. Thus, if the loop continues to iterate, eventually j ≤ 1, and the loop
then terminates.

Proving termination of a while loop can be much more difficult than the
proof of the above theorem. For example, consider the while loop shown
in Figure 2.1. The mod operation, when applied to positive integers, gives
the remainder obtained when an integer division is performed; thus, the
if statement tests whether n is even. Though many people have studied
this computation over a number of years, as of this writing, it is unknown
whether this loop terminates for all initial integer values of n. This question
is known as the Collatz problem.
On the other hand, when algorithms are designed using the top-down
approach, proving termination of any resulting while loops becomes much
CHAPTER 2. PROVING ALGORITHM CORRECTNESS 34

easier. Even if an examination of the while loop condition does not help
us to find a proof, we should be able to derive a proof from the reduction
we used to solve the problem. Specifically, a loop results from the reduction
of larger instances of a problem to smaller instances of the same problem,
where the size of the instance is a natural number. We should therefore
be able to prove that the expression denoting the size of the instance is a
natural number that is decreased by every iteration. Termination will then
follow.
For example, consider again the algorithm IterativeInsert. In the
design of this algorithm (see Section 1.4), we reduced larger instances to
smaller instances, where the size of an instance was n, the number of array
elements. In removing the tail recursion from the algorithm, we replaced n
by j. j should therefore decrease as the size decreases. We therefore base
our correctness proof on this fact.
Let us now consider an algorithm with nested loops, such as Insertion-
Sort, shown in Figure 1.7 on page 11. When loops are nested, we apply
the same technique to each loop as we encounter it. Specifically, in order
to prove maintenance for the outer loop, we need to prove that the inner
loop satisfies some correctness property, which should in turn be sufficient to
complete the proof of maintenance for the outer loop. Thus, nested within
the maintenance step of the outer loop is a complete proof (i.e., initialization,
maintenance, termination and correctness) for the inner loop.
When we prove initialization for the inner loop, we are not simply rea-
soning about the code leading to the first execution of that loop. Rather, we
are reasoning about the code that initializes the loop on any iteration of the
outer loop. For this reason, we cannot consider the initialization code for the
outer loop when proving the initialization step for the inner loop. Instead,
because the proof for the inner loop is actually a part of the maintenance
proof for the outer loop, we can use any facts available for use in the proof
of maintenance for the outer loop. Specifically, we can use the assumption
that the invariant holds at the beginning of the outer loop iteration, and
we can reason about any code executed prior to the inner loop during this
iteration. We must then show that the invariant of the inner loop is satisfied
upon executing this code.
We will now illustrate this technique by giving a complete proof that
InsertionSort meets its specification.

Theorem 2.7 InsertionSort meets its specification.

Proof: We must show that when the for loop finishes, A[1..n] is a permu-
CHAPTER 2. PROVING ALGORITHM CORRECTNESS 35

tation of its original values in nondecreasing order.

Initialization: (Outer loop) When the loop begins, i = 1 and the contents
of A[1..n] have not been changed. Because A[1..i − 1] is an empty array, it
is in nondecreasing order.

Maintenance: (Outer loop) Suppose the invariant holds at the beginning

of some iteration. Let A′ [1..n] denote the contents of A at the end of the
iteration, and let i′ denote the value of i at the end of the iteration. Then
i′ = i + 1. We must show that the while loop satisfies the correctness
property that A′ [1..n] is a permutation of the original values of A[1..n], and
that A′ [1..i′ − 1] = A′ [1..i] is in nondecreasing order.

Initialization: (Inner loop) Because A[1..n] has not been changed since
the beginning of the current iteration of the outer loop, from the outer loop
invariant, A[1..n] is a permutation of its original values. From the outer
loop invariant, A[1..i − 1] is in nondecreasing order; hence, because j = i,
we have for 1 ≤ k < k ′ ≤ i, where k ′ 6= j, A[k] ≤ A[k ′ ].

Maintenance: (Inner loop) Suppose the invariant holds at the beginning

of some iteration. Let A′ [1..n] denote the contents of A[1..n] following the
iteration, and let j ′ denote the value of j following the iteration. Hence,

i. A′ [j] = A[j − 1];

ii. A′ [j − 1] = A[j];

iii. A′ [k] = A[k] for 1 ≤ k ≤ n, k 6= j, and k 6= j − 1; and

iv. j ′ = j − 1.

Thus, A′ [1..n] is a permutation of A[1..n]. From the invariant, A′ [1..n] is

therefore a permutation of the original values of A[1..n]. Suppose 1 ≤ k <
k ′ ≤ i, where k ′ 6= j ′ = j − 1. We must show that A′ [k] ≤ A′ [k ′ ]. We
consider three cases.

Case 1: k ′ < j − 1. Then A′ [k] = A[k] and A′ [k ′ ] = A[k ′ ]. From the

invariant, A[k] ≤ A[k ′ ]; hence, A′ [k] ≤ A′ [k ′ ].

Case 2: k ′ = j. Then A′ [k ′ ] = A[j − 1]. If k = j − 1, then A′ [k] = A[j], and

from the while loop condition, A[j] < A[j − 1]. Otherwise, k < j − 1 and
CHAPTER 2. PROVING ALGORITHM CORRECTNESS 36

A′ [k] = A[k]; hence, from the invariant, A[k] ≤ A[j − 1]. In either case, we
conclude that A′ [k] ≤ A′ [k ′ ].

Case 3: k ′ > j. Then A′ [k ′ ] = A[k ′ ], and A′ [k] = A[l], where l is either

k, j, or j − 1. In each of these cases, l < k ′ ; hence, from the invariant,
A[l] ≤ A[k ′ ]. Thus, A′ [k] ≤ A′ [k ′ ].

Termination: (Inner loop) Each iteration decreases the value of j by 1;

hence, if the loop keeps iterating, j must eventually be no greater than 1.
At this point, the loop will terminate.

Correctness: (Inner loop) Let A′ [1..n] denote the contents of A[1..n] when
the while loop terminates, and let i and j denote their values at this point.
From the invariant, A′ [1..n] is a permutation of its original values. We must
show that A′ [1..i] is in nondecreasing order. Let 1 ≤ k < k ′ ≤ i. We consider
two cases.

Case 1: k ′ = j. Then j > 1. From the loop exit condition, it follows

that A′ [j − 1] ≤ A′ [j] = A′ [k ′ ]. From the invariant, if k 6= j − 1, then
A′ [k] ≤ A′ [j − 1]; hence, regardless of whether k = j − 1, A′ [k] ≤ A′ [k ′ ].

Case 2: k ′ 6= j. Then from the invariant, A′ [k] ≤ A′ [k ′ ].

This completes the proof for the inner loop, and hence the proof of
maintenance for the outer loop.

Termination: (Outer loop) Because the loop is a for loop, it must termi-
nate.

Correctness: (Outer loop) Let A′ [1..n] denote its final contents. From the
invariant, A′ [1..n] is a permutation of its original values. From the loop exit
condition (i = n + 1) and the invariant, A′ [1..n] is in nondecreasing order.
Therefore, the postcondition is satisfied.

Now that we have shown that InsertionSort is correct, let us consider

how we might have found the invariant for the inner loop. The inner loop
implements a transformation of larger instances of the insertion problem,
specified in Figure 1.3 on page 7, to smaller instances of the same prob-
lem. The loop invariant should therefore be related to the precondition for
Insert.
CHAPTER 2. PROVING ALGORITHM CORRECTNESS 37

The current instance of the insertion problem is represented by A[1..j].

Therefore, a first choice for an invariant might be that A[1..j] is a permu-
tation of its original values, and that A[1..j − 1] is sorted. However, this
invariant is not strong enough to prove the correctness property. To see
why, observe that the loop exit condition allows the loop to terminate when
j = 1. In this case, A[1..j] has only one element, A[1..j − 1] is empty, and
the invariant tells us almost nothing.
Clearly, we need to include in our invariant that A[1..n] is a permutation
of its initial values. Furthermore, we need more information about what has
already been sorted. Looking at the invariant for the outer loop, we might
try saying that both A[1..j − 1] and A[j..i] are in nondecreasing order. By
coupling this invariant with the loop exit condition (i.e, either j = 1 or
A[j − 1] ≤ A[j]), we can then show that A[1..i] is sorted. Furthermore, it is
possible to show that this invariant is true every time the loop condition is
tested. However, it still is not sufficient to prove the maintenance step for
this loop. To see why, observe that it tells us nothing about how A[j − 1]
compares with A[j +1]. Thus, when A[j −1] is swapped with A[j], we cannot
show that A[j] ≤ A[j + 1].
We need to express in our invariant that when we choose two indices
k < k ′ , where k ′ 6= j, we must have A[k] ≤ A[k ′ ]. The invariant in Figure
1.7 states precisely this fact. Arriving at this invariant, however, required
some degree of effort.
We mentioned in Section 1.4 that starting the for loop with i = 1, rather
than i = 2, simplifies the correctness proof without affecting the correctness.
We can now explain what we meant. Note that if we were to begin the for
loop with i = 2, its invariant would no longer be established initially if
n = 0. Specifically, A[1..i − 1] = A[1..1], and if n = 0, A[1] is not a valid
array location. A more complicated invariant — and consequently a more
complicated proof — would therefore be required to handle this special case.
By instead beginning the loop at 1, we have sacrificed a very small amount
of run-time overhead for the purpose of simplifying the invariant.

2.4 Combining Recursion and Iteration

In this section, we will present an alternative approach to solving the selec-
tion problem, specified in Figure 1.1 on page 4. This approach will ultimately
result in a recursive algorithm that also contains a loop. We will then show
how to combine the techniques presented in the last two sections in order
to prove such an algorithm to be correct.
CHAPTER 2. PROVING ALGORITHM CORRECTNESS 38

Figure 2.2 Specification of the median problem

Precondition: A[1..n] is an array of numbers, and n is a positive integer.

Postcondition: Returns the median of A[1..n]; i.e., returns x such that
fewer than ⌈n/2⌉ elements are less than x and at least ⌈n/2⌉ elements are
less than or equal to x.
Median(A[1..n])

We will reduce the selection problem to the following three problems:

• the Dutch national flag problem, defined below;
• the problem of finding the median of a nonempty array of numbers
(see Figure 2.2 for a specification of this problem); and Figure 2.2 uses
the notation ⌈x⌉,
• a smaller instance of the selection problem. pronounced the
ceiling of x, to
Somewhat informally, the input to the Dutch national flag problem is an denote the
smallest integer
array of items, each of which is colored either red, white, or blue. The goal no smaller than
is to arrange the items so that all of the red items precede all of the white x. Thus,
⌈3/2⌉ = 2, and
items, which in turn precede all of the blue items. This order is the order ⌈−3/2⌉ = −1.
of the colors appearing on the Dutch national flag, from top to bottom. We
will modify the problem slightly by assuming that all items are numbers,
and that a number is red if it is strictly less than some given value p, white
if it is equal to p, or blue if it is strictly greater than p.
The formal specification of this problem is given in Figure 2.3. Note that
we use the type Int to represent an integer. Notice also that because it may
be important to know the number of items of each color, these values are
returned in a 3-element array.
We can then find the kth smallest element in a nonempty array as follows:
1. Let p be the median element of the array.
2. Solve the resulting Dutch national flag problem.
3. If there are at least k red elements, return the kth smallest red element.
4. Otherwise, if there are at least k red and white elements combined,
return p.
5. Otherwise, return the (k − j)th smallest blue element, where j is the
number of red and white elements combined.
CHAPTER 2. PROVING ALGORITHM CORRECTNESS 39

Figure 2.3 Specification of DutchFlag

Precondition: A[lo..hi] is an array of Numbers, lo and hi are Ints such

that hi ≥ lo − 1, and p is a Number.
Postcondition: A[lo..hi] is a permutation of its original values such that,
all items less than p precede all items equal to p, which in turn precede all
items greater than p. Returns an array N [1..3] in which N [1] is the number
of items less than p, N [2] is the number of items equal to p, and N [3] is the
number of items greater than p in A[lo..hi].
DutchFlag(A[lo..hi], p)

Note that after we have solved the Dutch national flag problem, all
elements less than p appear first in the array, followed by all elements equal
to p, followed by all elements greater than p. Furthermore, because steps 3
and 5 apply to portions of the array that do not contain p, these steps solve
strictly smaller problem instances.
In what follows, we will develop a solution to the Dutch national flag
problem. We will then combine that solution with the above reduction to
obtain a solution to the selection problem (we will simply use the speci-
fication for Median). We will then prove that the resulting algorithm is
correct.
In order to conserve resources, we will constrain our solution to the
Dutch national flag problem to rearrange items by swapping them. We will
reduce a large instance of the problem to a smaller instance. We begin by
examining the last item. If it is blue, then we can simply ignore it and solve
what is left. If it is red, we can swap it with the first item and again ignore
it and solve what is left. If it is white, we need to find out where it belongs;
hence, we temporarily ignore it and solve the remaining problem. We then
swap it with the first blue item, or if there are no blue items, we can leave
it where it is. This algorithm is shown in Figure 2.4.
If we were to implement this solution, or to analyze it using the tech-
niques of Chapter 3, we would soon discover that its stack usage is too high.
Furthermore, none of the recursive calls occur at either the beginning or the
end of the computation; hence, the recursion is not tail recursion, and we
cannot implement it bottom-up.
We can, however, use a technique called generalization that will allow
us to solve the problem using a transformation. We first observe that the
CHAPTER 2. PROVING ALGORITHM CORRECTNESS 40

Figure 2.4 A top-down implementation DutchFlag, specified in Figure

2.3

DutchFlagTD(A[lo..hi], p)
if hi < lo
N ← new Array[1..3]; N [1] ← 0; N [2] ← 0; N [3] ← 0
else if A[hi] < p
A[lo] ↔ A[hi]; N ← DutchFlagTD(A[lo + 1..hi], p)
N [1] ← N [1] + 1
else if A[hi] = p
N ← DutchFlagTD(A[lo..hi − 1], p)
A[hi] ↔ A[lo + N [1] + N [2]]; N [2] ← N [2] + 1
else
N ← DutchFlagTD(A[lo..hi − 1], p); N [3] ← N [3] + 1
return N

only reason we must wait until after the recursive calls to increment the
appropriate element of N is that the recursive call is responsible for con-
structing and initializing N . If instead, we could provide initial values for
N [1..3] to the recursive calls, we could then incorporate the color of the last
element into these initial values. We therefore generalize the problem by
requiring as input initial values for the number of items of each color. The
returned array will then contain values representing the number of items of
the corresponding color, plus the corresponding initial value from the input.
By using 0 for all three initial values, we obtain the number of each color in
the entire array; hence, we have defined a more general problem.
We can use this generalization to make two of the calls tail recursion.
In order to be able to handle a white item, though, we need to modify our
generalization slightly. Specifically, we need to know in advance where to
put a white item. In order to be able to do this, let us specify that if w is
given as the initial value for the number of white items, then the last w items
in the array are white. Note that this variation is still a generalization of
the original problem, because if w = 0, no additional constraints are placed
on the input array.
Suppose we have an instance of this more general problem. If the initial
value for the number of white items is equal to the number of elements in the
array, then we can copy the initial values into N [1..3] and return. Otherwise,
CHAPTER 2. PROVING ALGORITHM CORRECTNESS 41

Figure 2.5 The transformation for Dutch national flag.

? ? · · ·? r w w · · · w w r ? · · ·? ? w w · · · w w

swap

? ? · · ·? w w w · · · w w ? ? · · ·? w w w · · · w w

? ? · · ·? b w w · · · w w ? ? · · ·? w w w · · · w b

swap

we examine the item preceding the first known white item (see Figure 2.5).
If it is red, we swap it with the first item and solve the smaller problem
obtained by ignoring the first item. If it is white, we solve the problem that
results from incrementing the initial number of white items. If it is blue, we
swap it with the last element, and solve the smaller problem obtained by
ignoring the last item. A recursive implementation of this strategy is shown
in Figure 2.6.
The way we handle the case in which an item is white is suspicious in
that the reduced instance is an array with the same number of elements.
However, note that in each case, the number of elements of unknown color
is decreased by the reduction. Thus, if we choose our definition of “size”
to be the number of elements of unknown color, then our reduction does
decrease the size of the problem in each case. Recall that our notion of
size is any natural number which decreases in all “smaller” instances. Our
reduction is therefore valid.
Figure 2.7 shows the result of eliminating the tail recursion from Dutch-
FlagTailRec, incorporating it into the selection algorithm described ear-
lier in this section, and making some minor modifications. First, lo and hi
have been replaced by 1 and n, respectively. Second, the array N has been
removed, and r, w, b are used directly instead. Finally, referring to Figure
2.6, note that when a recursive call is made, lo is incremented exactly when
r is incremented, and hi is decremented exactly when b is incremented. Be-
CHAPTER 2. PROVING ALGORITHM CORRECTNESS 42

Figure 2.6 Tail recursive solution to a generalization of the Dutch national

flag problem

Precondition: A[lo..hi] is an array of Numbers whose last w items are

equal to p, lo and hi are Ints such that w ≤ hi − lo + 1, and r, w, and b are
Nats.
Postcondition: A[lo..hi] is a permutation of its original values such that
all items less than p precede all items equal to p, which in turn precede all
items greater than p. Returns an array N [1..3] in which N [1] is r plus the
number of items less than p, N [2] is the number of items equal to p, and
N [3] is b plus the number of items greater than p in A[lo..hi].
DutchFlagTailRec(A[lo..hi], p, r, w, b)
if w ≥ hi − lo + 1
N ← new Array[1..3]; N [1] ← r; N [2] ← w; N [3] ← b
return N
else
j ← hi − w
if A[j] < p
A[j] ↔ A[lo]
return DutchFlagTailRec(A[lo + 1..hi], p, r + 1, w, b)
else if A[j] = p
return DutchFlagTailRec(A[lo..hi], p, r, w + 1, b)
else
A[j] ↔ A[hi]
return DutchFlagTailRec(A[lo..hi − 1], p, r, w, b + 1)

cause we are replacing lo with 1, which cannot be changed, and hi with

n, which we would rather not change, we instead use the expressions r + 1
and n − b, respectively. Thus, for example, instead of having a while loop
condition of w < hi − lo + 1, we replace lo with r + 1 and hi with n − b,
rearrange terms, and obtain r + w + b < n.
As we have already observed, the invariant for a loop implementing a
transformation is closely related to the precondition for the problem. Thus,
in order to obtain the loop invariant, we take the precondition for Dutch-
FlagTailRec, remove “A[lo..hi] is an array of Numbers”, as this is un-
derstood, and replace lo with r + 1 and hi with n − b. This gives us most of
CHAPTER 2. PROVING ALGORITHM CORRECTNESS 43

Figure 2.7 An algorithm for solving the selection problem, specified in

Figure 1.1, using the median

SelectByMedian(A[1..n], k)
p ← Median(A[1..n]); r ← 0; w ← 0; b ← 0
// Invariant: r, w, b ∈ N, r + w + b ≤ n, and A[i] < p for 1 ≤ i ≤ r,
// A[i] = p for n − b − w < i ≤ n − b, and A[i] > p for n − b < i ≤ n.
while r + w + b < n
j ←n−b−w
if A[j] < p
r ← r + 1; A[j] ↔ A[r]
else if A[j] = p
w ←w+1
else
A[j] ↔ A[n − b]; b = b + 1
if r ≥ k
return SelectByMedian(A[1..r], k)
else if r + w ≥ k
return p
else
return SelectByMedian(A[1 + r + w..n], k − (r + w))

the invariant. However, we must also take into account that the iterations
do not actually change the size of the problem instance; hence, the invari-
ant must also include a characterization of what has been done outside of
A[r + 1..n − b]. The portion to the left is where red items have been placed,
and the portion to the right is where blue items have been placed. We need
to include these constraints in our invariant.
Note that in Figure 2.7, the last line of SelectByMedian contains a
recursive call in which the first parameter is A[1 + r + w..n]. However, the
specification given in Figure 1.1 (page 4) states that the first parameter
must be of the form A[1..n]. To accommodate such a mismatch, we adopt a
convention that allows for automatic re-indexing of arrays when the specifi-
cation requires a parameter to be an array whose beginning index is a fixed
value. Specifically, we think of the sub-array A[1 + r + w..n] as an array
B[1..n − (r + w)]. B is then renamed to A when it is used as the actual
CHAPTER 2. PROVING ALGORITHM CORRECTNESS 44

parameter in the recursive call.

Let us now prove the correctness of SelectByMedian. Because Se-
lectByMedian contains a loop, we must prove this loop’s correctness us-
ing the techniques of Section 2.3. Specifically, we need the following lemma,
whose proof we leave as an exercise.

Lemma 2.8 If the precondition for SelectByMedian is satisfied, then

its while loop always terminates with A[1..n] being a permutation of its
original elements such that

• A[i] < p for 1 ≤ i ≤ r;

• A[i] = p for r < i ≤ r + w; and

• A[i] > p for r + w < i ≤ n.

Furthermore, when the loop terminates, r, w, and b are natural numbers

such that r + w + b = n.
We can then prove the correctness of SelectByMedian using induc-
tion.

Theorem 2.9 SelectByMedian meets the specification of Select given

in Figure 1.1.

Proof: By induction on n.

Induction Hypothesis: Assume that for some n ≥ 1, whenever 1 ≤ m <

n, SelectByMedian(A[1..m], k) meets its specification, where A[1..m] de-
notes (by re-indexing if necessary) any array with m elements.

Induction Step: Suppose the precondition is satisfied. By Lemma 2.8, the

while loop will terminate with A[1..n] being a permutation of its original
elements such that A[1..r] are less than p, A[r + 1..r + w] are equal to p, and
A[r + w + 1..n] are greater than p. We consider three cases.

Case 1: k ≤ r. In this case, there are at least k elements less than p,

so the kth smallest is less than p. Because A[1..r] are all the elements
smaller than p, the kth smallest of A[1..n] is the kth smallest of A[1..r].
Because p is an element of A[1..n] that is not in A[1..r], r < n. Furthermore,
because k ≤ r, the precondition of Select is satisfied by the recursive call
SelectByMedian(A[1..r], k). By the Induction Hypothesis, this recursive
CHAPTER 2. PROVING ALGORITHM CORRECTNESS 45

call returns the kth smallest element of A[1..r], which is the kth smallest of
A[1..n].

Case 2: r < k ≤ r + w. In this case, there are fewer than k elements less
than p and at least k elements less than or equal to p. p is therefore the kth
smallest element.

Case 3: r + w < k. In this case, there are fewer than k elements less than
or equal to p. The kth smallest must therefore be greater than p. It must
therefore be in A[r + w + 1..n]. Because every element in A[1..r + w] is less
than the kth smallest, the kth smallest must be the (k − (r + w))th smallest
element in A[r + w + 1..n]. Because p is an element of A[1..n] that is not in
A[r+w+1..n], r+w+1 > 1, so that the number of elements in A[r+w+1..n]
is less than n. Let us refer to A[r + w + 1..n] as B[1..n − (r + w)]. Then
because r + w < k, 1 ≤ k − (r + w), and because k ≤ n, k − (r + w) ≤
n − (r + w). Therefore, the precondition for Select is satisfied by the
recursive call SelectByMedian(B[1..n − (r + w)], k − (r + w)). By the
Induction Hypothesis, this recursive call returns the (k − (r + w))th smallest
element of B[1..n − (r + w)] = A[r + w + 1..n]. This element is the kth
smallest of A[1..n].

In some cases, a recursive call might occur inside a loop. For such cases,
we would need to use the induction hypothesis when reasoning about the
loop. As a result, it would be impossible to separate the proof into a lemma
dealing with the loop and a theorem whose proof uses induction and the
lemma. We would instead need to prove initialization, maintenance, termi-
nation, and correctness of the loop within the induction step of the induction
proof.

2.5 Mutual Recursion

The techniques we have presented up to this point are sufficient for prov-
ing the correctness of most algorithms. However, one situation can occur
which reveals a flaw in these techniques. Specifically, it is possible to prove
correctness using these techniques, when in fact the algorithm is incorrect.
The situation leading to this inconsistency is known as mutual recursion. In
a simple case, we have one algorithm, A, which calls another algorithm, B,
which in turn calls A. More generally, we may have a sequence of algorithms,
A1 , . . . , An , where Ai calls Ai+1 for 1 ≤ i < n, and An calls A1 .
CHAPTER 2. PROVING ALGORITHM CORRECTNESS 46

Figure 2.8 An implementation of Median, specified in Figure 2.7, using

Select, as specified in Figure 1.1

MedianBySelect(A[1..n])
return Select(A[1..n], ⌈n/2⌉)

For example, suppose we were to implement Median, as specified in Fig-

ure 2.7, by reducing it to the selection problem. This reduction is straight-
forward, as it is easily seen that the median is just the ⌈n/2⌉nd smallest
element. We therefore have the algorithm shown in Figure 2.8. Its proof of
correctness is trivial.
We therefore have the algorithm SelectByMedian, which correctly im-
plements Select if we use a correct implementation of Median. We also
have the algorithm MedianBySelect, which correctly implements Me-
dian if we use a correct implementation of Select. The problem arises
when we use both of these implementations together. The proof of cor-
rectness for the resulting implementation contains circular reasoning. In
fact, the implementation is not correct, as can be seen if we replace the call
to Median in Figure 2.7 with the call, SelectByMedian(A[1..n], ⌈n/2⌉).
We now have a recursive call whose argument is no smaller than that of the
original call. As a result, we have infinite recursion.
Though we will not prove it here, it turns out that nontermination is
the only way in which combining correct algorithms in a mutually recursive
fashion can result in an incorrect implementation. Thus, if we can prove
that the implementation terminates, we can conclude that it is correct. In
Chapter 3, we will present techniques for showing not just termination, but
the time it takes for an algorithm to terminate. These techniques will be
general enough to apply to mutually recursive algorithms.
In Chapter 15, we will present algorithms that use mutual recursion.
As we will see there, mutual recursion is sometimes useful for breaking a
complicated algorithm into manageable pieces. Apart from that chapter, we
will not dwell on mutual recursion. We should always be careful, however,
when we combine algorithms, that we do not inadvertently introduce mutual
recursion without proving termination of the implementation.
CHAPTER 2. PROVING ALGORITHM CORRECTNESS 47

2.6 Finding Errors

The process of proving correctness of an algorithm is more than just an
academic exercise. A proper correctness proof should give us confidence
that a given algorithm is, in fact, correct. It therefore stands to reason that
if a given algorithm is incorrect, the proof of correctness should fail at some
point. The process of proving correctness can therefore help us to find errors
in algorithms.
Suppose, for example, that in MaxSumBU (see Figure 1.14 on page 18),
we had miscalculated msuf using the statement

msuf ← msuf + A[i].

We could have made such an error by forgetting that there is a suffix of

A[0..i] — the empty suffix — that does not end with A[i]. We would expect
that a proof of correctness for such an erroneous algorithm should break
down at some point.
This error certainly wouldn’t affect the Initialization part of the proof, as
the initialization code is unchanged. Likewise, the Correctness part doesn’t
depend directly on code within the loop, but only on the invariant and the
loop exit condition. Because the Termination part is trivial, we are left with
the Maintenance part. Because we have changed the calculation of msuf ,
we would expect that we would be unable to prove that the second part of
the invariant is maintained (i.e., that msuf is the maximum suffix sum for
A[0..i − 1]).
Let us consider how this statement changes the Maintenance part of the
proof of Theorem 2.5. The first change is that now msuf ′ = msuf + A[i].
CHAPTER 2. PROVING ALGORITHM CORRECTNESS 48

This affects the derivation from msuf ′ in the following way:

msuf ′ = msuf + A[i]
= ti + A[i]
( i−1 )
X
= max A[k] | 0 ≤ l ≤ i + A[i]
k=l
i−1
( )
X
= max A[i] + A[k] | 0 ≤ l ≤ i
k=l
i
( )
X
= max A[k] | 0 ≤ l ≤ i
k=l
(i′ −1 )
X
′
= max A[k] | 0 ≤ l ≤ i − 1 .
k=l

However, (i′ −1 )
X
ti′ = max A[k] | 0 ≤ l ≤ i′ .
k=l
Note that the set on the right-hand side of this last equality has one
more element than does the set on the right-hand side of the preceding
equality. This element is generated by l = i′ , which results in an empty sum
having a value of 0. All of the remaining elements are derived from values
l ≤ i′ − 1, which result in nonempty sums of elements from A[0..i]. Thus, if
A[0..i] contains only negative values, msuf ′ < ti′ . It is therefore impossible
to prove that these values are equal.
A failure to come up with a proof of correctness does not necessarily
mean the algorithm is incorrect. It may be that we have not been clever
enough to find the proof. Alternatively, it may be that an invariant has
not been stated properly, as discussed in Section 2.3. Such a failure always
reveals, however, that we do not yet understand the algorithm well enough
to prove that it is correct.

2.7 Summary
We have introduced two main techniques for proving algorithm correctness,
depending on whether the algorithm uses recursion or iteration:
• The correctness of a recursive algorithm should be shown using induc-
tion.
CHAPTER 2. PROVING ALGORITHM CORRECTNESS 49

• The correctness of an iterative algorithm should be shown by proving

initialization, maintenance, termination, and correctness for each of
the loops.
Some algorithms might contain both recursion and iteration. In such
cases, both techniques should be used. Because the algorithm is recursive,
its correctness should be shown using induction. In order to complete the
induction, the loops will need to be handled by proving initialization, main-
tenance, termination, and correctness. In Chapter 4, we will see how these
techniques can be extended to proving the correctness of data structures.
Though correctness proofs are useful for finding errors in algorithms and
for giving us confidence that algorithms are correct, they are also quite
tedious. On the other hand, if an algorithm is fully specified and designed
in a top-down fashion, and if proper loop invariants are provided, working
out the details of a proof is usually not very hard. For this reason, we will
not provide many correctness proofs in the remainder of this text, but will
leave them as exercises. We will instead give top-down designs of algorithms
and provide invariants for most of the loops contained in them.

2.8 Exercises
Exercise 2.1 Induction can be used to prove solutions for summations. Use
induction to prove each of the following:
a. The arithmetic series:
n
X n(n + 1)
i= (2.1)
2
i=1

b. The geometric series:

n
X xn+1 − 1
xi = (2.2)
x−1
i=0

for any real x 6= 1.

Exercise 2.2 Let 

1
 if n = 0
f (n) = n−1
X
 f (i)
 if n > 0.
i=0
Use induction to prove that for all n > 0, f (n) = 2n−1 .
CHAPTER 2. PROVING ALGORITHM CORRECTNESS 50

* Exercise 2.3 The Fibonacci sequence is defined as follows:

(
n if 0 ≤ n ≤ 1
Fn = (2.3)
Fn−1 + Fn−2 if n > 1.

Use induction to prove each of the following properties of the Fibonacci

sequence:

a. For every n > 0,

Fn−1 Fn + Fn Fn+1 = F2n (2.4)
and
Fn2 + Fn+1
2
= F2n+1 . (2.5)
[Hint: Prove both equalities together in a single induction argument.]

b. For every n ∈ N,
φn − (−φ)−n
Fn = √ , (2.6)
5
where φ is the golden ratio:
√
1+ 5
φ= .
2

Exercise 2.4 Prove that RecursiveInsert, shown in Figure 1.4 on page

7, meets is specification, given in Figure 1.3 on page 7.

Exercise 2.5 Prove that MaxSuffixTD and MaxSumTD, given in Fig-

ure 1.13 (page 17), meet their specifications. For MaxSumTD, use the
specification of MaxSum given in Figure 1.9 (page 13).

Exercise 2.6 Prove that DotProduct, shown in Figure 1.17 on page 21,
meets its specification.

Exercise 2.7 Prove that Factorial, shown in Figure 2.9, meets its spec-
ification. n! (pronounced, “n factorial”) denotes the product 1 · 2 · · · n (0! is
defined to be 1).

Exercise 2.8 A minor modification of MaxSumOpt is shown in Figure

2.10 with its loop invariants. Prove that it meets the specification of Max-
Sum, given in Figure 1.9 (page 13).
CHAPTER 2. PROVING ALGORITHM CORRECTNESS 51

Figure 2.9 Algorithm for Factorial

Precondition: n is a Nat.
Postcondition: Returns n!.
Factorial(n)
p←1
// Invariant: p = (i − 1)!
for i ← 1 to n
p ← ip
return p

Figure 2.10 A minor modification of MaxSumOpt with loop invariants

MaxSumOpt2(A[0..n − 1])
m←0
// Invariant: m is the maximum of 0 and all sums of sequences A[l..h−1]
// such that 0 ≤ l < i and l ≤ h ≤ n.
for i ← 0 to n − 1
sum ← 0; p ← 0
// Invariant: sum is the sum of the sequence A[i..k − 1], and p is
// the maximum prefix sum of A[i..k − 1].
for k ← i to n − 1
sum ← sum + A[k]
p ← Max(p, sum)
m ← Max(m, p)
return m
CHAPTER 2. PROVING ALGORITHM CORRECTNESS 52

Figure 2.11 A minor modification of MaxSumIter with invariants

MaxSumIter2(A[0..n − 1])
m←0
// Invariant: m is the maximum of 0 and all sums of sequences A[l..h−1]
// such that 0 ≤ l < i and l ≤ h ≤ n.
for i ← 0 to n
p←0
// Invariant: p is the maximum prefix sum of A[i..j − 1].
for j ← i to n − 1
sum ← 0
// Invariant: sum is the sum of the sequence A[i..k − 1].
for k ← i to j
sum ← sum + A[k]
p ← Max(p, sum)
m ← Max(p, m)
return m

Exercise 2.9 A minor modification of MaxSumIter is shown in Figure

2.11 with its loop invariants. Prove that it meets the specification of Max-
Sum, given in Figure 1.9 (page 13).

Exercise 2.10 Prove that DutchFlagTD, given in Figure 2.4 (page 40),
meets its specification, given in Figure 2.3 (page 39).

* Exercise 2.11 Figure 2.12 shows a slightly optimized version of Inser-

tionSort. Prove that InsertionSort2 meets the specification given in
Figure 1.2 on page 6. You will need to find appropriate invariants for each
of the loops.

Exercise 2.12 Prove that DutchFlagTailRec, shown in Figure 2.6 on

page 42, meets its specification.

Exercise 2.13 Prove Lemma 2.8 (page 44).

* Exercise 2.14 Prove that Permutations, shown in Figure 2.13, meets Figure 2.13
corrected
it specification. Use the specifications of Copy, AppendToAll, and Fac- 2/10/10.
torial from Figures 1.18, 1.20, and 2.9, respectively.
CHAPTER 2. PROVING ALGORITHM CORRECTNESS 53

Figure 2.12 A slightly optimized version of InsertionSort

InsertionSort2(A[1..n])
for i ← 2 to n
j ← i; t ← A[j]
while j > 1 and A[j − 1] > t
A[j] ← A[j − 1]; j ← j − 1
A[j] ← t

Figure 2.13 Algorithm for Permutations

Precondition: A[1..n] is an array of distinct elements, and n is a Nat.

Postcondition: Returns an array P [1..n!] of all of the permutations of
A[1..n], where each permutation is itself an array Ai [1..n].
Permutations(A[1..n])
P ← new Array[1..Factorial(n)]
if n = 0
P [1] ← new Array[1..0]
else
k ← 1; nmin1fac ← Factorial(n − 1)
// Invariant: P [1..k − 1] contains all of the permutations of A[1..n]
// such that for 1 ≤ j < k, P [j][n] is in A[1..i − 1].
for i ← 1 to n
B ← new Array[1..n − 1]
Copy(A[1..i − 1], B[1..i − 1]); Copy(A[i + 1..n], B[i..n − 1])
C ← Permutations(B[1..n − 1])
AppendToAll(C[1..nmin1fac], A[i])
Copy(C[1..nmin1fac], P [k..k + nmin1fac − 1])
k ← k + nmin1fac
return P
CHAPTER 2. PROVING ALGORITHM CORRECTNESS 54

Figure 2.14 The algorithm for Exercise 2.15

Precondition: A[lo..hi] is an array of Numbers, lo and hi are Ints, and

p is a Number such that either all occurrences of p in A[lo..hi] precede all
occurrences of other values, or all occurrences of p follow all occurrences of
other values. We will refer to the first group of elements (i.e., either those
equal to p or those not equal to p, whichever comes first) as yellow, and the
other elements as green.
Postcondition: A[lo..hi] is a permutation of its initial values such that all
green elements precede all yellow elements.
SwapColors(A[lo..hi], p)
if lo ≤ hi + 1
i ← lo; j ← hi
// Invariant: lo ≤ i ≤ j + 1 ≤ hi + 1 and A[lo..hi] is a
// permutation of its original values such that A[k] is green for
// lo ≤ k < i, A[k] is yellow for j < k ≤ hi, and in A[i..j],
// all yellow elements precede all green elements.
while i < j and (A[i] = p) 6= (A[j] = p)
A[i] ↔ A[j]; i ← i + 1; j ← j − 1

Exercise 2.15 Prove that SwapColors, shown in Figure 2.14, meets its
specification. Note that the second conjunct in the while condition is com-
paring two boolean values; thus, it is true whenever exactly one of A[i] and
A[j] equals p.

Exercise 2.16 Figure 2.15 contains an algorithm for reducing the Dutch
national flag problem to the problem solved in Figure 2.14. However, the
algorithm contains several errors. Work through a proof that this algorithm
meets its specification (given in Figure 2.3 on page 39), pointing out each
place at which the proof fails. At each of these places, suggest a small change
that could be made to correct the error. In some cases, the error might be
in the invariant, not the algorithm itself.

* Exercise 2.17 Reduce the sorting problem to the Dutch national flag
problem and one or more smaller instances of itself.
CHAPTER 2. PROVING ALGORITHM CORRECTNESS 55

Figure 2.15 Buggy algorithm for Exercise 2.16

DutchFlagFiveBands(A[lo..hi], p)
i ← lo; j ← lo; k ← hi; l ← hi
// Invariant: lo ≤ i ≤ j ≤ hi, lo ≤ k ≤ l ≤ hi, A[lo..i − 1] all equal p,
// A[i..j − 1] are all less than p, A[k..l] are all greater than p, and
// A[l + 1..hi] all equal p.
while j < k
if A[j] < p
j ←j+1
else if A[j] = p
A[j] ↔ A[i]; i ← i + 1; j ← j + 1
else if A[k] = p
A[k] ↔ A[l]; k ← k − 1; l ← l − 1
else if A[k] > p
k ←k−1
else
A[j] ↔ A[k]; j ← j + 1; k ← k − 1
N ← new Array[1..3]
N [1] ← j − i; N [2] ← i − lo + hi − l; N [3] ← l − k
SwapColors(A[lo..j], p)
SwapColors(A[k..hi], p)
return N [1..3]

2.9 Chapter Notes

The techniques presented here for proving correctness of algorithms are
based on Hoare logic [60]. More complete treatments of techniques for prov-
ing program correctness can be found in Apt and Olderog [6] or Francez
[42]. Our presentation of proofs using invariants is patterned after Cormen,
et al. [25].
A discussion of the Dutch national flag problem and the iterative solution
used in SelectByMedian are given by Dijkstra [29]. The Collatz problem
was first posted by Lothar Collatz in 1937. An up-to-date summary of its
history is maintained by Eric Weisstein [111].
Chapter 3

Analyzing Algorithms

In Chapter 1, we saw that different algorithms for the same problem can have
dramatically different performance. In this chapter, we will introduce tech-
niques for mathematically analyzing the performance of algorithms. These
analyses will enable us to predict, to a certain extent, the performance of
programs using these algorithms.

3.1 Motivation
Perhaps the most common performance measure of a program is its running
time. The running time of a program depends not only on the algorithms it
uses, but also on such factors as the speed of the processor(s), the amount
of main memory available, the speeds of devices accessed, and the impact
of other software utilizing the same resources. Furthermore, the same algo-
rithm can perform differently when coded in different languages, even when
all other factors remain unchanged. When analyzing the performance of an
algorithm, we would like to learn something about the running time of any
of its implementations, regardless of the impact of these other factors.
Suppose we divide an execution of an algorithm into a sequence of steps,
each of which does some fixed amount of work. For example, a step could be
comparing two values or performing a single arithmetic operation. Assuming
the values used are small enough to fit into a single machine word, we could
reasonably expect that any processor could execute each step in a bounded
amount of time. Some of these steps might be faster than others, but for
any given processor, we should be able to identify both a lower bound l > 0
and an upper bound u ≥ l on the amount of time required for any single
execution step, assuming no other programs are being executed by that

56
CHAPTER 3. ANALYZING ALGORITHMS 57

processor. Thus, if we simply count execution steps, we obtain an estimate

on the running time, accurate to within a factor of u/l.
Obviously, these bounds will be different for different processors. Thus,
if an analysis of an algorithm is to be independent of the platform on which
the algorithm runs, the analysis must ignore constant factors. In other
words, our analyses will be unable to conclude, for example, that algorithm
A is twice as fast (or a million times as fast) as algorithm B. By ignoring
constant factors, we therefore lose a great deal of precision in measuring
performance. However, we will see that this loss of precision leads us to
focus on the more dramatic differences in algorithm performance. These
differences are important enough that they tend to transcend the differences
in platforms on which an algorithm is executed.
Because we are ignoring constant factors, it only makes sense to consider
the behavior of an algorithm on an infinite set of inputs. To see why, consider
that the execution times of two algorithms on the same single input are
always related by a constant factor — we simply divide the number of steps
in one execution by the number of steps in the other. This argument can be
extended to any finite set of inputs by dividing the number of steps in the
longest execution of one algorithm by the number of steps in the shortest
execution of the other.
Mathematically, we will describe the running time of an algorithm by
a function f : N → N. The input to f is a natural number representing
the size of an input. f (n) then represents the number of steps taken by the
algorithm on some particular input of size n. The context will determine
which input of size n we are considering, but usually we will be interested in
the worst-case input — an input of size n resulting in the maximum number
of execution steps.
Our analysis will then focus on this function f , not its value at specific
points. More precisely, we will focus our attention on the behavior of f (n)
as n increases. This behavior is known as the asymptotic behavior of f .
Most algorithms behave well enough if their inputs are small enough. By
focusing on asymptotic behavior, we can see how quickly the algorithm’s
performance will degrade as it processes larger inputs.
Throughout the remainder of this chapter, we will define various nota-
tions that allow us to relate the asymptotic behaviors of various functions
to each other. In this context, all functions will be of the form f : N → R≥0 ,
where R≥0 denotes the set of nonnegative real numbers (likewise, we will
use R to denote the set of all real numbers and R>0 to denote the set of
positive real numbers). Each of the notations we introduce will relate a set
of functions to one given function f based on their respective asymptotic
CHAPTER 3. ANALYZING ALGORITHMS 58

growth rates. Typically, f will be fairly simple, e.g., f (n) = n2 . In this way,
we will be able describe the growth rates of complicated — or even unknown
— functions using well-understood functions like n2 .

3.2 Big-O Notation

Definition 3.1 Let f : N → R≥0 . O(f (n)) is defined to be the set of all O(f (n)) is
pronounced
functions g : N → R≥0 such that for some natural number n0 and some “big-Oh of f of
strictly positive real number c, g(n) ≤ cf (n) whenever n ≥ n0 . n”.

The above definition formally defines big-O notation. Let us now dissect
this definition to see what it means. We start with some specific function f
which maps natural numbers to nonnegative real numbers. O(f (n)) is then
defined to be a set whose elements are all functions. Each of the functions in
O(f (n)) maps natural numbers to nonnegative real numbers. Furthermore,
if we consider any function g(n) in O(f (n)), then for every sufficiently large
n (i.e., n ≥ n0 ), g(n) cannot exceed f (n) by more than some fixed constant
factor (i.e., g(n) ≤ cf (n)). Thus, all of the functions in O(f (n)) grow no
faster than some constant multiple of f as n becomes sufficiently large. Note
that the constants n0 and c may differ for different f and g, but are the same
for all n.
Notice that big-O notation is defined solely in terms of mathematical
functions — not in terms of algorithms. Presently, we will show how it
can be used to analyze algorithms. First, however, we will give a series of
examples illustrating some of its mathematical properties.

Example 3.2 Let f (n) = n2 , and let g(n) = 2n2 . Then g(n) ∈ O(f (n))
because g(n) ≤ 2f (n) for every n ≥ 0. Here, the constant n0 is 0, and the
constant c is 2.

Example 3.3 Let f (n) = n2 , and let g(n) = 3n + 10. We wish to show
that g(n) ∈ O(f (n)). Hence, we need to find a positive real number c and
a natural number n0 such that 3n + 10 ≤ cn2 whenever n ≥ n0 . If n > 0,
we can divide both sides of this inequality by n, obtaining an equivalent
inequality, 3 + 10/n ≤ cn. The left-hand side of this inequality is maximized
when n is minimized. Because we have assumed n > 0, 1 is the minimum
value of n. Thus, if we can satisfy cn ≥ 13, the original inequality will be
satisfied. This inequality can be satisfied by choosing c = 13 and n ≥ 1.
Therefore, g(n) ∈ O(f (n)).
CHAPTER 3. ANALYZING ALGORITHMS 59

Example 3.4 n3 6∈ O(n2 ) because n3 = n(n2 ), so that whatever values we

pick for n0 and c, we can find an n ≥ n0 such that n(n2 ) > cn2 . Note that
in this example, we are using n3 and n2 to denote functions.

Example 3.5 1000 ∈ O(1). Here, 1000 and 1 denote constant functions
— functions whose values are the same for all n. Thus, for every n ≥ 0,
1000 ≤ 1000(1).

Example 3.6 O(n) ⊆ O(n2 ); i.e., every function in O(n) is also in O(n2 ).
To see this, note that for any function f (n) ∈ O(n), there exist a positive real
number c and a natural number n0 such that f (n) ≤ cn whenever n ≥ n0 .
Furthermore, n ≤ n2 for all n ∈ N. Therefore, f (n) ≤ cn2 whenever n ≥ n0 .

Example 3.7 O(n2 ) = O(4n2 + 7n); i.e., the sets O(n2 ) and O(4n2 + 7n)
contain exactly the same functions. It is easily seen that O(n2 ) ⊆ O(4n2 +
7n) using an argument similar to that of Example 3.6. Consider any function
f (n) ∈ O(4n2 + 7n). There exist a positive real number c and a natural
number n0 such that f (n) ≤ c(4n2 + 7n) whenever n ≥ n0 . Furthermore,
4n2 + 7n ≤ 11n2 for all n ∈ N. Letting c′ = 11c, we therefore have f (n) ≤
c′ n2 whenever n ≥ n0 . Therefore, f (n) ∈ O(n2 ). Note that although O(n2 )
and O(4n2 + 7n) denote the same set of functions, the preferred notation is
O(n2 ) because it is simpler.
Let us now illustrate the use of big-O notation by analyzing the run-
ning time of MaxSumBU from Figure 1.14 on page 18. The initialization
statements prior to the loop, including the initialization of the loop index i,
require a fixed number of steps. Their running time is therefore bounded by
some constant a. Likewise, the number of steps required by any single iter-
ation of the loop (including the loop test and the increment of i) is bounded
by some constant b. Because the loop iterates n times, the total number of
steps required by the loop is at most bn. Finally, the last loop condition
test and the return statement require a number of steps bounded by some
constant c. The running time of the entire algorithm is therefore bounded
by a + bn + c, where a, b, and c are fixed positive constants. The running
time of MaxSumBU is in O(n), because a + bn + c ≤ (a + b + c)n for all
n ≥ 1.
We can simplify the above analysis somewhat using the following theo-
rem.

Theorem 3.8 Suppose f1 (n) ∈ O(g1 (n)) and f2 (n) ∈ O(g2 (n)). Then
CHAPTER 3. ANALYZING ALGORITHMS 60

1. f1 (n)f2 (n) ∈ O(g1 (n)g2 (n)); and

2. f1 (n) + f2 (n) ∈ O(max(g1 (n), g2 (n))).

(By f1 (n)f2 (n), we mean the function that maps n to the product of f1 (n)
and f2 (n). Likewise, max(g1 (n), g2 (n)) denotes the function that maps n to
the maximum of g1 (n) and g2 (n).)

Proof: Because f1 (n) ∈ O(g1 (n)) and f2 (n) ∈ O(g2 (n)), there exist positive
real numbers c1 and c2 and natural numbers n1 and n2 such that

f1 (n) ≤ c1 g1 (n) whenever n ≥ n1 (3.1)

and
f2 (n) ≤ c2 g2 (n) whenever n ≥ n2 . (3.2)
Because both of the above inequalities involve only nonnegative numbers,
we may multiply the inequalities, obtaining

f1 (n)f2 (n) ≤ c1 c2 g1 (n)g2 (n)

whenever n ≥ max(n1 , n2 ). Let c = c1 c2 and n0 = max(n1 , n2 ). Then

f1 (n)f2 (n) ≤ cg1 (n)g2 (n)

whenever n ≥ n0 . Therefore, f1 (n)f2 (n) ∈ O(g1 (n)g2 (n)).

If we add inequalities (3.1) and (3.2), we obtain

f1 (n) + f2 (n) ≤ c1 g1 (n) + c2 g2 (n)

≤ c1 max(g1 (n), g2 (n)) + c2 max(g1 (n), g2 (n))
= (c1 + c2 ) max(g1 (n), g2 (n))

whenever n ≥ max(n1 , n2 ). Therefore, f1 (n)+f2 (n) ∈ O(max(g1 (n), g2 (n))). Corrected

2/24/10.

Let us now apply these two theorems to obtain a simpler analysis of

the running time of MaxSumBU. Recall that in our original analysis, we
concluded that the running time of a single iteration of the loop is bounded
by a fixed constant. We can therefore conclude that the running time of a
single iteration is in O(1). Because there are n iterations, the running time
for the entire loop is bounded by the product of n and the running time of
a single iteration. By Theorem 3.8 part 1, the running time of the loop is
in O(n). Clearly, the running times of the code segments before and after
CHAPTER 3. ANALYZING ALGORITHMS 61

the loop are each in O(1). The total running time is then the sum of the
running times of these segments and that of the loop. By applying Theorem
3.8 part 2 twice, we see that the running time of the algorithm is in O(n)
(because max(1, n) ≤ n whenever n ≥ 1).
Recall that the actual running time of the program implementing Max-
SumOpt (Figure 1.11 on page 15) was much slower than that of Max-
SumBU. Let us now analyze MaxSumOpt to see why this is the case.
We will begin with the inner loop. It is easily seen that each iteration
runs in O(1) time. The number of iterations of this loop varies from 1 to n.
Because the number of iterations is in O(n), we can conclude that this loop
runs in O(n) time. It is then easily seen that a single iteration of the outer
loop runs in O(n) time. Because the outer loop iterates n times, this loop,
and hence the entire algorithm, runs in O(n2 ) time.
It is tempting to conclude that this analysis explains the difference in
running times of the implementations of the algorithms; i.e., because n2
grows much more rapidly than does n, MaxSumOpt is therefore much
slower than MaxSumBU. However, this conclusion is not yet warranted,
because we have only shown upper bounds on the running times of the two
algorithms. In particular, it is perfectly valid to conclude that the running
time of MaxSumBU is in O(n2 ), because O(n) ⊆ O(n2 ). Conversely, we
have not shown that the running time of MaxSumOpt is not in O(n).
In general, big-O notation is useful for expressing upper bounds on the
growth rates of functions. In order to get a complete analysis, however, we
need additional notation for expressing lower bounds.

3.3 Big-Ω and Big-Θ

Definition 3.9 Let f : N → R≥0 . Ω(f (n)) is defined to be the set of all Ω(f (n)) is
pronounced
functions g : N → R≥0 such that for some natural number n0 and some “big-Omega of f
strictly positive real number c, g(n) ≥ cf (n) whenever n ≥ n0 . of n”.

Note that the definition of Ω is identical to the definition of O, except

that the inequality, g(n) ≤ cf (n), is replaced by the inequality, g(n) ≥ cf (n).
Thus, Ω notation is used to express a lower bound in the same way that O
notation is used to express an upper bound. Specifically, if g(n) ∈ Ω(f (n)),
then for sufficiently large n, g(n) is at least some constant multiple of f (n).
This constant multiple is only required to be a positive real number, so it
may be very close to 0.

Example 3.10 Let f (n) = 3n + 10 and g(n) = n2 . We wish to show that

CHAPTER 3. ANALYZING ALGORITHMS 62

g(n) ∈ Ω(f (n)). We therefore need to find a positive real number c and a
natural number n0 such that n2 ≥ c(3n + 10) for every n ≥ n0 . We have
already found such values in Example 3.3: c = 1/13 and n0 = 1.
The above example illustrates a duality between O and Ω, namely, that
for any positive real number c, g(n) ≤ cf (n) iff f (n) ≥ g(n)/c. The following
theorem summarizes this duality.

Theorem 3.11 Let f : N → R≥0 and g : N → R≥0 . Then g(n) ∈ O(f (n))
iff f (n) ∈ Ω(g(n)).
By applying Theorem 3.11 to Examples 3.2, 3.4, 3.6, and 3.7, we can see
that n2 ∈ Ω(2n2 ), n2 6∈ Ω(n3 ), Ω(n2 ) ⊆ Ω(n), and Ω(n2 ) = Ω(4n2 + 7n).
When we analyze the growth rate of a function g, we would ideally like
to find a simple function f such that g(n) ∈ O(f (n)) and g(n) ∈ Ω(f (n)).
Doing so would tell us that the growth rate of g(n) is the same as that of
f (n), within a constant factor in either direction. We therefore have another
notation for expressing such results.

Definition 3.12 Let f : N → R≥0 . Θ(f (n)) is defined to be Θ(f (n)) is

pronounced
“big-Theta of f
O(f (n)) ∩ Ω(f (n)). of n”.

In other words, Θ(f (n)) is the set of all functions belonging to both
O(f (n)) and Ω(f (n)) (see Figure 3.1). We can restate this definition by
the following theorem, which characterizes Θ(f (n)) in terms similar to the
definitions of O and Ω.

Theorem 3.13 g(n) ∈ Θ(f (n)) iff there exist positive constants c1 and c2
and a natural number n0 such that

c1 f (n) ≤ g(n) ≤ c2 f (n) (3.3)

whenever n ≥ n0 .

Proof: We must prove the implication in two directions.

⇒: Suppose g(n) ∈ Θ(f (n)). Then g(n) ∈ O(f (n)) and g(n) ∈ Ω(f (n)).
By the definition of Ω, there exist a positive real number c1 and a natural
number n1 such that c1 f (n) ≤ g(n) whenever n ≥ n1 . By the definition of
O, there exist a positive real number c2 and a natural number n2 such that
CHAPTER 3. ANALYZING ALGORITHMS 63

Figure 3.1 Venn diagram depicting the relationships between the sets
O(f (n)), Ω(f (n)), and Θ(f (n))
11111111111
00000000000
00000000000
11111111111
00000000000
11111111111 00
11 000
111
00000000000
11111111111
00000000000
11111111111 00
11 000
111
00000000000
11111111111 00
11
00
11 000
111
000
111
00000000000
11111111111
00000000000
11111111111 00
11 000
111
00111
11 000
00000000000
11111111111 00
11 000
111
00000000000
11111111111
00000000000
11111111111 00
11 000
111
10101010 O(f (n))

10
00
11 Ω(f (n))

00
11 Θ(f (n))

g(n) ≤ c2 f (n) whenever n ≥ n2 . Let n0 = max(n1 , n2 ). Then (3.3) holds

whenever n ≥ n0 .

⇐: Suppose (3.3) holds whenever n ≥ n0 . From the first inequality, g(n) ∈

Ω(f (n)). From the second inequality, g(n) ∈ O(f (n)). Therefore, g(n) ∈
Θ(f (n)).

The definition of Θ also gives us the following corollary to Theorem 3.11.

Corollary 3.14 Let f : N → R≥0 and g : N → R≥0 . Then g(n) ∈ Θ(f (n))
iff f (n) ∈ Θ(g(n)).
Let us now use these definitions to continue the analysis of MaxSumBU.
The analysis follows the same outline as the upper bound analysis; hence,
we need the following theorem, whose proof is left as an exercise.

Theorem 3.15 Suppose f1 (n) ∈ Ω(g1 (n)) and f2 (n) ∈ Ω(g2 (n)). Then
CHAPTER 3. ANALYZING ALGORITHMS 64

1. f1 (n)f2 (n) ∈ Ω(g1 (n)g2 (n)); and

2. f1 (n) + f2 (n) ∈ Ω(max(g1 (n), g2 (n))).

By combining Theorems 3.8 and 3.15, we obtain the following corollary.

Corollary 3.16 Suppose f1 (n) ∈ Θ(g1 (n)) and f2 (n) ∈ Θ(g2 (n)). Then

1. f1 (n)f2 (n) ∈ Θ(g1 (n)g2 (n)); and

2. f1 (n) + f2 (n) ∈ Θ(max(g1 (n), g2 (n))).

We are now ready to proceed with our analysis of MaxSumBU. Clearly,

the body of the loop must take some positive number of steps, so its running
time is in Ω(1). Furthermore, the loop iterates n times. We may therefore
use Theorem 3.15 to conclude that the running time of the algorithm is in
Ω(n). Because we have already shown the running time to be in O(n), it
therefore is in Θ(n).
Let us now analyze the lower bound for MaxSumOpt. Again, the inner
loop has a running time in Ω(1). Its number of iterations ranges from 1 to n,
so the best lower bound we can give on the number of iterations is in Ω(1).
Using this lower bound, we conclude that the running time of the inner loop
is in Ω(1). Because the outer loop iterates n times, the running time of the
algorithm is in Ω(n).
Unfortunately, this lower bound does not match our upper bound of
O(n2 ). In some cases, we may not be able to make the upper and lower
bounds match. In most cases, however, if we work hard enough, we can
bring them together.
Clearly, the running time of a single iteration of the inner loop will
require a constant number of steps in the worst case. Let a > 0 denote that
constant. The loop iterates n − i times, so that the total number of steps
required by the inner loop is (n − i)a. An iteration of the outer loop requires
a constant number of steps apart from the inner loop. Let b > 0 denote that
constant. The loop iterates n times. However, because the number of steps
required for the inner loop depends on the value of i, which is different for
each iteration of the outer loop, we must be more careful in computing the
total number of steps required by the outer loop. That number is given by
n−1
X n−1
X
b+ (n − i)a = b + a (n − i).
i=0 i=0
CHAPTER 3. ANALYZING ALGORITHMS 65

The above summation can be simplified if we observe that the quantity

(n − i) takes on the values n, n − 1, . . . , 1. We can therefore rewrite the sum
by taking the terms in the opposite order:
n
X
1 + 2 + ··· + n = i.
i=1

Thus, the number of steps required by the inner loop is

n
X
b+a i.
i=1

We can now use (2.1) from page 49 to conclude that the number of steps
taken by the outer loop is

an(n + 1)
b+ ∈ Θ(n2 ).
2
Therefore, the running time of the algorithm is in Θ(n2 ).
This is a rather tedious analysis for such a simple algorithm. Fortunately,
there are techniques for simplifying analyses. In the next two sections, we
will present some of these techniques.

3.4 Operations on Sets

Asymptotic analysis can be simplified if we extend operations on functions
to operations on sets of functions. Such an extension will allow us to stream-
line our notation without the need to introduce new constants or functions
representing the running times of various code segments.

Definition 3.17 Let ◦ be a binary operation on functions of the form f :

N → R≥0 (for example, ◦ might represent addition or multiplication). Let
f be such a function, and let A and B be sets of such functions. We then
define:

• f ◦ A = {f ◦ g | g ∈ A};

• A ◦ f = {g ◦ f | g ∈ A}; and

• A ◦ B = {g ◦ h | g ∈ A, h ∈ B}.
CHAPTER 3. ANALYZING ALGORITHMS 66

Example 3.18 n2 + Θ(n3 ) is the set of all functions that can be written
n2 + g(n) for some g(n) ∈ Θ(n3 ). This set includes such functions as:
• n2 + 3n3 ;
3 n3 +1
• (n3 +1)/2, which can be written n2 +( n 2+1 −n2 ) (note that 2 −n
2 ≥
0 for all natural numbers n); and
• n3 + 2n, which can be written n2 + (n3 + 2n − n2 ).
Because all functions in this set belong to Θ(n3 ), n2 + Θ(n3 ) ⊆ Θ(n3 ).

Example 3.19 O(n2 ) + O(n3 ) is the set of functions that can be written
f (n) + g(n), where f (n) ∈ O(n2 ) and g(n) ∈ O(n3 ). Functions in this set
include:
• 2n2 + 3n3 ;
• 2n, which can be written as n + n; and
• 2n3 , which can be written as 0 + 2n3 .
Because all functions in this set belong to O(n3 ), O(n2 ) + O(n3 ) ⊆ O(n3 ).

Definition 3.20 Let A be a set of functions of the form f : N → R≥0 . We

define
Xn
A(i)
i=k

to be the set of all functions g : N → R≥0 such that

n
X
g(n) = f (i)
i=k

for some f ∈ A. We define products analogously.

Example 3.21
n
X
Θ(i2 )
i=1
is the set of all functions of the form
n
X
f (i)
i=1

such that f (i) ∈ Θ(i2 ).

CHAPTER 3. ANALYZING ALGORITHMS 67

Example 3.22 f (n) ∈ f (n − 1) + Θ(n) for n ≥ 1. Here, we interpret

“f (n − 1) . . . for n ≥ 1” as shorthand for the following function:
(
f (n − 1) for n ≥ 1
g(n) =
0 otherwise

This is an example of an asymptotic recurrence. The meaning is that f is a

function satisfying a recurrence of the form
(
f (n − 1) + h(n) for n ≥ 1
f (n) =
h(n) otherwise

for some h(n) ∈ Θ(n). Note that because h(0) may have any nonnegative
value, so may f (0).
We can use the above definitions to simplify our analysis of the lower
bound for MaxSumOpt. Instead of introducing the constant a to represent
the running time of a single iteration of the inner loop, we can simply use
Ω(1) to represent the lower bound for this running time. We can therefore
conclude that the total running time of the inner loop is in Ω(n − i). Using
Definition 3.20, we can then express the running time of the outer loop, and
hence, of the entire algorithm, as being in
n−1
X
Ω(n − i).
i=0

While this notation allows us to simplify the expression of bounds on

running times, we still need a way of manipulating such expressions as the
one above. In the next section, we present powerful tools for performing
such manipulation.

3.5 Smooth Functions and Summations

Asymptotic analysis involving summations can be simplified by applying a
rather general property of summations. This property relies on the fact
that our summations typically involve well-behaved functions — functions
that obey three important properties. The following definitions characterize
these properties.

Definition 3.23 Let f : N → R≥0 . f is said to be eventually nondecreasing

if there is a natural number n0 such that f (n) ≤ f (n + 1) whenever n ≥ n0 .
CHAPTER 3. ANALYZING ALGORITHMS 68

Definition 3.24 Let f : N → R≥0 . f is said to be eventually positive if

there is a natural number n0 such that f (n) > 0 whenever n ≥ n0 .

Definition 3.25 Let f : N → R≥0 be an eventually nondecreasing and

eventually positive function. f is said to be smooth if there exist a real
number c and a natural number n0 such that f (2n) ≤ cf (n) whenever
n ≥ n0 .

Example 3.26 f (n) = n is a smooth function. Clearly, f is eventually

nondecreasing and eventually positive, and f (2n) = 2f (n) for all n ∈ N.

Example 3.27 f (n) = 2n is not smooth. f is eventually nondecreasing,

and eventually positive, but f (2n) = 22n = f 2 (n) for all n ∈ N. Because f
is unbounded, for any real c, f (2n) > cf (n) for all sufficiently large n.
We will soon discuss in more detail which functions are smooth. First,
however, let’s see why this notion is important. Suppose we want to give
asymptotic bounds for a summation of the form
g(n)
X
Ω(f (i))
i=1

for some smooth function f . The following theorem, whose proof is outlined
in Exercise 3.10, can then be applied.

Theorem 3.28 Let f : N → R≥0 be a smooth function, g : N → N be an

eventually nondecreasing and unbounded function, and let X denote either
O, Ω, or Θ. Then
g(n)
X
X(f (i)) ⊆ X(g(n)f (g(n))).
i=1
We leave as an
exercise the
Thus, if we know that f is smooth, we have an asymptotic solution to proof that this
containment is
the summation. We therefore need to examine the property of smoothness proper.
more closely. The following theorem can be used to show a wide variety of
functions to be smooth.

Theorem 3.29 Let f : N → R≥0 and g : N → R≥0 be smooth functions,

and let c ∈ R≥0 . Then the following functions are smooth:
CHAPTER 3. ANALYZING ALGORITHMS 69

• f (n) + g(n);

• f (n)g(n);

• f c (n); and

• f (g(n)), provided g is unbounded.

The proof is left as an exercise. Knowing that f (n) = n is smooth, we

can apply Theorem 3.29 to conclude
√ that any polynomial is smooth. In fact,
√
such functions as n and n 2 are also smooth. We can extend this idea to
logarithms as well. In particular, let lg x denote the base-2 logarithm; i.e.,

2lg x = x (3.4)

for all positive x. Strictly speaking, lg is not a function of the form f : N →

R≥0 , because lg 0 is undefined. However, whenever we have a function that
maps all but finitely many natural numbers to nonnegative real numbers,
we simply “patch” the function by defining it to be 0 at all other points.
This is safe to do when we are performing asymptotic analysis because for
sufficiently large n, the patched function matches the original function.

Example 3.30 lg n is smooth. Clearly lg n is eventually nondecreasing and

eventually positive. Furthermore, lg(2n) = 1 + lg n ≤ 2 lg n whenever n ≥ 2.
Thus far, the only example we have seen of a non-smooth function is 2n .
Indeed, almost any polynomial-bounded, eventually nondecreasing, eventu-
ally positive function we encounter will turn out to be smooth. However,
we can contrive exceptions. For example, we leave it as an exercise to show
⌊lg lg n⌋
that 22 ∈ O(n), but is not smooth. Given a real
number x, ⌊x⌋
We can now continue the analysis of the lower bound for MaxSumOpt. (pronounced the
As we showed in the previous section, this lower bound is in floor of x) is the
greatest integer
n−1
X no greater than
x. Thus,
Ω(n − i). ⌊3/2⌋ = 1, and
i=0 ⌊−3/2⌋ = −2.

Unfortunately, Theorem 3.28 does not immediately apply to this summa-

tion. First, the lower limit of the index i is 0, not 1 as required by Theorem
3.28. Furthermore, the theorem requires the expression inside the asymp-
totic notation to be a function of the summation index i, not of n − i.
On the other hand, we can take care of both of the above problems
using the same technique that we used in our original analysis in Section
CHAPTER 3. ANALYZING ALGORITHMS 70

3.3. Specifically, we reverse the order of the summation to obtain

n−1
X n
X
Ω(n − i) = Ω(i).
i=0 i=1

Now the initial index of i is 1, and i is a smooth function. In order to

apply Theorem 3.28, we observe that g(n) in the theorem corresponds to
n in the above summation, and that f (i) in the theorem corresponds to i
in the above summation. g(n)f (g(n)) is therefore just n2 . From Theorem
3.28 the running time of MaxSumOpt is in Ω(n2 ). Note that this is the
same bound that we obtained in Section 3.3, but instead of using Equation
2.1, we used the more general (and hence, more widely applicable) Theorem
3.28.
To further illustrate the power of Theorem 3.28, let’s now analyze the
running time of MaxSumIter, given in Figure 1.10 on page 14. A single
iteration of the inner loop has a running time in Θ(1). This loop iterates
j − i times, so its running time is in Θ(j − i). The total running time of the
middle loop is then in
X n
Θ(j − i).
j=i

Again, this summation does not immediately fit the form of Theorem
3.28, as the starting value of the summation index j is i, not 1. Furthermore,
j − i is not a function of j. Notice that the expression j − i takes on the
values 0, 1, . . . , n − i. We can therefore rewrite this sum as
n
X n−i+1
X
Θ(j − i) = Θ(j − 1).
j=i j=1

What we have done here is simply to shift the range of j downward by i − 1

(i.e., from i, . . . , n to 1, . . . , n − i + 1), and to compensate for this shift by
adding i − 1 to each occurrence of j in the expression being summed.
Applying Theorem 3.28 to the above sum, we find that the running time
of the middle loop is in Θ((n − i + 1)(n − i)) = Θ((n − i)2 ). The running
time of the outer loop is then
n
X
Θ((n − i)2 ).
i=0
CHAPTER 3. ANALYZING ALGORITHMS 71

The values in this summation are n2 , (n−1)2 , . . . , 0; hence, we can rewrite

this sum as
n+1
X
(i − 1)2 .
i=1
Applying Theorem 3.28 to this sum, we find that the running time of this
loop is in

Θ((n + 1)(((n + 1) − 1)2 )) = Θ((n + 1)n2 )

= Θ(n3 ).

The running time of the algorithm is therefore in Θ(n3 ).

3.6 Analyzing while Loops

To analyze algorithms with while loops, we can use the same techniques as
we have used to analyze for loops. For example, consider InsertionSort,
shown in Figure 1.7 on page 11. Let us consider the while loop. The value
of j begins at i and decreases by 1 on each loop iteration. Furthermore, if
its value reaches 1, the loop terminates. The loop therefore iterates at most
i − 1 times. Because each iteration runs in Θ(1) time, the while loop runs
in O(i) time in the worst case.
In order to be able to conclude that the loop runs in Θ(i) time in the
worst case, we must determine that for arbitrarily large i, the loop may
iterate until j = 1. This is certainly the case if, prior to the beginning of the
loop, A[i] is strictly less than every element in A[1..i − 1]. Thus, the while
loop runs in Θ(i) time in the worst case.
It is now tempting to use Theorem 3.28 to conclude that the entire
algorithm’s running time is in
n
X
Θ(1) + Θ(i) ⊆ Θ(1) + Θ(n2 )
i=1
= Θ(n2 ).

However, we must be careful, because we have not shown that the while
loop runs in Ω(i) time for every iteration of the for loop; hence the running
time of the for loop might not be in
n
X
Θ(i).
i=1
CHAPTER 3. ANALYZING ALGORITHMS 72

We must show that there are inputs of size n, for every sufficiently large
n, such that the while loop iterates i − 1 times for each iteration of the for
loop. It is not hard to show that an array of distinct elements in decreasing
order will produce the desired behavior. Therefore, the algorithm indeed
operates in Θ(n2 ) time.

3.7 Analyzing Recursion

Before we consider how to analyze recursion, let us first consider how to
analyze non-recursive function calls. For example, consider SimpleSelect
from Figure 1.2 on page 6. This algorithm is easy to analyze if we know the
running time of Sort. Suppose we use InsertionSort (Figure 1.7, page
11). We saw in the last section that InsertionSort runs in Θ(n2 ) time.
The running time of SimpleSelect is therefore in

Θ(1) + Θ(n2 ) ⊆ Θ(n2 ).

Suppose now that we wish to analyze an algorithm that makes one or

more recursive calls. For example, consider MaxSuffixTD from Figure
1.13 on page 17. We analyze such an algorithm in exactly the same way.
Specifically, this algorithm has a running time in Θ(1) plus whatever is
required by the recursive call. The difficulty here is in how to determine the
running time of the recursive call without knowing the running time of the
algorithm.
The solution to this difficulty is to express the running time as a recur-
rence. Specifically, let f (n) denote the worst-case running time of Max-
SuffixTD on an array of size n. Then for n > 0, we have the equation,

f (n) = g(n) + f (n − 1) (3.5)

where g(n) ∈ Θ(1) is the worst-case running time of the body of the function,
excluding the recursive call. Note that f (n − 1) has already been defined to
be the worst-case running time of MaxSuffixTD on an array of size n − 1;
hence, f (n − 1) gives the worst-case running time of the recursive call.
The solution of arbitrary recurrences is beyond the scope of this book.
However, asymptotic solutions are often much simpler to obtain than are
exact solutions. First, we observe that (3.5) can be simplified using set
operations:
f (n) ∈ f (n − 1) + Θ(1) (3.6)
CHAPTER 3. ANALYZING ALGORITHMS 73

for n > 0.
It turns out that most of the recurrences that we derive when analyzing
algorithms fit into a few general forms. With asymptotic solutions to these
general forms, we can analyze recursive algorithms without using a great
deal of detailed mathematics. (3.6) fits one of the most basic of these forms.
The following theorem, whose proof is outlined in Exercise 3.20, gives the
asymptotic solution to this form.

Theorem 3.31 Let

f (n) ∈ af (n − 1) + X(bn g(n))

for n > n0 , where n0 ∈ N, a ≥ 1 and b ≥ 1 are real numbers, g(n) is a

smooth function, and X is either O, Ω, or Θ. Then

 n
X(b g(n)) if a < b
n
f (n) ∈ X(na g(n)) if a = b


X(an ) if a > b.

When we apply this theorem to the analysis of algorithms, a in the

recurrence denotes the number of recursive calls. The set X(bn g(n)) contains
the function giving the running time of the algorithm, excluding recursive
calls. Note that the expression bn g(n) is general enough to describe a wide
variety of functions. However, the main restriction on the applicability of
this theorem is that f (n) is in terms of f (n − 1), so that it applies only to
those algorithms whose recursive calls reduce the size of the problem by 1.
Let us now see how Theorem 3.31 can be applied to the analysis of
MaxSuffixTD. (3.6) fits the form given in Theorem 3.31, where a = 1,
b = 1, g(n) = 1, and X = Θ. Therefore, the second case of Theorem 3.31
applies. Substituting the values for X, a, and g(n) in that solution, we
obtain f (n) ∈ Θ(n).
Knowing that MaxSuffixTD operates in Θ(n) time, we can now ana-
lyze MaxSumTD in the same way. In this case, the time required excluding
the recursive call is in Θ(n), because a call to MaxSuffixTD is made. Let-
ting f (n) denote the running time for MaxSumTD on an array of size n,
we see that
f (n) ∈ f (n − 1) + Θ(n)
for n > 0. Again, this recurrence fits the form of Theorem 3.31 with a = 1,
b = 1, g(n) = n, and X = Θ. The second case again holds, so that the
running time is in Θ(n2 ).
CHAPTER 3. ANALYZING ALGORITHMS 74

Figure 3.2 When applying divide-and-conquer, the maximum subsequence

sum may not lie entirely in either half

0 n−1

It is no coincidence that both of these analyses fit the second case of

Theorem 3.31. Note that unless a and b are both 1, Theorem 3.31 yields an
exponential result. Thus, efficient algorithms will always fit the second case
if this theorem applies. As a result, we can observe that an algorithm that
makes more than one recursive call of size n−1 will yield an exponential-time
algorithm.
We have included the first and third cases in Theorem 3.31 because they
are useful in deriving a solution for certain other types of recurrences. To
illustrate how these recurrences arise, we consider another solution to the
maximum subsequence sum problem (see Section 1.6).
The technique we will use is called divide-and-conquer. This technique,
which we will examine in detail in Chapter 10, involves reducing the size of
recursive calls to a fixed fraction of the size of the original call. For example,
we may attempt to make recursive calls on arrays of half the original size.
We therefore begin this solution by dividing a large array in half, as
nearly as possible. The subsequence giving us the maximum sum can then
lie in one of three places: entirely in the first half, entirely in the second
half, or partially in both halves, as shown in Figure 3.2. We can find the
maximum subsequence sum of each half by solving the two smaller problem
instances recursively. If we can then find the maximum sum of any sequence
that begins in the first half and ends in the second half, then the maximum
of these three values is the overall maximum subsequence sum.
For example, consider again the array A[0..5] = h−1, 3, −2, 7, −9, 7i from
Example 1.1 (page 13). The maximum subsequence sum of the first half,
namely, of A[0..2] = h−1, 3, −2i, has a value of 3. Likewise, the maximum
subsequence sum of the second half, h7, −9, 7i, is 7. In examining the two
halves, we have missed the actual maximum, A[1..3] = h3, −2, 7i, which
CHAPTER 3. ANALYZING ALGORITHMS 75

resides in neither half. However, notice that such a sequence that resides in
neither half can be expressed as a suffix of the first half followed by a prefix
of the last half; e.g., h3, −2, 7i can be expressed as h3, −2i followed by h7i.
Let us define the maximum prefix sum analogously to the maximum
suffix sum as follows:
( i−1 )
X
max A[k] | 0 ≤ i ≤ n .
k=0

It is not hard to see that the maximum sum of any sequence crossing the
boundary is simply the maximum suffix sum of the first half plus the max-
imum prefix sum of the second half. For example, returning to Example
1.1, the maximum suffix sum of the first half is 1, obtained from the suffix
h3, −2i. Likewise, the maximum prefix sum of the second half is 7, obtained
from the prefix h7i. The sum of these two values gives us 8, the maximum
subsequence sum.
Note that when we create smaller instances by splitting the array in half,
one of the two smaller instances — the upper half — does not begin with
index 0. For this reason, let us describe the input array more generally, as
A[lo..hi]. We can then modify the definitions of maximum subsequence sum,
maximum suffix sum, and maximum prefix sum by replacing 0 with lo and
n − 1 with hi. We will discuss the ranges of lo and hi shortly.
We must be careful that each recursive call is of a strictly smaller size.
We wish to divide the array in half, as nearly as possible. We begin by
finding the midpoint between lo and hi; i.e,

lo + hi
mid = .
2
Note that if hi > lo, then lo ≤ mid < hi. In this case, we can split
A[lo..hi] into A[lo..mid] and A[mid + 1..hi], and both sub-arrays are smaller
than the original. However, a problem occurs when lo = hi — i.e., when the
array contains only one element — because in this case mid = hi. In fact,
it is impossible to divide an array of size 1 into two subarrays, each smaller
than the original. Fortunately, it is easy to solve a one-element instance
directly. Furthermore, it now makes sense to consider an empty array as a
special case, because it can only occur when we begin with an empty array,
and not as a result of dividing a nonempty array in half. We will therefore
require in our precondition that lo ≤ hi, and that both are natural numbers.
We can compute the maximum suffix sum as in MaxSumBU (see Figure
1.14 on page 18), and the maximum prefix sum in a similar way. The entire
CHAPTER 3. ANALYZING ALGORITHMS 76

algorithm is shown in Figure 3.3. Note that the specification has been
changed from the one given in Figure 1.9. However, it is a trivial matter to
give an algorithm that takes as input A[0..n−1] and calls MaxSumDC if n >
0, or returns 0 if n = 0. Such an algorithm would satisfy the specification
given in Figure 1.9.
This algorithm contains two recursive calls on arrays of size ⌊n/2⌋ and
⌈n/2⌉, respectively. In addition, it calls MaxSuffixBU on an array of size
⌊n/2⌋ and MaxPrefixBU on an array of size ⌈n/2⌉. These two algorithms
are easily seen to have running times in Θ(n); hence, if f (n) denotes the
worst-case running time of MaxSumDC on an array of size n, we have

f (n) ∈ f (⌊n/2⌋) + f (⌈n/2⌉) + Θ(n) (3.7)

for n > 1.
This equation does not fit the form of Theorem 3.31. However, suppose
we focus only on those values of n that are powers of 2; i.e., let n = 2k for
some k > 0, and let g(k) = f (2k ) = f (n). Then

g(k) = f (2k )
∈ 2f (2k−1 ) + Θ(2k )
= 2g(k − 1) + Θ(2k ) (3.8)

for k > 0. Theorem 3.31 applies to (3.8), yielding g(k) ∈ Θ(k2k ). Because
n = 2k , we have k = lg n, so that

f (n) = g(k) = g(lg n). (3.9)

It is now tempting to conclude that because g(lg n) ∈ Θ(n lg n), f (n) ∈

Θ(n lg n); however, (3.9) is valid only when n is a power of 2. In order
to conclude that f (n) ∈ Θ(n lg n), we must know something about f (n) for
every sufficiently large n. However, we can show by induction on n that f (n)
is eventually nondecreasing (the proof is left as an exercise). This tells us
that for sufficiently large n, when 2k ≤ n ≤ 2k+1 , f (2k ) ≤ f (n) ≤ f (2k+1 ).
From the fact that f (2k ) = g(k) ∈ Θ(k2k ), there exist positive real numbers
c1 and c2 such that c1 k2k ≤ f (n) ≤ c2 (k + 1)2k+1 . Furthermore, because
n lg n is smooth, there is a positive real number d such that for sufficiently
large m, 2m lg(2m) ≤ dm lg m. Hence, substituting 2k for m, we have
CHAPTER 3. ANALYZING ALGORITHMS 77

Figure 3.3 Divide-and-conquer algorithm for maximum subsequence sum,

specified in Figure 1.9

Precondition: A[lo..hi] is an array of Numbers, lo ≤ hi, and both lo and

hi are Nats.
Postcondition: Returns the maximum subsequence sum of A[lo..hi].
MaxSumDC(A[lo..hi])
if lo = hi
return Max(0, A[lo])
else
mid ← ⌊(lo + hi)/2⌋; mid1 ← mid + 1;
sum1 ← MaxSumDC(A[lo..mid])
sum2 ← MaxSumDC(A[mid1..hi])
sum3 ← MaxSuffixBU(A[lo..mid]) + MaxPrefixBU(A[mid1..hi])
return Max(sum1, sum2, sum3)

Precondition: A[lo..hi] is an array of Numbers, lo ≤ hi, and both lo and

hi are Nats.
Postcondition: Returns the maximum suffix sum of A[lo..hi].
MaxSuffixBU(A[lo..hi])
m←0
// Invariant: m is the maximum suffix sum of A[lo..i − 1]
for i ← lo to hi
m ← Max(0, m + A[i])
return m

Precondition: A[lo..hi] is an array of Numbers, lo ≤ hi, and both lo and

hi are Nats.
Postcondition: Returns the maximum prefix sum of A[lo..hi].
MaxPrefixBU(A[lo..hi])
m←0
// Invariant: m is the maximum prefix sum of A[i + 1..hi]
for i ← hi to lo by −1
m ← Max(0, m + A[i])
return m
CHAPTER 3. ANALYZING ALGORITHMS 78

2k+1 (k + 1) ≤ d2k k. Putting it all together, we have

f (n) ≤ c2 (k + 1)2k+1
≤ c2 dk2k
≤ c2 dn lg n
∈ O(n lg n).

Likewise,

f (n) ≥ c1 k2k
c1 (k + 1)2k+1
≥
d
c1 n lg n
≥
d
∈ Ω(n lg n).

Thus, f (n) ∈ Θ(n lg n). The running time of MaxSumDC is therefore

slightly worse than that of MaxSumBU.
The above technique is often useful when we have a recurrence which is
not of a form for which we have a solution. More importantly, however, we
can generalize this technique to prove the following theorem; the details are
left as an exercise.

Theorem 3.32 Let a ≥ 1 and q ≥ 0 be real numbers, and let n0 ≥ 1 and

b ≥ 2 be integers. Let g : N → R≥0 be such that g ′ (n) = g(n0 bn ) is smooth.
Finally, let f : N → R≥0 be an eventually nondecreasing function satisfying

f (n) ∈ af (n/b) + X(nq g(n))

whenever n = n0 bk for a positive integer k, where X is either O, Ω, or Θ.

Then 
 q if a < bq
X(n g(n))
f (n) ∈ X(nq g(n) lg n) if a = bq


X(nlogb a ) if a > bq .

Let us first see that (3.7) fits the form of Theorem 3.32. As we have
already observed, f is eventually nondecreasing (this requirement is typically
met by recurrences obtained in the analysis of algorithms). When n = 2k ,
(3.7) simplifies to
f (n) ∈ 2f (n/2) + Θ(n).
CHAPTER 3. ANALYZING ALGORITHMS 79

Therefore, we can let n0 = 1, a = b = 2, q = 1, and g(n) = 1. This yields

g ′ (n) = g(2n ) = 1, which is smooth. Therefore, the second case applies,
yielding f (n) ∈ Θ(n lg n).
An important prerequisite for applying Theorem 3.32 is that g(n0 bn )
is smooth. Due to the exponential term, any function that satisfies this
property must be in O(lgk n) for some fixed k. This is not really a restriction,
however, because the term expressing the non-recursive part of the analysis
may be in X(nq g(n)) for arbitrary real q ≥ 0; hence, we can express most
polynomially-bounded functions. What is important is that we separate this
function into a polynomial part and a “polylogarithmic” part, because the
degree of the polynomial affects the result.

Example 3.33 Let f : N → R≥0 be an eventually nondecreasing function

such that
f (n) ∈ 3f (n/2) + Θ(n2 lg n)
whenever n = 2k for a positive integer k. We then let n0 = 1, a = 3, b = 2,
q = 2, and g(n) = lg n. Then

g ′ (n) = g(2n )
= lg 2n
=n

is smooth. We can therefore apply Theorem 3.32. Because bq = 22 = 4 and

a = 3, the first case applies. Therefore, f (n) ∈ Θ(n2 lg n).

Example 3.34 Let f : N → R≥0 be an eventually nondecreasing function

such that
f (n) ∈ 4f (n/3) + O(n lg2 n)
whenever n = 5 · 3k for a positive integer k. We then let n0 = 5, a = 4,
b = 3, q = 1, and g(n) = lg2 n. Then

g ′ (n) = g(5 · 3n )
= lg2 (5 · 3n )
= lg2 5 + n2 lg2 3

is smooth. We can therefore apply Theorem 3.32. Because bq = 3 and a = 4,

the third case applies. Therefore, f (n) ∈ O(nlog3 4 ) (log3 4 is approximately
1.26).
CHAPTER 3. ANALYZING ALGORITHMS 80

3.8 Analyzing Space Usage

As we mentioned earlier, running time is not the only performance measure
we may be interested in obtaining. For example, recall that the imple-
mentation of MaxSumTD from Figure 1.13 on page 17 terminated with a
StackOverflowError on an input of 4096 elements. As we explained in Sec-
tion 1.6, this error was caused by high stack usage due to the recursion. In
contrast, the implementation of MaxSumDC can handle an input of sev-
eral million elements, even though it, too, is recursive. In order to see why,
we can analyze the space usage of these algorithms using the techniques we
have already developed.
Let us first consider MaxSuffixTD from Figure 1.13. Because there
is no need to copy the array in order to perform the recursive call, this
algorithm requires only a constant amount of space, ignoring that needed
by the recursive call. (We typically do not count the space occupied by the
input or the output in measuring the space usage of an algorithm.) Thus,
the total space usage is given by
f (n) ∈ f (n − 1) + Θ(1) (3.10)
for n > 0. From Theorem 3.31, f (n) ∈ Θ(n).
Already this is enough to tell us why MaxSumTD has poor space per-
formance. If MaxSuffixTD requires Θ(n) space, then MaxSumTD surely
must require Ω(n) space. Furthermore, it is easily seen from the above anal-
ysis that (3.10) the space used is almost entirely from the runtime stack;
hence, the stack usage is in Θ(n). We typically would not have a runtime
stack capable of occupying space proportional to an input of 100,000 ele-
ments.
Let us now complete the analysis of MaxSumTD. Ignoring the space
usage of the recursive call, we see that MaxSumTD uses Θ(n) space, due
to the space usage of MaxSuffixTD. However, this does not mean that
the following recurrence describes the total space usage:
f (n) ∈ f (n − 1) + Θ(n)
for n > 0. The reason is that the call made to MaxSuffixTD can reuse
the space used by the recursive call. Furthermore, any calls made to Max-
SuffixTD as a result of the recursive call will be on arrays of fewer than n
elements, so they may reuse the space used by MaxSuffixTD(A[0..n − 1]).
Therefore, the total space used by all calls to MaxSuffixTD is in Θ(n).
Ignoring this space, the space used by MaxSumTD is given by
f (n) ∈ f (n − 1) + Θ(1)
CHAPTER 3. ANALYZING ALGORITHMS 81

Figure 3.4 An algorithm to add two matrices.

Precondition: A[1..m, 1..n] and B[1..m, 1..n] are arrays of Numbers, and
m and n are positive Nats.
Postcondition: Returns the sum of A[1..m, 1..n] and B[1..m, 1..n]; i.e.,
returns the array C[1..m, 1..n] such that for 1 ≤ i ≤ m and 1 ≤ j ≤ n,
C[i, j] = A[i, j] + B[i, j].
AddMatrices(A[1..m, 1..n], B[1..m, 1..n])
C ← new Array[1..m, 1..n]
for i ← 1 to m
for j ← 1 to n
C[i, j] ← A[i, j] + B[i, j]
return C[1..m, 1..n]

for n > 0, so that f (n) ∈ Θ(n). The total space used is therefore in Θ(n) +
Θ(n) = Θ(n).
Now let’s consider MaxSumDC. MaxSuffixBU and MaxPrefixBU
each use Θ(1) space. Because the two recursive calls can reuse the same
space, the total space usage is given by

f (n) ∈ f (⌈n/2⌉) + Θ(1)

for n > 1. Applying Theorem 3.32, we see that f (n) ∈ Θ(lg n). Because
lg n is such a slow-growing function (e.g., lg 106 < 20), we can see that
MaxSumDC is a much more space-efficient algorithm than MaxSumTD.
Because the space used by both algorithms is almost entirely from the run-
time stack, MaxSumDC will not have the stack problems that MaxSumTD
has.

3.9 Multiple Variables

Consider the algorithm AddMatrices shown in Figure 3.4. Applying the
techniques we have developed so far, we can easily see that the inner loop
runs in Θ(n) time. Furthermore, the outer loop iterates exactly m times,
so that the nested loops run in mΘ(n) time. It is tempting at this point
to simplify this bound to Θ(mn) using Theorem 3.8; however, we must be
careful here because we have defined asymptotic notation for single-variable
CHAPTER 3. ANALYZING ALGORITHMS 82

Figure 3.5 An algorithm illustrating difficulties with asymptotic notation

with multiple variables

Precondition: m and n are Nats.

Postcondition: true.
F(m, n)
for i ← 0 to m − 1
if i = 0
for j ← 1 to 2n
// Do nothing
else
for j ← 1 to in
// Do nothing

functions only. In this section, we discuss how to apply asymptotic analysis

to functions on more than one variable.
We would like to extend the definitions to multiple variables in as straight-
forward a manner as possible. For example, we would like for O(f (m, n))
to include all functions g : N × N → R≥0 such that for some c ∈ R>0 and For sets A and
B, A × B
n0 ∈ N, g(m, n) ≤ cf (m, n) whenever certain conditions hold. The question denotes the set
is what exactly these “certain conditions” should be. Should the inequality of all ordered
be required to hold whenever at least one of m or n is at least n0 ? Or should pairs (a, b) such
that a ∈ A and
it be required to hold only when both m and n are at least n0 ? b ∈ B.
Suppose first that we were to require that the inequality hold whenever
at least one of m or n is at least n0 . Unfortunately, a consequence of such a
definition would be that mn+1 6∈ O(mn). To see why, observe that whatever
values we choose for c and n0 , when m = 0 and n ≥ n0 , mn + 1 > cmn.
As a result, working with asymptotic notation would become much messier
with multiple variables than with a single variable.
On the other hand, only requiring the inequality to hold when both m
and n are at least n0 also presents problems. Consider, for example, how we
would analyze the rather silly algorithm shown in Figure 3.5. We can observe
that the first of the inner loops iterates 2n times, and that the second iterates
in times. However, the first loop is only executed when i = 0; hence, when
both i and n are sufficiently large, only the second loop is executed. We
could therefore legitimately conclude that the body of the outer loop runs
in O(in) time. Unfortunately, this would lead to an incorrect analysis of the
CHAPTER 3. ANALYZING ALGORITHMS 83

algorithm because the first inner loop will always execute once, assuming m
is a Nat.
Thus, we can see that if we want to retain the properties of asymptotic
notation on a single variable, we must extend it to multiple variables in
a way that is not straightforward. Unfortunately, the situation is worse
than this — it can be shown that it is impossible to extend the notation to
multiple variables in a way that retains the properties of asymptotic notation
on a single variable. What we can do, however, is to extend it so that
these properties are retained whenever the function inside the asymptotic
notation is strictly nondecreasing. Note that restricting the functions in this We say that a
function
way does not avoid the problems discussed above, as the functions inside f : N × N → R≥0
the asymptotic notation in this discussion are all strictly nondecreasing. We is strictly
therefore must use some less straightforward extension. nondecreasing if,
for every m ∈ N
The definition we propose for O(f (m, n)) considers all values of a func- and n ∈ N,
tion g(m, n), rather than ignoring values when m and/or n are small. How- f (m, n) ≤
f (m + 1, n) and
ever, it allows even infinitely many values of g(m, n) to be large in compar- f (m, n) ≤
ison to f (m, n), provided that they are not too large in comparison to the f (m, n + 1).
overall growth rate of f . In order to accomplish these goals, we first give
the following definition.

Definition 3.35 For a function f : N×N → R≥0 , we define fb : N×N → R≥0

so that
fb(m, n) = max{f (i, j) | 0 ≤ i ≤ m, 0 ≤ j ≤ n}.
Using the above definition, we can now define big-O for 2-variable func-
tions.

Definition 3.36 For a function f : N × N → R≥0 , we define O(f (m, n)) to

be the set of all functions g : N × N → R≥0 such that there exist c ∈ R>0
and n0 ∈ N so that
g(m, n) ≤ cf (m, n)
and
gb(m, n) ≤ cfb(m, n)
whenever m ≥ n0 and n ≥ n0 .
Likewise, we can define big-Ω and big-Θ for 2-variable functions.

Definition 3.37 For a function f : N × N → R≥0 , we define Ω(f (m, n)) to

be the set of all functions g : N × N → R≥0 such that there exist c ∈ R>0
and n0 ∈ N so that
g(m, n) ≥ cf (m, n)
CHAPTER 3. ANALYZING ALGORITHMS 84

and
gb(m, n) ≥ cfb(m, n)
whenever m ≥ n0 and n ≥ n0 .

Definition 3.38 For a function f : N × N → R≥0 ,

Θ(f (m, n)) = O(f (m, n)) ∩ Ω(f (m, n)).

We extend these definitions to more than two variables in the obvious

way. Using the above definitions, it is an easy matter to show that Theorem
3.11 extends to more than one variable. The proof is left as an exercise.

Theorem 3.39 Let f : N × N → R≥0 and g : N × N → R≥0 . Then

g(m, n) ∈ O(f (m, n)) iff f (m, n) ∈ Ω(g(m, n)).
We would now like to show that the theorems we have presented for single
variables extend to multiple variables, provided the functions within the
asymptotic notation are strictly nondecreasing. Before we do this, however,
we will first prove a theorem that will allow us to simplify the proofs of the
individual properties.

Theorem 3.40 Let f : N × N → R≥0 be a strictly nondecreasing function.

Then

1. O(f (m, n)) is the set of all functions g : N × N → R≥0 such that there
exist c ∈ R>0 and n0 ∈ N such that

gb(m, n) ≤ cfb(m, n)

whenever m ≥ n0 and n ≥ n0 .

2. Ω(f (m, n)) is the set of all functions g : N × N → R≥0 such that there
exist c ∈ R>0 and n0 ∈ N such that

g(m, n) ≥ cf (m, n)

whenever m ≥ n0 and n ≥ n0 .

Proof: From the definitions, for any function g(m, n) in O(f (m, n)) or in
Ω(f (m, n)), respectively, there are a c ∈ R>0 and an n0 ∈ N such that when-
ever m ≥ n0 and n ≥ n0 , the corresponding inequality above is satisfied.
We therefore only need to show that if there are c ∈ R>0 and n0 ∈ N such
CHAPTER 3. ANALYZING ALGORITHMS 85

that whenever m ≥ n0 and n ≥ n0 , the given inequality is satisfied, then

g(m, n) belongs to O(f (m, n)) or Ω(f (m, n), respectively.
We first observe that if f is strictly nondecreasing, then

fb(m, n) = f (m, n)

for all natural numbers m and n. Furthermore, for any function g : N × N →

R≥0 ,
gb(m, n) ≥ g(m, n).
Now suppose c ∈ R>0 and n0 ∈ N such that whenever m ≥ n0 and
n ≥ n0 , gb(m, n) ≤ cfb(m, n). Then for m ≥ n0 and n ≥ n0 ,

g(m, n) ≤ gb(m, n)
≤ cfb(m, n)
= cf (m, n).

Hence, g(m, n) ∈ O(f (m, n)).

Likewise, suppose now that c ∈ R>0 and n0 ∈ N such that whenever
m ≥ n0 and n ≥ n0 , g(m, n) ≥ cf (m, n). Then for m ≥ n0 and n ≥ n0 ,

gb(m, n) ≥ g(m, n)
≥ cf (m, n)
= cfb(m, n).

Therefore, g(m, n) ∈ Ω(f (m, n)).

As a result of the above theorem, in order to prove properties about

either O(f (m, n)) or Ω(f (m, n)), where f is strictly nondecreasing, we only
need to prove one of the two inequalities in the definition. Consider, for
example, the following extension to Theorem 3.8.

Theorem 3.41 Suppose f1 (m, n) ∈ O(g1 (m, n)) and f2 (m, n) ∈ O(g2 (m, n)),
where g1 and g2 are strictly nondecreasing. Then

1. f1 (m, n)f2 (m, n) ∈ O(g1 (m, n)g2 (m, n)); and

2. f1 (m, n) + f2 (m, n) ∈ O(max(g1 (m, n), g2 (m, n))).

Proof: We will only show part 1; part 2 will be left as an exercise. Because
f1 (m, n) ∈ O(g1 (m, n)) and f2 (m, n) ∈ O(g2 (m, n)), there exist positive
CHAPTER 3. ANALYZING ALGORITHMS 86

real numbers c1 and c2 and natural numbers n1 and n2 such that whenever
m ≥ n1 and n ≥ n1 ,
fb1 (m, n) ≤ c1 gb1 (m, n),
and whenever m ≥ n2 and n ≥ n2 ,

fb2 (m, n) ≤ c2 gb2 (m, n).

In what follows we will let

fd
1 f2 (m, n) = max{f1 (i, j)f2 (i, j) | 0 ≤ i ≤ m, 0 ≤ j ≤ n}.

We first observe that for any natural numbers m and n,

fd b b
1 f2 (m, n) ≤ f1 (m, n)f2 (m, n).

Furthermore, because both g1 and g2 are strictly nondecreasing, so is g1 g2 .

Let n0 = max(n1 , n2 ). Then whenever m ≥ n0 and n ≥ n0 ,

fd b b
1 f2 (m, n) ≤ f1 (m, n)f2 (m, n)
≤ c1 gb1 (m, n)c2 gb2 (m, n)
= c1 c2 gb1 (m, n)gb2 (m, n)
= cgd
1 g2 (m, n),

where c = c1 c2 . From Theorem 3.40, Corrected

2/25/11.
f1 (m, n)f2 (m, n) ∈ O(g1 (m, n)g2 (m, n)).

In a similar way, the following extension to Theorem 3.15 can be shown.

The proof is left as an exercise.

Theorem 3.42 Suppose f1 (m, n) ∈ Ω(g1 (m, n)) and f2 (m, n) ∈ Ω(g2 (m, n)),
where g1 and g2 are strictly nondecreasing. Then
1. f1 (m, n)f2 (m, n) ∈ Ω(g1 (m, n)g2 (m, n)); and

2. f1 (m, n) + f2 (m, n) ∈ Ω(max(g1 (m, n), g2 (m, n))).

We therefore have the following corollary.

Corollary 3.43 Suppose f1 (m, n) ∈ Θ(g1 (m, n)) and f2 (m, n) ∈ Θ(g2 (m, n)),
where g1 and g2 are strictly nondecreasing. Then
CHAPTER 3. ANALYZING ALGORITHMS 87

1. f1 (m, n)f2 (m, n) ∈ Θ(g1 (m, n)g2 (m, n)); and

2. f1 (m, n) + f2 (m, n) ∈ Θ(max(g1 (m, n), g2 (m, n))).

Before we can extend Theorem 3.28 to more than one variable, we must
first extend the definition of smoothness. In order to do this, we must first
extend the definitions of eventually nondecreasing and eventually positive.

Definition 3.44 Let f : N × N → R≥0 . f is said to be eventually nonde-

creasing if there is a natural number n0 such that f (m, n) ≤ f (m + 1, n)
and f (m, n) ≤ f (m, n + 1) whenever both m ≥ n0 and n ≥ n0 .

Definition 3.45 Let f : N × N → R≥0 . f is said to be eventually positive if

there is a natural number n0 such that f (m, n) > 0 whenever both m ≥ n0
and n ≥ n0 .

Definition 3.46 Let f : N × N → R≥0 be an eventually nondecreasing

and eventually positive function. f is said to be smooth if there exist a
real number c and a natural number n0 such that f (2m, n) ≤ cf (m, n) and
f (m, 2n) ≤ cf (m, n) whenever both m ≥ n0 and n ≥ n0 .
The following extension to Theorem 3.28 can now be shown — the proof
is left as an exercise.

Theorem 3.47 Let f : N × N → R≥0 be a strictly nondecreasing smooth

function. Let g : N → N be an eventually nondecreasing and unbounded
function, and let X denote either O, Ω, or Θ. Then
g(m)
X
X(f (i, n)) ⊆ X(g(m)f (g(m), n)).
i=1

Having the above theorems, we can now complete the analysis of Add-
Matrices. Because we are analyzing the algorithm with respect to two
parameters, we view n as the 2-variable function f (m, n) = n, and we view
m as the 2-variable function g(m, n) = m. We can then apply Corollary 3.43
to Θ(m)Θ(n) to obtain a running time in Θ(mn). Alternatively, because n
is smooth, we could apply Theorem 3.47 to obtain
m
X
Θ(n) ⊆ Θ(mn).
i=1
CHAPTER 3. ANALYZING ALGORITHMS 88

The results from this section give us the tools we need to analyze it-
erative algorithms with two natural parameters. Furthermore, all of these
results can be easily extended to more than two parameters. Recursive
algorithms, however, present a greater challenge. In order to analyze recur-
sive algorithms using more than one natural parameter, we need to be able
to handle asymptotic recurrences in more than one variable. This topic is
beyond the scope of this book.

3.10 Little-o and Little-ω

Occasionally, we would like to use asymptotic notation without ignoring
constant factors. Consider, for example, f (n) = 3n2 +7n+2. As n increases,
the 7n + 2 term becomes less relevant. In fact, as n increases, the ratio
3n2 /f (n) approaches 1. We might therefore wish to say that f (n) is 3n2 ,
plus some low-order terms. We would like to be able to express the fact that
these low-order terms are insignificant as n increases. To this end, we give
the following definitions.

Definition 3.48 Let f : N → R≥0 . o(f (n)) is the set of all functions o(f (n)) is
pronounced
g : N → R≥0 such that for every positive real number c, there is a natural “little-oh of f of
number n0 such that g(n) < cf (n) whenever n ≥ n0 . n”.

Definition 3.49 Let f : N → R≥0 . ω(f (n)) is the set of all functions ω(f (n)) is
pronounced
g : N → R≥0 such that for every positive real number c, there is a natural “little-omega of
number n0 such that g(n) > cf (n) whenever n ≥ n0 . f of n”.

Example 3.50 7n + 2 ∈ o(n2 ). In proof, suppose c > 0. We need to find a

natural number n0 such that 7n+2 < cn2 whenever n ≥ n0 . We first observe
that this inequality holds if n > 0 and (7 + 2/n)/c < n. The left-hand side
of this inequality is maximized when n = 1; therefore, if n ≥ ⌊9/c⌋ + 1,
7n + 2 < cn2 .
Thus, if f (n) = 3n2 + 7n + 2, then f (n) ∈ 3n2 + o(n2 ).
These definitions are similar to the definitions of O and Ω, respectively,
except that the inequalities hold for every positive real number c, rather than
for some positive real number c. Thus, g(n) ∈ o(f (n)) is a strictly stronger
statement than g(n) ∈ O(f (n)), and g(n) ∈ ω(f (n)) is a strictly stronger
statement than g(n) ∈ Ω(f (n)) (see Figure 3.6). This idea is formalized by
the following theorem.

Theorem 3.51 Let f : N → R≥0 be an eventually positive function. Then

CHAPTER 3. ANALYZING ALGORITHMS 89

Figure 3.6 Venn diagram depicting the relationships between the sets
O(f (n)), Ω(f (n)), Θ(f (n)), o(f (n)), and ω(f (n))
11111111111
00000000000
00000000000
11111111111
00000000000
11111111111
000
111 00
11
000000
111111
00000000000
11111111111
000
111
000000
111111 0000000
1111111
00
11
00000000000
11111111111
000
111
000000
111111 0000000
1111111
00
11
0000000
1111111
00000000000
11111111111
000
111
000000
111111 00
11
0000000
1111111
00000000000
11111111111
000
111
000000
111111
00000000000
11111111111
000
111 00
11
0000000
1111111
00
11
000000
111111
00000000000
11111111111
000
111
000000
111111 0000000
1111111
00
11
00000000000
11111111111
000
111 0000000
1111111
00
11
00000000000
11111111111
00
11
00
11
00
1100
11
00
11
00
11 000
111
000
111
O(f (n))

0011
1100
000
111
o(f (n))

000
111
Ω(f (n))

000
111
ω(f (n))
Θ(f (n))

1. o(f (n)) ⊆ O(f (n)) \ Θ(f (n)); and

2. ω(f (n)) ⊆ Ω(f (n)) \ Θ(f (n)),
where A \ B denotes the set of elements in A but not in B.

Proof: We will only prove part 1; the proof of part 2 is symmetric. Let
g(n) ∈ o(f (n)), and let c be any positive real number. Then there is a natural
number n0 such that g(n) < cf (n) whenever n ≥ n0 . Hence, g(n) ∈ O(f (n)).
Furthermore, because the choice of c is arbitrary, we can conclude that
g(n) 6∈ Ω(f (n)); hence, g(n) 6∈ Θ(f (n)).

It may seem at this point that the above theorem could be strengthened
to say that o(f (n)) = O(f (n)) \ Θ(f (n)) and ω(f (n)) = Ω(f (n)) \ Θ(f (n)).
Indeed, for functions f and g that we typically encounter in the analysis
of algorithms, it will be the case that if g(n) ∈ O(f (n)) \ Θ(f (n)) then
g(n) ∈ o(f (n)). However, there are exceptions. For example, let f (n) = n,
⌊lg lg n⌋
and let g(n) = 22 . Then g(n) ∈ O(f (n)) because g(n) ≤ f (n) for all
CHAPTER 3. ANALYZING ALGORITHMS 90

k k−1 √
n ∈ N. Furthermore, when n = 22 − 1 for k > 0, g(n) = 22 = n + 1;
k
hence, g(n) 6∈ Θ(f (n)). Finally, when n = 22 , g(n) = n, so g(n) 6∈ o(f (n)).
Note that we have the same duality between o and ω as between O and
Ω. We therefore have the following theorem.

Theorem 3.52 Let f : N → R≥0 and g : N → R≥0 . Then g(n) ∈ o(f (n))
iff f (n) ∈ ω(g(n)).
The following theorems express relationships between common functions
using o-notation.

Theorem 3.53 Let p, q ∈ R≥0 such that p < q, and suppose f (n) ∈ O(np )
and g(n) ∈ Ω(nq ). Then f (n) ∈ o(g(n)).

Proof: Because f (n) ∈ O(np ), there exist a positive real number c1 and a
natural number n1 such that
f (n) ≤ c1 np (3.11)
whenever n ≥ n1 . Because g(n) ∈ Ω(nq ), there exist a positive real number
c2 and a natural number n2 such that
g(n) ≥ c2 nq (3.12)
whenever n ≥ n2 . Combining (3.11) and (3.12), we have
c1 g(n)
f (n) ≤
c2 nq−p
whenever n ≥ max(n1 , n2 ). Let c be an arbitrary positive real number.
Let n0 = max(n1 , n2 , ⌈(c1 /(c2 c))1/(q−p) ⌉) + 1. Then when n ≥ n0 , nq−p >
c1 /(c2 c) because q > p. We therefore have,
c1 g(n)
f (n) ≤
c2 nq−p
< cg(n).
Therefore, f (n) ∈ o(g(n)).

Theorem 3.54 Let p and q be any positive real numbers. Then

1. O(lgp n) ⊆ o(nq ); and
2. O(np ) ⊆ o(2qn ).
The proof of Theorem 3.54 requires some additional techniques, which
we present in the next section.
CHAPTER 3. ANALYZING ALGORITHMS 91

3.11 * Use of Limits in Asymptotic Analysis

The astute reader may have noticed a relationship between asymptotic anal-
ysis and the concept of a limit. Both of these concepts involve the behavior
of a function f (n) as n increases. In order to examine this relationship
precisely, we now give the formal definition of a limit.

Definition 3.55 Let f : N → R, and let u ∈ R. We say that

lim f (n) = u
n→∞

if for every positive real number c, there is a natural number n0 such that
|f (n) − u| < c whenever n ≥ n0 . Likewise, for a function g : R≥0 → R, we
say that
lim g(x) = u
x→∞

if for every positive real number c, there is a real number x0 such that
|g(x) − u| < c whenever x ≥ x0 .
Note that for f : N → R and g : R≥0 → R, if f (n) = g(n) for every
n ∈ N, it follows immediately from the above definition that

lim f (n) = lim g(x)

n→∞ x→∞

whenever the latter limit exists. It is also possible to define infinite limits,
but for our purposes we only need finite limits as defined above. Given this
definition, we can now formally relate limits to asymptotic notation.

Theorem 3.56 Let f : N → R≥0 and g : N → R≥0 . Then

1. g(n) ∈ o(f (n)) iff limn→∞ g(n)/f (n) = 0; and

2. g(n) ∈ Θ(f (n)) if limn→∞ g(n)/f (n) = x > 0.

Note that part 1 is an “if and only if”, whereas part 2 is an “if”. The
reason for this is that there are four possibilities, given arbitrary f and g:

1. limn→∞ g(n)/f (n) = 0. In this case g(n) ∈ o(f (n)) and f (n) ∈
ω(g(n)).

2. limn→∞ f (n)/g(n) = 0. In this case f (n) ∈ o(g(n)) and g(n) ∈

ω(f (n)).
CHAPTER 3. ANALYZING ALGORITHMS 92

3. limn→∞ g(n)/f (n) = x > 0. In this case, g(n) ∈ Θ(f (n)) and f (n) ∈
Θ(g(n)). (Note that limn→∞ f (n)/g(n) = 1/x > 0.)

4. Neither limn→∞ g(n)/f (n) nor limn→∞ f (n)/g(n) exists. In this case,
we can only conclude that g(n) 6∈ o(f (n)) and f (n) 6∈ o(g(n)) — we do
not have enough information to determine whether g(n) ∈ Θ(f (n)).

Proof of Theorem 3.56:

1. This follows immediately from the definitions of limit and o.

2. Suppose limn→∞ g(n)/f (n) = x > 0. Then for every positive real
number c, there is a natural number n0 such that

x − c < g(n)/f (n) < x + c

whenever n ≥ n0 . Multiplying the above inequalities by f (n), we have

(x − c)f (n) < g(n) < (x + c)f (n).

Because these inequalities hold for every positive real number c, and
because x > 0, we may choose c = x/2, so that both x − c and x + c
are positive. Therefore, g(n) ∈ Θ(f (n)).

A powerful tool for evaluating limits of the form given in Theorem 3.56
is L’Hôpital’s rule, which we present without proof in the following theorem.

Theorem 3.57 (L’Hôpital’s rule) Let f : R≥0 → R and g : R≥0 → R be

functions such that limx→∞ 1/f (x) = 0 and limx→∞ 1/g(x) = 0. Let f ′ and
g ′ denote the derivatives of f and g, respectively. If limx→∞ g ′ (x)/f ′ (x) = We are implicitly
assuming that for
u ∈ R, then limx→∞ g(x)/f (x) = u. sufficiently large
x, the derivatives
With this theorem, we can now prove Theorem 3.54. are defined and
f ′ (x) 6= 0.
Proof of Theorem 3.54:

1. We will use L’Hôpital’s rule to show that limx→∞ lg x/xq/p = 0. It

will therefore follow that limx→∞ lgp x/xq = 0. From Theorem 3.56, it
will then follow that lgp n ∈ o(nq ). We leave it as an exercise to show
that if g(n) ∈ o(f (n)), then O(g(n)) ⊆ o(f (n)).
CHAPTER 3. ANALYZING ALGORITHMS 93

We first note that because both lg x and xq/p are nondecreasing and
unbounded (because q and p are both positive), limx→∞ 1/ lg x = 0
and limx→∞ 1/xq/p = 0. In order to compute the derivative of lg x,
we first observe that lg x ln 2 = ln x, where ln denotes the natural
logarithm or base-e logarithm, where e ≈ 2.718. Thus, the derivative
of lg x is 1/(x ln 2). The derivative of xq/p is
q
−1
qx p /p.

Using L’Hôpital’s rule,

lg x 1
lim q/p
= lim q
−1
x→∞ x x→∞ qx p x ln 2/p
p
= lim q
x→∞
qx p ln 2
= 0.

Hence, limx→∞ lgp x/xq = 0. Therefore, lgp n ∈ o(nq ) and O(lgp n) ⊆

o(nq ).

2. Because limx→∞ lgp x/xq = 0 and 2x is nondecreasing and unbounded,

it follows that

lim xp /2qx = lim lgp (2x )/(2x )q

x→∞ x→∞
= 0.

Therefore, np ∈ o(2qn ) and O(np ) ⊆ o(2qn ).

3.12 Summary
Asymptotic notation can be used to express the growth rates of functions
in a way that ignores constant factors and focuses on the behavior as the
function argument increases. We can therefore use asymptotic notation to
analyze performance of algorithms in terms of such measures as worst-case
running time or space usage. O and Ω are used to express upper and lower
bounds, respectively, while Θ is used to express the fact that the upper and
lower bounds are tight. o gives us the ability to abstract away low-order
CHAPTER 3. ANALYZING ALGORITHMS 94

Figure 3.7 Asymptotic worst-case running times of maximum subsequence

sum algorithms

Algorithm Running Time

MaxSumIter Θ(n3 )
MaxSumOpt Θ(n2 )
MaxSumTD Θ(n2 )
MaxSumDC Θ(n lg n)
MaxSumBU Θ(n)

terms when we don’t want to ignore constant factors. ω provides a dual for
o.
Analysis of iterative algorithms typically involves summations. Theorem
3.28 gives us a powerful tool for obtaining asymptotic solutions for summa-
tions. Analysis of recursive algorithms, on the other hand, typically involves
recurrence relations. Theorems 3.31 and 3.32 provide asymptotic solutions
for the most common forms of recurrences.
The analyses of the various algorithms for the maximum subsequence
sum problem illustrate the utility of asymptotic analysis. We saw that the
five algorithms have worst-case running times shown in Figure 3.7. These
results correlate well with the actual running times shown in Figure 1.15.
The results of asymptotic analyses can also be used to predict perfor-
mance degradation. If an algorithm’s running time is in Θ(f (n)), then as n
increases, the running time of an implementation must lie between cf (n) and
df (n) for some positive real numbers c and d. In fact, for most algorithms,
this running time will approach cf (n) for a single positive real number c.
Assuming that this convergence occurs, if we run the algorithm on suffi-
ciently large input, we can approximate c by dividing the actual running
time by f (n), where n is the size of the input.
For example, our implementation of MaxSumIter took 1283 seconds
to process an input of size 214 = 16,384. Dividing 1283 by (16,384)3 , we
obtain a value of c = 2.92 × 10−10 . Evaluating cn3 for n = 213 , we obtain The results of
floating-point
a value of 161 seconds. This is very close to the actual running time of 160 computations in
seconds on an input of size 213 . Thus, the running time does appear to be this discussion
converging to cn3 for sufficiently large n. are all rounded
to three
Figure 3.8 shows a plot of the functions estimating the running times significant digits.
of the various maximum subsequence sum implementations, along with the
CHAPTER 3. ANALYZING ALGORITHMS 95

Figure 3.8 Estimated performance of implementations of maximum subse-

quence sum algorithms

1020
MaxSumIter
MaxSumOpt
MaxSumTD
1015 MaxSumDC
MaxSumBU
Time in seconds

1010

105

100

10−5 10
2 215 220 225 230
Array size

measured running times from Figure 1.15. The functions were derived via
the technique outlined above using the timing information from Figure 1.15,
taking the largest data set tested for each algorithm. We have extended
both axes to show how these functions compare as n grows as large as
230 = 1,073,741,824.
For example, consider the functions estimating the running times of
MaxSumIter and MaxSumBU. As we have already shown, the function
estimating the running time of MaxSumIter is f (n) = (2.92 × 10−10 )n3 .
The function we obtained for MaxSumBU is g(n) = (1.11 × 10−8 )n. Let us
now use these functions to estimate the time these implementations would
require to process an array of 230 elements. g(230 ) = 11.9 seconds, whereas
f (230 ) = 3.61×1017 seconds, or over 11 billion years! Even if we could speed
up the processor by a factor of one million, this implementation would still
require over 11,000 years.
Though this example clearly illustrates the utility of asymptotic analysis,
a word of caution is in order. Asymptotic notation allows us to focus on
growth rates while ignoring constant factors. However, constant factors
CHAPTER 3. ANALYZING ALGORITHMS 96

Figure 3.9 Functions illustrating the practical limitations of asymptotic

notation

2250
√
n
lg16 n
2200

2150

2100

250

20
20 2100 2200 2300 2400 2500

can be relevant. For example, two linear-time algorithms will not yield
comparable performance if the hidden constants are very different.
√
For a more subtle example, consider the functions lg16 n and n, shown
√
in Figure 3.9. From Theorem 3.54, O(lg16 n) ⊆ o( n), so that as n increases,
√
lg16 n grows much more slowly than does n. However, consider n = 232 =
√
4,294,967,296. For this value, n = 216 = 65,536, whereas

lg16 n = 3216 = 1,208,925,819,614,629,174,706,176.

√
lg16 n remains larger than n until n = 2256 — a 78-digit number. After
√
that, n does grow much more rapidly than does lg16 n, but it is hard to
see any practical value in studying the behaviors of these functions at such
large values.
Finally, the running time analyses we have seen in this chapter have all
been worst-case analyses. For some algorithms, the worst case is much worse
than typical cases, so that in practice, the algorithm performs much better
than a worst-case analysis would suggest. Later, we will see other kinds of
analyses that may be more appropriate in such cases. However, we must
realize that there is a limit to what can be determined analytically.
CHAPTER 3. ANALYZING ALGORITHMS 97

3.13 Exercises
Exercise 3.1 Prove that if g(n) ∈ O(f (n)), then O(g(n)) ⊆ O(f (n)).

Exercise 3.2 Prove that for any f : N → R≥0 , f (n) ∈ Θ(f (n)).

Exercise 3.3 Prove that if f (n) ∈ O(g(n)) and g(n) ∈ O(h(n)), then
f (n) ∈ O(h(n)).

Exercise 3.4 Prove Theorem 3.15.

Exercise 3.5 For each of the following, give functions f (n) ∈ Θ(n) and
g(n) ∈ Θ(n) that satisfy the given property.

a. f (n) − g(n) ∈ Θ(n).

b. f (n) − g(n) 6∈ Θ(n).

Exercise 3.6 Suppose that g1 (n) ∈ Θ(f1 (n)) and g2 (n) ∈ Θ(f2 (n)), where
g2 and f2 are eventually positive. Prove that g1 (n)/g2 (n) ∈ Θ(f1 (n)/f2 (n)).

Exercise 3.7 Show that the result in Exercise 3.6 does not necessarily hold
if we replace Θ by O.

Exercise 3.8 Let f : N → R≥0 and g : N → R≥0 , where g is eventually

positive. Prove that f (n) ∈ O(g(n)) iff there is a positive real number c
such that f (n) ≤ cg(n) whenever g(n) > 0.
⌊lg lg n⌋
* Exercise 3.9 Let f (n) = 22 , where we assume that f (n) = 0 for
n ≤ 1.

a. Show that f (n) ∈ O(n).

b. Show that f (n) is not smooth; i.e., show that for every c ∈ R>0 and
every n0 ∈ N, there is some n ≥ n0 such that f (2n) > cf (n). [Hint:
k
Consider a sufficiently large value of n having the form 22 −1 .]

* Exercise 3.10 The goal of this exercise is to prove Theorem 3.28. Let f :
N → R≥0 be a smooth function, g : N → N be an eventually nondecreasing
and unbounded function, and h : N → R≥0 .
CHAPTER 3. ANALYZING ALGORITHMS 98

a. Show that if h(n) ∈ O(f (n)), then there exist natural numbers n0 and
n1 , a positive real number c, and a nonnegative real number d such
that for every n ≥ n1 ,
g(n) g(n)
X X
h(i) ≤ d + cf (g(n)).
i=1 i=n0

b. Use part a to prove that

g(n)
X
O(f (i)) ⊆ O(g(n)f (g(n))).
i=1

c. Show that if h(n) ∈ Ω(f (n)), then there exist natural numbers n0 and
n1 and positive real numbers c and d such that for every n ≥ n0 ,

f (n) ≥ f (2n)/d,

and for every n ≥ n1 , both

g(n) g(n)
X X
h(i) ≥ cf (⌈g(n)/2⌉)
i=1 i=⌈g(n)/2⌉

and
g(n) ≥ 2n0
hold.

d. Use part c to prove that

g(n)
X
Ω(f (i)) ⊆ Ω(g(n)f (g(n))).
i=1

e. Use parts b and d to prove that

g(n)
X
Θ(f (i)) ⊆ Θ(g(n)f (g(n))).
i=1
CHAPTER 3. ANALYZING ALGORITHMS 99

* Exercise 3.11 Prove that for every smooth function f : N → R≥0 and
every eventually nondecreasing and unbounded function g : N → N, and
every X ∈ {O, Ω, Θ},
g(n)
X
X(f (i)) 6= X(g(n)f (g(n))).
i=1

[Hint: First identify a property that every function in the set on the left-
hand side must satisfy, but which functions in the set on the right-hand side
need not satisfy.]

Exercise 3.12 Prove Theorem 3.29.

Exercise 3.13 Analyze the worst-case running time of the following code
fragments, assuming that n represents the problem size. Express your result
as simply as possible using Θ-notation.

a. for i ← 0 to 2n
for j ← 0 to 3n
k ←k+i+j

b. for i ← 1 to n2
for j ← i to i3
k ←k+1

* c. i ← n
while i > 0
for j ← 1 to i2
x ← (x + j)/2
i ← ⌊i/2⌋

Exercise 3.14 Give asymptotic solutions to the following asymptotic re-

currences. In each case, you may assume that f : N → R≥0 is an eventually
nondecreasing function.
a.
f (n) ∈ 2f (n − 1) + Θ(1)
for n > 0.
CHAPTER 3. ANALYZING ALGORITHMS 100

b.
f (n) ∈ f (n − 1) + Ω(n lg n)
for n > 0.

c.
f (n) ∈ 4f (n/2) + O(lg2 n)
whenever n = 3 · 2k for a positive integer k.

d.
f (n) ∈ 5f (n/3) + Θ(n2 )
whenever n = 3k for a positive integer k.

e.
f (n) ∈ 3f (n/2) + O(n)
whenever n = 8 · 2k for a positive integer k.

Exercise 3.15 Analyze the worst-case running time of SelectByMedian,

shown in Figure 2.7, assuming that Median is implemented to run in Θ(n)
time. Express your result as simply as possible using Θ-notation.

Exercise 3.16 Analyze the worst-case running time of the following func-
tions. Express your result as simply as possible using Θ-notation.

a. SlowSort(A[1..n])
if n = 2 and A[1] > A[2]
A[1] ↔ A[2]
else if n > 2
SlowSort(A[1..n − 1])
SlowSort(A[2..n])
SlowSort(A[1..n − 1])

b. FindMax(A[1..n])
if n = 0
error
else if n = 1
return A[1]
else
return Max(FindMax(A[1..⌊n/2⌋]), FindMax(A[⌊n/2⌋+1..n]))
CHAPTER 3. ANALYZING ALGORITHMS 101

c. FindMin(A[1..n])
if n = 0
error
else if n = 1
return A[1]
else
B ← new Array[1..⌈n/2⌉]
for i ← 1 to ⌊n/2⌋
B[i] ← Min(A[2i − 1], A[2i])
if n mod 2 = 1
B[⌈n/2⌉] ← A[n]
return FindMin(B[1..⌈n/2⌉])

Exercise 3.17 Analyze the worst-case space usage of each of the functions
given in Exercise 3.16. Express your result as simply as possible using Θ-
notation.
Changed to a
starred exercise,
* Exercise 3.18 Prove that if f : N → R≥0 is smooth and g(n) ∈ Θ(n), 2/25/11.
then f (g(n)) ∈ Θ(f (n)).

* Exercise 3.19 Prove that for any smooth function g : N → R≥0 , there is
a natural number k such that g(n) ∈ O(nk ).

* Exercise 3.20 The goal of this exercise is to prove Theorem 3.31. Let
f (n) ∈ af (n − 1) + X(bn g(n))
for n > n0 , where n0 ∈ N, a ≥ 1 and b ≥ 1 are real numbers, g(n) is a smooth
function, and X is either O, Ω, or Θ. In what follows, let n1 be any natural
number such that n1 ≥ n0 and whenever n ≥ n1 , 0 < g(n) ≤ g(n + 1).

a. Prove by induction on n that if X is O, then there is a positive real

number c such that for n ≥ n1 ,
n
X
f (n) ≤ an−n1 f (n1 ) + can (b/a)i g(i).
i=n1 +1

b. Prove by induction on n that if X is Ω, then there is a positive real

number d such that
n
X
f (n) ≥ an−n1 f (n1 ) + dan (b/a)i g(i).
i=n1 +1
CHAPTER 3. ANALYZING ALGORITHMS 102

c. Use parts a and b, together with Equation (2.2), to show that if a < b,
then f (n) ∈ X(bn g(n)).

d. Use parts a and b, together with Theorem 3.28, to show that if a = b,

then f (n) ∈ X(nan g(n)).
p
e. Suppose a > b, and let r = a/b. Show that there is a natural number
n2 ≥ n0 such that for every n ≥ n2 , 0 < g(n) ≤ g(n + 1) and
n
X r
(b/a)i g(i) ≤ .
r−1
i=n2 +1

[Hint: Use the result of Exercise 3.19 and Theorem 3.54 to show that
for sufficiently large i, g(i) ≤ ri ; then apply Equation (2.2).]

f. Use parts a, b, and e to show that if a > b, then f (n) ∈ X(an ).

Exercise 3.21 Let f : N → R≥0 be a function satisfying (3.7). Prove by

induction on n that f (n) ≤ f (n + 1) for n ≥ 1.

Exercise 3.22 Prove Theorem 3.32.

Exercise 3.23 Show that Copy, specified in Figure 1.18 on page 22, can be
implemented to run in Θ(n) time, Θ(n) space, and Θ(1) stack space, where
n is the size of both of the arrays. Note that function calls use space from the
stack, but constructed arrays do not. Also recall that the parameters A[1..n]
and B[1..n] should not be included in the analysis of space usage. Your
algorithm should work correctly even for calls like Copy(A[1..n−1], A[2..n])
(see Exercise 1.4).

Exercise 3.24 Prove Theorem 3.39.

Exercise 3.25 Complete the proof of Theorem 3.41.

Exercise 3.26 Prove Theorem 3.42.

* Exercise 3.27 Prove Theorem 3.47. [Hint: First work Exercise 3.10,
but note that not all parts of that exercise extend directly to multiple vari- Corrected
2/25/11.
ables.]
CHAPTER 3. ANALYZING ALGORITHMS 103

Exercise 3.28 Let A[1..n] be an array of numbers. An inversion is a pair

of indices 1 ≤ i < j ≤ n such that A[i] > A[j]. The number of inversions
in A is a way to quantify how nearly sorted A is — the fewer inversions A
has, the more nearly sorted it is. Let I denote the number of inversions in
A. Show that InsertionSort (Figure 1.7, page 11) runs in Θ(n + I) time
in the worst case. (Thus, InsertionSort is very efficient when the array is
nearly sorted.) Note that because the analysis is in terms of two variables,
“worst case” refers to the worst-case input for each pair of values n and I.

Exercise 3.29 Prove that if g(n) ∈ o(f (n)), then O(g(n)) ⊆ o(f (n)).

** Exercise 3.30 Find two smooth functions f : N → R≥0 and g : N →

R≥0 such that g(n) ∈ O(f (n)), but g(n) is in neither Θ(f (n)) nor o(f (n)).

Exercise 3.31 Prove that for any real numbers a > 1 and b > 1,

O(loga n) = O(logb n).

* Exercise 3.32 Prove that

lg(n!) ∈ Θ(n lg n).

3.14 Chapter Notes

Asymptotic notation predates electronic computing by several decades. Big-
O notation was introduced by Bachman [7] in 1894, but with a meaning
slightly different from our definition. In the original definition, O(f (n)) was
used to denote a specific, but unknown, function belonging to the set we
have defined to be O(f (n)). According to the original definition, it was
proper to write,
2n2 + 7n − 4 = O(n2 ).
However, one would never have written,

O(n2 ) = 2n2 + 7n − 4.

Thus, the “=” symbol was used to denote not equality, but a relation that
is not even symmetric.
Over the years, many have observed that a set-based definition, as we
have given here, is more sound mathematically. In fact, Brassard [17] claims
that as long ago as 1962, a set-based treatment was taught consistently
CHAPTER 3. ANALYZING ALGORITHMS 104

in Amsterdam. It was Brassard’s paper [17], however, that in 1985 first

made a strong case for using set-based notation consistently. Though we
are in full agreement with his position, use of the original definition is still
widespread. Alternatively, some authors give set-based definitions, then
abuse the notation by using “=” instead of “∈” or “⊆”. For a justification
of this practice, see the second edition of Knuth [76] or Cormen, et al.
[25]. For more information on the development of asymptotic notation,
including variations not discussed here, see Brassard [17]. The definitions of
asymptotic notation on multiple variables are from [64].
Knuth [76] introduced the study of the analysis of running times of al-
gorithms. The notion of a smooth function is due to Brassard [17]. Many
techniques exist for solving summations and recurrences; a good resource is
Graham, Knuth, and Patashnik [56].
Part II

Data Structures

105
Chapter 4

Basic Techniques

Algorithms must manipulate data, often in large quantities. Therefore, the

way in which an algorithm stores and accesses data can have a significant
impact on performance. The principles developed in the first three chapters
apply to data structures, but in a somewhat different way than to algorithms.
In this chapter, we will examine the ways in which top-down design, correct-
ness proofs, and performance analysis can be applied to data structures. We
will use rather simple data structures as examples. In succeeding chapters,
we will apply these techniques to more involved structures.

4.1 Stacks
One of the strengths of both top-down design and object-oriented design is
their use of abstraction to express high-level solutions to problems. In fact,
we can apply abstraction to the problems themselves to obtain high-level
solutions to many similar problems. Such high-level solutions are known
as design patterns. For example, consider the “undo” operation in a word
processor. We have some object that is undergoing a series of modifications.
An application of the “undo” operation restores the object to its state prior
to the last modification. Subsequent applications of “undo” restore the
object to successively earlier states in its history.
We have captured the essence of the “undo” operation without specifying
any of the details of the object being modified or the functionality of the
document formatter in which it will appear. In fact, our description is
general enough that it can apply to other applications, such as a spreadsheet
or the search tree viewer on this book’s web site. We have therefore specified
a design pattern for one aspect of functionality of an application.

106
CHAPTER 4. BASIC TECHNIQUES 107

Figure 4.1 The Stack ADT

Precondition: true.
Postcondition: Constructs an empty Stack.
Stack()
Precondition: true.
Postcondition: a is added to the end of the represented sequence.
Stack.Push(a)
Precondition: The stack is nonempty.
Postcondition: The last element of the represented sequence is removed
and returned.
Stack.Pop()
Precondition: true.
Postcondition: Returns true iff the represented sequence is empty.
Stack.IsEmpty()

We may apply the top-down approach to designing an implementation

of the undo operation specified above. We first observe that we need to
store a history of the edits applied to the edited object. These edits must be
represented in such a way that we can undo or invert them. Furthermore,
we only need last-in-first-out (LIFO) access to the history of edits. This
suggests that the history of edits should be stored in a stack. We can then
implement undo by popping the top edit from the stack, and applying its
inverse to the edited object. If the stack is empty, then no undo is possible.
Likewise, when edits are performed, each edit must be pushed onto the stack.
Let us now make these ideas a bit more formal. In order to be able
to reason about this solution, we must have a formal definition of a stack
and its operations. We therefore define a stack as a finite sequence of ele-
ments ha1 , . . . an i, together with the operations specified in Figure 4.1. A
mathematical structure, together with a specification of operations on that
structure, is known as an abstract data type (ADT). The operations speci-
fied are the interface of the ADT. If we have a stack S, we refer to its Pop
operation by S.Pop(). We refer to the Pop operation of the Stack ADT
by Stack.Pop(). The Stack() operation is a specification of a constructor,
CHAPTER 4. BASIC TECHNIQUES 108

Figure 4.2 Specification of the Editable ADT

Precondition: op is an EditOp.
Postcondition: Applies op to this Editable.
Editable.Apply(op)
Precondition: op is an EditOp.
Postcondition: Applies the inverse operation of op to this Editable.
Editable.ApplyInverse(op)

Figure 4.3 An algorithm to undo an editing operation

Precondition: history is a nonempty Stack containing the sequence of

EditOps performed on the Editable x.
Postcondition: The last element e has been removed from history, and
the inverse of e has been applied to x.
Undo(history, x)
e ← history.Pop()
x.ApplyInverse(e)

meaning that it is possible to construct an instance of any implementation

without supplying any arguments.
In practice, we might want additional operations such as an operation
that returns the top element without changing the sequence; however, the
above operations are sufficient for now. Later, we will add to this set.
Note that the definition of an ADT is purely mathematical; i.e., it defines
a stack as a sequence, which is a mathematical object. It says nothing
about how a stack is to be implemented. Such a mathematical definition is
sufficient for proving the correctness of Undo, shown in Figure 4.3, together
with the specification of the Editable ADT in Figure 4.2. Such a proof is,
in fact, trivial. Note that by specifying in the precondition that the stack is
nonempty, we place on the caller the responsibility of checking this condition
prior to calling Undo.
Continuing with the top-down approach, we need to design an implemen-
tation for the Stack ADT. In what follows, we will first consider a simple
CHAPTER 4. BASIC TECHNIQUES 109

approach that does not quite meet the specification. We will then consider
two full implementations of the Stack ADT.

4.2 A Simple Stack Implementation

The first step in designing an implementation for a data structure is to decide
upon a representation of the data. Perhaps the simplest representation of a
stack is an array storing the sequence of elements. Because we will want to
push additional elements onto the stack, we should use an array larger than
the number of elements. We therefore need a way to find the last element
of the sequence, or the top element of the stack. We can accomplish this by
keeping track of the number of elements in the stack.
Such a representation is a bit too simple, because the size of the array
limits the size of the stack — i.e., once we have constructed the array,
we have limited the size of the sequence we can represent. In order to
accommodate this shortcoming, we will associate with each of these stacks a
capacity, which gives the maximum number of elements it can hold. Later,
we will consider how this restriction might be removed. As a result of this
restriction, we must modify our specification of the Push operation so that
if n is strictly less than the stack’s capacity, then Push(a) adds a to the end
of the sequence. In order to be able to check this condition, we will replace
the IsEmpty operation with a more general Size operation that returns the
number of elements in the stack.
Our representation, therefore, consists of:

• an Array elements[1..M ] for some M ≥ 0; and

• a Nat size.

The value M above is not an explicit part of the representation, but we

assume that we can obtain its value by using the function SizeOf, specified
in Figure 1.20 on page 23.
In order to make sense of a given representation, we need an inter-
pretation, which relates a given set of values for the variables in the rep-
resentation to a specific instance of the formal definition. The variables
used by the interpretation may include not only the representation vari-
ables, but also variables that may be accessed using the representation
variables. In our present example, elements[1..size] describes the stack
helements[1], . . . , elements[size]i, and SizeOf(elements) gives the capacity
of the stack.
CHAPTER 4. BASIC TECHNIQUES 110

The above interpretation is problematic when size is outside the bounds

of the array. We therefore need a mechanism to ensure that the values of
a given representation are valid. To this end we use a structural invariant.
This invariant is a statement about the values of the representation variables,
and perhaps other variables that can be accessed using the representation
variables. It should be true at virtually all times. The only exception is that
we allow it to be temporarily violated while an operation is modifying the
structure, provided that it is true by the time the operation completes. The
structural invariant for our present example will be:

0 ≤ size ≤ SizeOf(elements). (4.1)

The values of the representation variables, together with all values used
by the interpretation and the structural invariant, comprise the state of the
data structure. Thus, the state of our stack implementation consists of the
value of size, the array elements, and the values stored in elements[1..size].
(We will clarify shortly the distinction between the array and the values
stored in the array.)
We can now complete our implementation by giving algorithms for the
SimpleStack constructor and operations. These algorithms are shown in
Figure 4.4.
Note that the preconditions and postconditions for the constructor and
operations are stated in terms of the definition of a stack, not in terms
of our chosen representation. For example, the precondition for the Push
operation could have been stated as,

size < SizeOf(elements).

However, preconditions and postconditions for operations on data struc-

tures should specify the externally observable behavior of the operation,
and hence should not refer to the representation of the structure. Thus,
the preconditions and postconditions still make sense even if we change the
representation.
The performance of these operations can be analyzed using the tech-
niques given in Chapter 3. It is easily seen that each of the operations
Push, Pop, and Size operate in Θ(1) time. Analysis of the constructor
is more problematic because we must include the time for constructing an
array. This time depends on such factors as how the operating system al-
locates memory and whether the elements are initialized to some default
value. For the sake of simplicity, we will assume that the memory can be
allocated in constant time, and that the array will not be initialized. Thus,
CHAPTER 4. BASIC TECHNIQUES 111

Figure 4.4 The data type SimpleStack, which does not quite implement
the Stack ADT

Structural Invariant: 0 ≤ size ≤ SizeOf(elements).

Precondition: cap is a Nat.
Postcondition: The constructed stack is empty, and its capacity is cap.
SimpleStack(cap)
size ← 0; elements ← new Array[1..cap]

Precondition: The number of elements in the represented sequence is

strictly less than the capacity.
Postcondition: a is added to the end of the represented sequence.
SimpleStack.Push(a)
if size < SizeOf(elements)
size ← size + 1; elements[size] ← a
else
error

Precondition: The represented sequence is nonempty.

Postcondition: The last element of the represented sequence is removed
and returned.
SimpleStack.Pop()
if size > 0
size ← size − 1; return elements[size + 1]
else
error

Precondition: true.
Postcondition: Returns the length of the represented sequence.
SimpleStack.Size()
return size
CHAPTER 4. BASIC TECHNIQUES 112

the time to construct a new array is in Θ(1), and the constructor operates
in Θ(1) time.
Proving correctness of operations on a data structure is similar to proving
correctness of ordinary algorithms. There are five parts:

1. Initialization: If the precondition holds at the beginning of a con-

structor invocation, then the postcondition holds upon completion of
the constructor. If the constructor terminates normally, then the struc-
tural invariant holds after the data structure has been constructed,
regardless of the truth of the precondition. (If the constructor termi-
nates abnormally, i.e., with an error condition, then we assume the
structure has not been constructed.)

2. Maintenance: If the structural invariant holds prior to the beginning

of an operation, then it holds following completion of that operation.

3. Security: If the structural invariant holds, then the state can only be
modified by invoking one of this structure’s operations.

4. Termination: Each operation and constructor terminates.

5. Correctness: If the structural invariant and the precondition hold

prior to the beginning of an operation, then the postcondition holds
following the completion of that operation.

We have already seen four of these five parts in proofs of algorithm

correctness. Security is needed not only to make sure that malicious or
untrusted code cannot violate the intended purpose of the data structure,
but also to guarantee that the structural invariant is maintained between
operations. In order to guarantee security, we need a mechanism for re-
stricting access to the representation variables. Before we can discuss this
mechanism, however, we first need to provide some details about the compu-
tational model we are assuming. We will tackle all of this shortly; however,
let us first give an example of the other four parts via a correctness proof for
SimpleStack. We will first state security as a lemma to be proved later,
then we will show that SimpleStack meets its specification.

Lemma 4.1 Security holds for SimpleStack.

Theorem 4.2 SimpleStack meets its specification.

CHAPTER 4. BASIC TECHNIQUES 113

Proof: We must show initialization, maintenance, security, termination,

and correctness.

Initialization: First, suppose the precondition to the constructor is met;

i.e., that cap is a Nat. Then the constructor will terminate normally with
size = 0, which we interpret as meaning the represented sequence is empty.
Because we interpret SizeOf(elements), which is cap, to be the capacity of
the stack, the postcondition is therefore met. Furthermore, regardless of the
truth of the precondition, if the constructor terminates without error, size
will be 0 and elements will refer to an array. Because an array cannot have a
negative number of elements, its size is at least 0. Therefore, the structural
invariant holds (0 ≤ size ≤ SizeOf(elements)).

Maintenance: Suppose the structural invariant holds prior to the execution

of an operation. We will only consider the operation Push(a); proofs for
the other two operations are left as an exercise.
The size of elements is not changed by this operation. The value of size
is only changed if it is strictly less than the size of elements. In this case,
because it is incremented by 1, the value of size will remain nonnegative,
but will not exceed the size of elements. The structural invariant therefore Corrected
2/28/10.
holds after this operation completes.

Security: Follows from Lemma 4.1.

Termination: Because there are no loops or recursive calls, all constructors

and operations terminate.

Correctness: Suppose the structural invariant holds prior to the execution

of an operation. We will only consider the operation Push(a); proofs for
the other two operations are left as an exercise.
Suppose the precondition holds, i.e., that the number of elements in the
stack is strictly less than the stack’s capacity. Because we interpret the
size of elements to be the stack’s capacity and the value of size to be the
number of elements in the stack, it follows that size < SizeOf(elements).
The if condition is therefore true. size is therefore incremented by 1, which
we interpret as increasing the number of elements on the stack by 1. a is
then assigned to elements[size], which we interpret as the last element of the
represented sequence. The postcondition is therefore met.

As the above proof illustrates, initialization, maintenance, termination,

CHAPTER 4. BASIC TECHNIQUES 114

and correctness can be shown using the techniques introduced in Chapter

2, although the specific statements to be shown are somewhat different.
Proving security, on the other hand, not only uses different techniques, it
also requires a more detailed specification of the underlying computational
model that we are assuming. We have chosen a model that is reasonably
simple and consistent, and which may be implemented easily in a variety of
programming languages. In what follows, we will give its details.
One characteristic of our model involves the way in which data items are
associated with variables. With each data item, there is a reference that
uniquely identifies it (i.e., two distinct references may not identify the same
data item). Furthermore, a reference may not itself be a data item; i.e.,
a reference may not refer to another reference. It is the reference, not the
data item itself, that will be stored in a variable. We do not specify anything
else regarding the reference, but often a reference will be implemented as
the address in memory at which the data item resides. However, we will
assume that a constant like the integer 3 also has a reference. In this case
the reference may simply be the binary encoding of 3.
Such a distinction between a data constant and its reference may seem
artificial, but it allows for a uniform treatment of variables and data. Thus,
when variable assignments are made, the reference to the assigned data
item is stored in the modified variable. Likewise, when formal parameters
take their values from actual parameters, the references are copied from the
actual parameters to the formal parameters.
Given the distinction between a data item and its reference, we can now
define more precisely the state of a SimpleStack. It must first include
the values of the reference variables, namely size (a reference to an integer)
and elements (a reference to an array). Because the interpretation uses the
values of elements[1..size], these values, which are references to arbitrary
data items, are also part of the state. However, the values of the data items
to which the references in elements[1..size] refer are not included in the state.
Thus, if elements[size] contains a reference to an array A, that reference is a
part of the state of the stack, but the contents of A are not. In particular, if
the value stored in elements[size] changes, then the stack contents change,
but if the contents of A change, the stack contents do not change — A is
still the item at the top of the stack.
Our model uses a simple hierarchical type system. Each implementation
has a unique type. This type may be a subtype of one or more interfaces (i.e.,
ADTs) that it implements. Thus, if an implementation A implements ADTs
B and C, then any instance of type A also belongs to types B and C. We do
not allow implementations to be subtypes of other implementations, so our
CHAPTER 4. BASIC TECHNIQUES 115

model includes no inheritance. Our algorithms will not always specify the
type of a data item if its type is irrelevant to the essence of the algorithm.
For example, we have not specified the type of the parameter a for the
Stack.Push operation in Figure 4.4 because we do not care what kind of
data is stored in the stack.
When the data type of a parameter is important, we can specify it in the
precondition, as we have done for the constructor in Figure 4.4. Unless we
explicitly state otherwise, when we state in a precondition that a variable
refers to an item of some particular type, we mean that this variable must
be non-nil. Note, however, that a precondition does not affect the execution
of the code. When it is important that the type actually be checked (e.g.,
for maintaining a structural invariant), we will attach a type declaration in
the parameter list, as in the two-argument constructor in Figure 4.10 (page
127). A type declaration applies to a single parameter only, so that in this
example, L is of type ConsList, but a is untyped. We interpret a type
declaration as generating an error if the value passed to that parameter is
not nil and does not refer to an instance of the declared type.
As we have already suggested, the elements of a particular data type may
have operations associated with them. Thus, each instance of the Stack
type has a Push operation and a Pop operation. For the sake of consistency,
we will consider that when a constructor is invoked, it belongs to the data
item that it is constructing. In addition, the elements of a data type may
have internal functions associated with them. Internal functions are just
like operations, but with restricted access, as described below.
In order to control the way in which a data structure can be changed, we
place the following restrictions on how representation variables and internal
functions can be accessed:
• Write access to a representation variable of an instance of data type
A is given only to the operations, constructors, and internal functions
of that instance.
• Read access to a representation variable of an instance of data type
A is given only to operations, constructors, and internal functions of
instances of type A.
• Access to an internal function of an instance of a data type A is given
only to operations, constructors, and internal functions of that in-
stance.
These restrictions are severe enough that we will often need to relax
them. In order to relax either of the first two restrictions, we can provide
CHAPTER 4. BASIC TECHNIQUES 116

accessor operations. Because we frequently need to do this, we will adopt

some conventions that allow us to avoid cluttering our algorithms with trivial
code.

• If we want to provide read access to a variable var in the represen-

tation of type A, we define the operation A.Var(), which simply re-
turns var. Using this convention, we could have omitted operation
SimpleStack.Size() from Figure 4.4.

• If we want to provide write access to var, we define the operation

A.SetVar(x), which assigns the value of x to var.

Explicitly allowing write access does not technically violate security, be-
cause any changes are made by invoking operations of the data structure.
What can be problematic is allowing read access. For example, suppose we
were to allow read access to the variable elements in the representation of
a stack. Using this reference, a user’s code could change the contents of
that array. Because this array’s contents belong to the state of the data
structure, security would then be violated. We must therefore check for the
following conditions, each of which might compromise security:

• An operation returns a reference to a portion of the state of the struc-

ture. This condition can include an operation that gives explicit read
access to a representation variable. This condition will violate security
if the reference refers to a data item whose value can change.

• An operation causes the data item to which one of its parameters refers
to be a part of the state of the structure. Under this condition, the
code that invokes the operation has a copy of the parameter, and hence
has access to the state of the structure. If the data item in question
can be changed, security is violated.

• A reference to a portion of the state is copied to the state of another

instance of the same type. For example, if S and T are of type Sim-
pleStack, and their elements variables have the same value, then the
operation S.Push(x) could change the contents of T . Thus, if a shared
data item can be changed, security is violated.

We can now illustrate the technique of proving security by proving

Lemma 4.1.
CHAPTER 4. BASIC TECHNIQUES 117

Proof of Lemma 4.1: Read access is explicitly given to size. However,

size refers to a Nat, which cannot be changed. The only other values Specifically, the
variable size can
returned are references to elements stored in the stack, and the values of receive a
these elements are not part of the state of the stack. Likewise, the only different value,
parameters are the capacity and the parameter to Push, neither of which and hence refer
to a different
refers to a data item that becomes a part of the state of the stack. Finally, Nat; however,
no operations copy any part of the state of a SimpleStack to the state of the Nats
themselves
another SimpleStack. We therefore can conclude that SimpleStack is cannot change.
secure.

Designing secure data structures is sometimes rather challenging. In fact,

there are occasions when security becomes too much of a burden — for ex-
ample, when we are designing a data structure to be used only locally within
some algorithm. In such a case, it may be easier to prove that our algorithm
doesn’t violate the security of the data structure, rather than to prove that
such a violation is impossible. If we define a data structure for which we
can prove initialization, maintenance, termination, and correctness, we say
that this structure is insecure, but otherwise satisfies its specification.
Together, initialization, maintenance, and security are almost sufficient
to prove that the structural invariant holds between execution of any opera-
tions on the structure. The only caveat is similar to the difficulty associated
with mutual recursion, as was discussed in Section 2.5. Suppose that dur-
ing execution of some operation x.Op1, a function call is made while the
structural invariant is false. Suppose that through some sequence of nested
calls, some operation x.Op2 is then called (see Figure 4.5). At this point,
the structural invariant is false, and the operation’s correctness cannot be
guaranteed. This scenario is known as a callback. Note that it does not
matter how long the sequence of nested calls is. In particular, the function
call made by x.Op1 may be a direct call to x.Op2, or it may be a recursive
call.
Though callbacks are common in software systems, they are more prob-
lematic than beneficial in the design of data structures. Furthermore, as is
the case for mutual recursion, callbacks may be impossible to detect when
we are designing a single data structure, as we may need to call a function
whose implementation we don’t have. For these reasons, we will assume that
if a callback is attempted, a runtime error results. Our correctness proofs A locking
mechanism can
will then rest on the assumption that data structures and algorithms are be used to cause
not combined in such a way as to result in a callback. any callback to
Given this assumption, once initialization, maintenance, and security generate a
runtime error.
are shown, it can be shown by induction that the structural invariant holds
CHAPTER 4. BASIC TECHNIQUES 118

Figure 4.5 Illustration of a callback — when x.Op2() is called, the struc-

tural invariant of x is false

x: Op1()

// Structural Invariant violated

y.Op3()

// Structural Invariant reestablished

Op2()

y: Op3()

z.Op4()

z: Op4()

x.Op2()
CHAPTER 4. BASIC TECHNIQUES 119

between execution of any operations that construct or alter an instance of

the data structure. Note that the structural invariant will hold after the
structure is constructed, regardless of whether preconditions to operations
are met. Thus, we can be convinced that a structural invariant holds even
if operations are not invoked properly. Because we can know that the struc-
tural invariant holds, we can then use it in addition to the precondition in
proving the correctness of an individual operation.

4.3 Expandable Arrays

SimpleStack does not quite implement the Stack ADT because each Sim-
pleStack must be constructed with a fixed capacity. We will now present
a true implementation Stack.
We can use the basic idea from SimpleStack if we have a way to perform
a Push when the array is full. Our solution will be to construct a new, larger
array, and to copy the elements in the stack to this new array. This new
array then replaces the original array. We now have room to add the new
element.
Though this idea is simple, there are some performance issues to consider.
In particular, Θ(n) time is required to copy n elements from one array to
another. We might be willing to pay this performance price occasionally, but
if we are not careful, the overall performance of manipulating a stack may be
significantly degraded. Suppose, for example, that the new array contains
only one more location than the original array. Consider what happens if we
push n elements onto such a stack, where n is much larger than the size of
the original array. After the original array is filled, each Push requires Θ(i)
time, where i is number of elements currently in the stack. It is not hard to
see that the total time for pushing all n elements is in Θ(n2 ), assuming the
size of the original array is a fixed constant.
In order to avoid this bad performance, when we need to allocate a new
array, we should make sure that it is significantly larger than the array we
are replacing. As we will see shortly, we can achieve this goal by doubling the
size of the array. The ExpandableArrayStack implementation shown in
Figure 4.6 implements this idea. In order for this to work, however, the size
of the array must always be nonzero; hence, we will need to include this
restriction in the structural invariant. Note that we have added a construc-
tor that was not specified in the interface (Figure 4.1). The no-argument
constructor simply invokes this new constructor with a default value for its
argument.
CHAPTER 4. BASIC TECHNIQUES 120

Figure 4.6 ExpandableArrayStack implementation of Stack

Structural Invariant: 0 ≤ size ≤ SizeOf(elements), and

SizeOf(elements) > 0.

ExpandableArrayStack()
ExpandableArrayStack(10)

Precondition: cap is a strictly positive Nat.

Postcondition: An empty stack is constructed.
ExpandableArrayStack(cap)
if cap > 0
size ← 0; elements ← new Array[1..cap]
else
error

ExpandableArrayStack.Push(a)
if size = SizeOf(elements)
el ← new Array[1..2 · size]
for i ← 1 to size
el[i] ← elements[i]
elements ← el
size ← size + 1; elements[size] ← a

ExpandableArrayStack.Pop()
if size > 0
size ← size − 1
return elements[size + 1]
else
error

ExpandableArrayStack.IsEmpty()
return size = 0
CHAPTER 4. BASIC TECHNIQUES 121

At this point it is tempting to apply the top-down design principle by

defining an ADT for an expandable array. However, when an idea is sim-
ple enough, designing an ADT often becomes more cumbersome than it is
worth. To attain the full functionality of an expandable array, we would
need operations to perform each of the following tasks:
• reading from and writing to arbitrary locations;
• obtaining the current size of the array; and
• explicitly expanding the array (we might want to do this before the
array is full) to a size we choose.
Furthermore, we might wish to redistribute the data in the larger array when
we expand it. It therefore seems best to characterize the expandable array
design pattern as the practice of moving data from one array to a new one
of at least twice the size whenever the current array becomes too full.
Clearly, the worst-case running time of the Push operation shown in
Figure 4.6 is in O(n), where n is the number of elements in the stack.
Furthermore, for any n > 0, if we construct a stack with the constructor call
ExpandableArrayStack(n), then when the (n + 1)-st element is pushed
onto the stack, Ω(n) time is required. Therefore, the worst-case running
time of the Push operation is in Θ(n). All other operations clearly require
Θ(1) time.
The above analysis seems inadequate because in any actual use of a
stack, the Θ(n) behavior will occur for only a few operations. If an Ex-
pandableArrayStack is used by some algorithm, the slow operations
may be few enough that they do not significantly impact the algorithm’s
overall performance. In such a case, it makes sense to consider the worst-
case performance of an entire sequence of operations, rather than a single
operation. This idea is the basis for amortized analysis.
With amortized analysis, we consider an arbitrary sequence of opera-
tions performed on an initially empty data structure. We then do a form of
worst-case analysis of this sequence of operations. Clearly, longer sequences
will have longer running times. In order to remove the dependence upon
the length of the sequence, we amortize the total running time of the se-
quence over the individual operations in the sequence. For the time being,
this amortization will simply be to compute the average running time for an
individual operation in the sequence; later in this chapter we will generalize
this definition. The analysis is still worst-case because the sequence is arbi-
trary — we are finding the worst-case amortized time for the operations on
the data structure.
CHAPTER 4. BASIC TECHNIQUES 122

For example, consider any sequence of n operations on an initially empty

stack constructed with ExpandableArrayStack(k). We first analyze the
worst-case running time of such a sequence. We can use the techniques
presented in Chapter 3, but the analysis is easier if we apply a new technique.
We first analyze the running time ignoring all iterations of the for loop in
the Push operation and any loop overhead that results in the execution of
an iteration. Having ignored this code, it is easily seen that each operation
requires Θ(1) time, so that the entire sequence requires Θ(n) time.
We now analyze the running time of all iterations of the for loop through-
out the entire sequence of operations. In order to accomplish this, we must
compute the total number of iterations. The array will be expanded to size
2k the first time the stack reaches size k + 1. For this first expansion, the
for loop iterates k times. The array will be expanded to size 4k the first
time the stack reaches size 2k + 1. For this expansion, the loop iterates 2k
times. In general, the array will be expanded to size 2i+1 k the first time
the stack reaches size 2i k + 1, and the loop will iterate 2i k times during this
expansion. Because the sequence contains n operations, the stack can never
exceed size n. Therefore, in order to compute an upper bound on the total
number of iterations, we must sum 2i k for all i ≥ 0 such that

2i k + 1 ≤ n
2i ≤ (n − 1)/k
i ≤ lg(n − 1) − lg k.

The total number of iterations is therefore at most

⌊lg(n−1)−lg k⌋ ⌊lg(n−1)−lg k⌋
X X
2i k = k 2i
i=0 i=0
⌊lg(n−1)−lg k⌋+1
= k(2 − 1) by (2.2)
lg(n−1)−lg k+1
≤ k2
k2lg(n−1)+1
=
2lg k
= 2(n − 1).

Because each loop iteration requires Θ(1) time, the time required for all loop
iterations is in O(n). Combining this result with the earlier analysis that
ignored the loop iterations, we see that the entire sequence runs in Θ(n)
time.
CHAPTER 4. BASIC TECHNIQUES 123

Now to complete the amortized analysis, we must average the total run-
ning time over the n operations in the sequence. By Exercise 3.6 on page 97,
if f (n) ∈ Θ(n), then f (n)/n ∈ Θ(1). Therefore, the worst-case amortized
time for the stack operations is in Θ(1). We conclude that, although an
individual Push operation may be expensive, the expandable array yields
a stack that performs well on any sequence of operations starting from an
initially empty stack.

4.4 The ConsList ADT

On this textbook’s web site is a search tree viewer that allows users to
insert and remove strings from various kinds of search trees and to view the
results. Included are “Back” and “Forward” buttons that allow the user to
step through the history of the trees created. Two stacks are used, one to
store the history, and one to store the “future” after the user has stepped
back into the history. Also included is a “Clone” button, which causes
an identical window to be created, with identical history and future. This
new window can be manipulated independently from the first. In order to
accomplish this independence, the two stacks must be cloned.
Let us consider how we might clone an ExpandableArrayStack. In
order to simplify the discussion, we will restrict our attention to shallow
cloning. Shallow cloning consists of cloning only the state of the structure,
and not the items contained in the structure. Thus, if a data item is stored
in a stack which is then cloned, any subsequent changes to that data item
will be reflected in both stacks. However, changes to one of the stacks will
not affect the other. In order to perform a shallow clone of an Expand-
ableArrayStack, the array must clearly be copied, so that the two stacks
can be manipulated independently. Copying one array to another requires
Θ(n) time, where n is the number of elements copied.
We might be able to improve on this running time if we can use a data
structure that facilitates nondestructive updates. An update is said to be
nondestructive if it does not change any of the existing structure, but instead
builds a new structure, perhaps using some or all of the existing structure.
If all updates are nondestructive (i.e., the structure is immutable), it is
possible for different structures to share substructures that are common to
both. This sharing can sometimes lead to improved efficiency; for example,
to clone an immutable structure all that we need to copy is the reference to
it.
In order to apply this idea to stacks, it is helpful to think of a finite
CHAPTER 4. BASIC TECHNIQUES 124

sequence as nested ordered pairs. In particular, a sequence of length n > 0

is an ordered pair consisting of a sequence of length n − 1 followed by an
element. As a special case, the sequence of length 0 is denoted (). Thus,
the sequence ha1 , a2 , a3 i can be thought of as the pair ((((), a1 ), a2 ), a3 ). If
we think of this sequence as a Stack S, then we can think of S.Push(a4 )
as a function returning a new sequence (((((), a1 ), a2 ), a3 ), a4 ). Note that
this new sequence can be constructed simply by pairing S with a4 , leaving
S unchanged.
Nested pairs form the basic data structure in the programming language
Lisp and its derivatives. The Lisp function to build an ordered pair is called
cons. Based on this background is the ADT known as a ConsList. It is
useful to think of a nonempty ConsList as a pair (head, tail), where head
is an element and tail is a ConsList. (Note that the two components of
the pair are in the reverse order of that described in the above paragraph.)
More formally, we define a ConsList to be a finite sequence ha1 , . . . , an i,
together with the operations specified in Figure 4.7. Note that none of these We use Bool to
denote the type
operations changes the ConsList. We therefore say that a ConsList is an whose only
immutable structure, meaning that though the elements in the sequence may values are true
change their state, the sequence itself will not change. and false.

In what follows, we will show how to implement Stack using a Cons-

List. We will have thus applied top-down design to the task of implementing
Stack, as we will have reduced this problem to the problem of implementing
ConsList. The resulting Stack implementation will support constant-time
Push, Pop, and shallow cloning, which we will support via an additional
constructor. We will then complete the design by showing how to implement
ConsList.
Our Stack representation will be a ConsList elements. We interpret
elements as storing the stack in reverse order; i.e., the head of elements is
the top element on the stack. The structural invariant will be that elements
refers to a ConsList. The implementation is shown in Figure 4.8. The
one-argument constructor is used to construct a shallow clone.
Again, all constructors and operations can easily be seen to run in Θ(1)
time, and proving initialization, maintenance, termination, and correctness
is straightforward. Regarding security, we note that elements is shared when
the one-argument constructor is used; however, because a ConsList is im-
mutable, this sharing cannot violate security. The full correctness proof is
left as an exercise.
We will now complete the implementation of ConsListStack by im-
plementing ConsList. Our representation consists of the following:
CHAPTER 4. BASIC TECHNIQUES 125

Figure 4.7 The ConsList ADT

Precondition: true
Postcondition: Constructs a ConsList representing an empty sequence.
ConsList()
Precondition: L is a ConsList ha1 , . . . an i.
Postcondition: Constructs a ConsList representing the sequence
ha, a1 , . . . , an i.
ConsList(a, L)
Precondition: true.
Postcondition: Returns a Bool that is true iff the represented sequence
is empty.
ConsList.IsEmpty()
Precondition: The represented sequence is nonempty.
Postcondition: Returns the first element of the sequence.
ConsList.Head()
Precondition: The represented sequence ha1 , . . . an i is nonempty.
Postcondition: Returns a ConsList representing the sequence
ha2 , . . . , an i.
ConsList.Tail()

• a readable Bool isEmpty;

• a readable element head; and

• a readable ConsList tail.

If isEmpty is true, then we interpret the ConsList to represent an empty
sequence. Otherwise, we interpret head as the first element of the sequence,
and tail as the remainder of the sequence. As our structural invariant, we
require a ConsList to represent a finite sequence according to the above
interpretation. The representation of the ConsList ha1 , a2 , a3 , a4 i is illus-
trated in Figure 4.9.
Because our representation fits so closely with the definition of a Cons-
List, the implementation is trivial — the operations are simply the acces-
CHAPTER 4. BASIC TECHNIQUES 126

Figure 4.8 ConsListStack implementation of Stack

Structural Invariant: elements refers to a ConsList.

ConsListStack()
elements ← new ConsList()

Precondition: S refers to a ConsListStack.

Postcondition: The constructed stack is a shallow clone of the stack S.
ConsListStack(S)
elements ← S.elements

ConsListStack.Push(a)
elements ← new ConsList(a, elements)

ConsListStack.Pop()
if elements.IsEmpty()
error
else
top ← elements.Head(); elements ← elements.Tail()
return top

ConsListStack.IsEmpty()
return elements.IsEmpty()

Figure 4.9 An illustration of the representation of a ConsList

a1 a2 a3 a4
CHAPTER 4. BASIC TECHNIQUES 127

Figure 4.10 Implementation of ConsList

Structural Invariant: The ConsList represents a finite sequence.

ConsList()
isEmpty ← true

ConsList(a, L : ConsList)
if L = nil
error
else
isEmpty ← false; head ← a; tail ← L

sors for the three representation variables. Note that our specification says
nothing about the contents of head and tail when isEmpty is true; hence,
if these accessors are called for an empty list, arbitrary values may be re-
turned. The implementation is shown in Figure 4.10. Because we will only
present a single implementation of ConsList, we use the same name for
the implementation as for the interface.
It is easily seen that each constructor and operation runs in Θ(1) time.
We will now prove that the implementation meets its specification.

Theorem 4.3 The ConsList implementation meets its specification.

Proof:

Initialization: We consider the two constructors as separate cases.

Case 1: ConsList(). Because isEmpty is set to true, we interpret the

constructed ConsList as representing an empty sequence. The structural
invariant and postcondition therefore hold.

Case 2: ConsList(a, L : ConsList). If this constructor terminates nor-

mally, then L refers to a ConsList. Let us therefore assume that this is the
case. Let L represent the sequence ha1 , . . . , an i. isEmpty is set to false,
so the constructed instance is interpreted to be the nonempty sequence
ha, a1 , . . . , an i. Because this is a finite sequence, the structural invariant
CHAPTER 4. BASIC TECHNIQUES 128

and postcondition both hold.

Maintenance: Because no operations change any representation variables,

maintenance holds trivially.

Security: Read access is explicitly given to the three representation vari-

ables. However, isEmpty and tail are immutable, so this read access cannot
result in changes to either of them. Because head refers to a data item that
is not a part of the state, changes that may result from reading this reference
do not affect the security of the ConsList. Finally, although the parameter
L to the two-argument constructor is copied to a representation variable,
because it refers to an immutable data item, security is not violated.

Termination: Because there are no loops or recursion, all constructors and

operations terminate.

Correctness: The only operations simply provide read access, and so are
trivially correct.

Example 4.4 Suppose we construct an empty ConsListStack S, then

push data items a1 , a2 , and a3 in sequence onto S. Figure 4.11(a) illus-
trates the result of these operations. Suppose we then construct T using
ConsListStack(S). At this point T.elements is equal to S.elements. If we
then execute T.Pop() twice, T.elements is assigned T.elements.Tail().Tail(),
as shown in Figure 4.11(b). Notice that this does not affect the contents of
S. If we then push a4 onto T , we obtain the result shown in Figure 4.11(c).
Again, the contents of S are unchanged.
We conclude our discussion of ConsLists by noting that there are some
disadvantages to this implementation of Stack. The running time of the
Push operations is likely to be slower than either of the other implementa-
tions because new memory is always allocated. Furthermore, this memory
is never explicitly released, so this implementation should only be coded in
a language that provides automatic garbage collection.
The idea behind a ConsList can be modified to form a mutable struc-
ture if we allow the value of tail to be modified. It is difficult to define a
general-purpose ADT based on this idea other than to allow write access to
tail. If we do this, then there is little security; i.e., the user can construct
complicated linked structures that share data or perhaps form loops. Never-
CHAPTER 4. BASIC TECHNIQUES 129

Figure 4.11 Example of stacks implemented with ConsLists

S.elements S.elements T.elements

a3 a2 a1 a3 a2 a1
(a) Stack S after 3 pushes (b) After cloning and popping twice

T.elements

S.elements
a4

a3 a2 a1
(c) After pushing a4 onto T

theless, if we are careful, we can use this idea as a building block for several
more advanced data structures. We will therefore refer to this idea as the
linked list design pattern.

4.5 Amortized Analysis Using Potential Functions

In Section 4.3, we introduced the technique of amortized analysis. The ac-
tual analysis was rather straightforward, mainly because the worst case is
easily identifiable. For many data structures, amortized analysis is not so
straightforward. Furthermore, we would like to be able to amortize in a
more general way than simply averaging over all operations in a sequence.
Specifically, we would like to be able to amortize in such a way that op-
erations on small structures receive smaller amortized cost than operations
on large structures. For example, if n represents the size of a structure, we
would like to be able to speak about an amortized running time in O(lg n).
In this section, we introduce a more general notion of amortized cost and
present a corresponding technique for performing amortized analysis.
In order to motivate this technique, it is helpful to think of amortized
analysis using an analogy. Suppose we have a daily budget for gasoline for
CHAPTER 4. BASIC TECHNIQUES 130

a car. We want to track our gasoline purchases to ensure that we don’t

exceed this budget. Further suppose that we begin tracking these expenses
when the tank is full. We may then have several days in which we spend
no money on gasoline. At some point, we will refill the tank, thus incurring
a large expense which may be greater than our daily budget. However, if
we amortize this cost over all of the days since we last filled the tank, we
will hopefully find that we have remained within our daily budget for each
of these days.
One way to monitor our budget more closely is to consider the potential
cost of filling the tank at the end of each day. Specifically, suppose that
we have a very precise gas gauge on our car. In order to keep the analogy
simple, we will also suppose that the cost of gasoline remains constant. At
the end of each day, we could then measure the amount of gasoline in the
tank and compute the cost of potentially filling the tank at that point.
For example, suppose gasoline costs $3 per gallon. Further suppose that
on consecutive days, we find that our 10-gallon tank contains 8 gallons and
6.5 gallons, respectively. On the first day, the potential cost of filling the
tank was $6, as the tank would hold 2 additional gallons (see Figure 4.12).
On the second day, the potential cost was $10.50, as the tank would hold 3.5
additional gallons. Assuming that no gasoline was added to the tank that
day, the cost of the gasoline used that day was then $4.50 — the difference
in the two potential costs.
On days in which we fill the tank, the computation is only slightly more
complicated. Note that on these days, the potential cost is likely to decrease.
For example, suppose that the previous day’s level (day 4 in Figure 4.12)
was 3 gallons, today’s level is 9 gallons, and that we spent $24 to purchase 8
gallons of gasoline. The potential cost of filling the tank has decreased from
$21 to $3; hence, the change in potential cost is negative $18. However, we
should include the cost of the gasoline we actually purchased, resulting in
an amortized cost of $6 for that day. In general, we compute the amortized
cost by adding the actual cost to the change in potential cost.
Note that this amortization process is somewhat pessimistic, as we are
assessing costs before we actually incur them; however, it is a safe way of
verifying our budget. Specifically, suppose we sum up the amortized costs
for any sequence of days, beginning with a day in which the tank is full.
The sum of changes in potential costs will be the net change in potential
cost. Because the tank is initially full, the initial potential cost is 0; hence
the net change in potential cost is the final potential cost. The remainder of
the sum of amortized costs is the sum of actual costs of gasoline purchases.
Thus, the sum of amortized costs is the sum of actual costs plus the final
CHAPTER 4. BASIC TECHNIQUES 131

Figure 4.12 Example of actual, potential, and amortized costs for gasoline

$30
$27
$24
$21
$18
$15
$12
$9
$6
$3
$0
Day 1 Day 2 Day 3 Day 4 Day 5 Day 6

Actual cost
Potential cost to fill tank
Amortized cost: actual plus change in potential

potential cost. Because the potential cost can never be negative (the tank
can’t be “overfull”), the sum of the amortized costs will be at least the sum
of the actual costs.
Let us now consider how we might apply this technique to the amortized
analysis of a data structure such as an ExpandableArrayStack. The
potential gasoline cost is essentially a measure of how “bad” the state of
the gas tank is. In a similar way, we could measure how “bad” the state
of an ExpandableArrayStack is by considering how full the array is —
the closer the array is to being filled, the closer we are to an expensive
operation. We can formalize this measure by defining a potential function
Φ, which maps states of a data structure into the nonnegative real numbers,
much like the potential gasoline cost maps “states” of the gas tank into
CHAPTER 4. BASIC TECHNIQUES 132

nonnegative real numbers.

As we assumed our potential gasoline cost to be initially 0, so also we
require that Φ maps the initial state (usually an empty data structure) to
0. Each operation, by changing the state of the data structure, also changes
the value of Φ. An increase in Φ corresponds to using gasoline, whereas
a decrease in Φ corresponds to adding gasoline to the tank (though not
necessarily filling it, as Φ might not reach 0). Thus, for ExpandableAr-
rayStack, we would want the potential function to increase when a Push
that does not expand the array is performed, but to decrease when either a
Pop or a Push that expands the array is performed.
Let σ denote the state of a data structure prior to some operation, and
let σ ′ denote the state of that structure following the operation. We then
define the change in Φ to be Φ(σ ′ ) − Φ(σ). We further define the amortized
cost of the operation relative to Φ to be the actual cost plus the change in
Φ.
The above defines what a potential function is and suggests how it might
be used to perform amortized analysis. It does not, however, tell us precisely
how we can obtain a potential function. We will address this issue in detail
shortly; for now however, we will show that an amortized analysis using any
valid potential function will give a true upper bound on amortized cost. The
proof is essentially the same argument that we gave justifying our use of the
potential gasoline cost for amortized analysis.

Theorem 4.5 Let Φ be a valid potential function for some data structure;
i.e., if σ0 is the initial state of the structure, then Φ(σ0 ) = 0, and if σ is any
state of the structure, then Φ(σ) ≥ 0. Then for any sequence of operations
from the initial state σ0 , the sum of the amortized costs of the operations
relative to Φ is at least the sum of the actual costs of the operations.

Proof: Let o1 , . . . , om be a sequence of operations from σ0 , and let σi be

the state of the data structure after operation oi has been performed. Also,
CHAPTER 4. BASIC TECHNIQUES 133

let ci be the actual cost of oi applied to state σi−1 . Then

m
X m
X m
X m−1
X
(ci + Φ(σi ) − Φ(σi−1 )) = ci + Φ(σi ) − Φ(σi )
i=1 i=1 i=1 i=0
Xm
= ci + Φ(σm ) − Φ(σ0 )
i=1
Xm
≥ ci
i=1

because Φ(σm ) ≥ 0 and Φ(σ0 ) = 0.

This notion of amortized cost is therefore meaningful and more general

than the one introduced in Section 4.3. Specifically, because an amortized
cost is defined for each operation, we can analyze this cost much like we
would analyze the actual running time of an operation. Note, however, that
this notion of amortization only provides an upper bound. For this reason,
we only use O-notation (not Ω or Θ) when we perform this type of analysis.
This technique can now be used to analyze the amortized performance
of ExpandableArrayStack. However, finding an appropriate potential
function for this analysis turns out to be a bit tricky. As Theorem 4.5
implies, we can perform an amortized analysis using any valid potential
function; however, a poor choice of potential function may result in a poor
upper bound on the amortized cost. For example, we could choose as our
potential function the constant function 0 — i.e., for each state, the po-
tential is 0. This function meets the requirements of a potential function;
however, because the change in potential will always be 0, the amortized cost
relative to this potential function is the same as the actual cost. Finding
a potential function that yields constant amortized cost for the Expand-
ableArrayStack operations requires a bit of insight.
For this reason, before we give a potential-function analysis for Expand-
ableArrayStack, we will begin with a simpler example. In particular,
consider the ADT BinaryCounter, specified in Figure 4.13. A Bina-
ryCounter maintains a value that is initially 1. The Increment opera-
tion can be used to increment the value by 1, and the Value operation can
be used to retrieve the value as a ConsList of 1s and 0s, least significant
bit first. It is unlikely that this ADT has any useful purpose; however, the
implementation shown in Figure 4.14 yields an amortized analysis that is
simple enough to illustrate clearly the potential-function technique.
CHAPTER 4. BASIC TECHNIQUES 134

Figure 4.13 Specification of the BinaryCounter ADT

Precondition: true.
Postcondition: Constructs a counter with value 1.
BinaryCounter()
Precondition: true.
Postcondition: Increases the counter value by 1.
BinaryCounter.Increment()
Precondition: true.
Postcondition: Returns a ConsList of 0s and 1s ending with a 1 and giv-
ing the binary representation of the counter value, with the least significant
bit first; i.e. if the sequence is ha0 , . . . , an i, the value represented is
n
X
ai 2i .
i=0

BinaryCounter.Value()

This implementation uses a single readable representation variable, value.

The structural invariant states that value refers to the ConsList specified
by the Value operation. We leave it as an exercise to show that this imple-
mentation satisfies its specification.
Let us now analyze the worst-case running time of Increment. In the
worst case, the first loop can iterate n times, where n is the length of value.
This case occurs when value consists entirely of 1s; however, when value
begins with a 0, this loop will not iterate at all. It is easily seen that the
second loop always iterates the same number of times as the first loop; hence,
in the worst case, the Increment operation runs in Θ(n) time, or in Θ(lg v)
time, where v is the value represented.
We wish to show that the amortized costs of the IterBinCounter op-
erations are in O(1). We first need to identify the actual costs. Observe that
the Value operation runs in Θ(1) time and does not change the structure;
hence, we can ignore this operation (we will be ignoring only some constant
time for each operation). Because the two loops in Increment iterate the
same number of times, the running time of Increment is proportional to
CHAPTER 4. BASIC TECHNIQUES 135

Figure 4.14 IterBinCounter implementation of BinaryCounter,

specified in Figure 4.13

Structural Invariant: value is a ConsList of 0s and 1s ending with a 1.

The sequence represented by this ConsList gives the current value of the
BinaryCounter in binary, with the least significant bit first.

IterBinCounter()
value ← new ConsList(1, new ConsList())

IterBinCounter.Increment()
k ← 0; c ← value
// Invariant: value contains k 1s, followed by c.
while not c.IsEmpty() and c.Head() = 1
k ← k + 1; c ← c.Tail()
if c.IsEmpty()
c ← new ConsList(1, c)
else
c ← new ConsList(1, c.Tail())
// Invariant: c contains i − 1 0s and a 1, followed by the ConsList
// obtained by removing all initial 1s and the first 0 (if any) from value.
for i ← 1 to k
c ← new ConsList(0, c)
value ← c

the number of iterations of the while loop, plus some constant. We can
therefore use the number of iterations of the while loop as the actual cost
of this operation. Note that the actual cost varies from 0 to n, depending
on the current value represented.
The next step is to define an appropriate potential function. This step is
usually the most challenging part of this technique. While finding a suitable
potential function requires some creativity, there are several guidelines we
can apply.
First, we can categorize operations of a data structure according to two
criteria relevant to amortized analysis:

• the actual cost of the operation; and

CHAPTER 4. BASIC TECHNIQUES 136

• how much it degrades or improves future performance of the data

structure.

Using the above criteria, we can divide operations into four categories:

1. Operations that cost little and improve future performance. IterBin-

Counter contains no such operation; however, we needn’t be too con-
cerned with operations of this type because they cause no problems
for our analysis.

2. Operations that cost little but degrade future performance. The In-
crement operation when the head of value is 0 is an example of this
type. It performs no loop iterations, but causes value to have at least
one leading 1, so that the next Increment will perform at least one
iteration.

3. Operations that cost much but improve future performance. The In-
crement operation when value has many leading 1s is an example of
this type. It performs a loop iteration for each leading 1, but replaces
these leading 1s with 0s. Thus, the next Increment will not perform
any iterations. In fact, a number of Increment operations will be
required before we encounter another expensive one.

4. Operations that cost much and degrade future performance. Our

IterBinCounter includes no operations of this type. In fact, op-
erations of this type usually make amortized analysis futile.

The key to finding an appropriate potential function is in striking a good

balance between operations of types 2 and 3 above. Consider an operation
of type 3. The potential function needs to decrease enough to cancel out the
high cost of the operation. On the other hand, it cannot increase too much
on an operation of type 2, or this operation’s amortized cost will be too
high. We are trying to show that the Increment operation has a constant
amortized cost. Therefore, an operation of type 2 must increase the potential
function by at most a constant value. Furthermore, an operation of type
3 requires k iterations, so our potential function must have a decrease of
roughly k for such an operation. In addition, any potential function must
be 0 initially and always nonnegative.
Based upon the above discussion, it would appear that the number of
leading 1s in value would be a good measure of the structure’s degradation.
Let us therefore consider using as our potential function the number of
leading 1s. Unfortunately, we immediately encounter a problem with this
CHAPTER 4. BASIC TECHNIQUES 137

function — it is not initially 0, because an IterBinCounter initially has

one leading 1. This function is therefore not a valid potential function. We
could make a small adjustment by subtracting 1 from the number of leading
1s; however, the resulting function will then go negative whenever there are
no leading 1s.
Before we look for an alternative potential function, we should make one
more observation regarding the number of leading 1s. Suppose value begins
with a 0, which is followed by a large number of 1s. When an Increment
is performed on this state, the leading 0 is replaced by a 1, thus causing
the number of leading 1s to increase by a large amount. Hence, even if the
number of leading 1s qualified as a valid potential function, it wouldn’t be
an appropriate one — the amortized cost of this operation would be high
due to the large increase in potential caused by a single operation.
This observation suggests that it is not just the leading 1s that degrade
the structure, but that all of the 1s in value contribute to the degradation.
We might therefore consider using the total number of 1s in value as our
potential function. Again, this number is initially 1, not 0; however, from
the structural invariant, value will always contain at least one 1. Therefore,
if we subtract 1 from the number of 1s in value, we obtain a valid potential
function.
Let us now analyze the amortized cost of Increment relative to this
potential function. Suppose the actual cost is k (i.e., the loops both iterate
k times). The change in potential is simply the number of 1s in value after
the operation, minus the number of 1s in value before the operation (the
two −1s cancel each other when we subtract). The while loop removes k
1s from c. The if statement adds a 1. The for loop does not change the
number of 1s. The total change is therefore 1 − k. The amortized cost is
then the actual cost plus the change in potential, or

k + (1 − k) = 1.

We can therefore conclude that the amortized running time of the IterBin-
Counter operations is in O(1).
Let us now use this technique to analyze the amortized performance of
ExpandableArrayStack. We first observe that operations which result
in an error run in Θ(1) time and do not change the state of the structure;
hence, we can ignore these operations. As we did in Section 4.3, we will
again amortize the number of loop iterations; i.e., the actual cost of an
operation will be the number of loop iterations performed by that operation.
An operation that does not require expanding the array performs no loop
CHAPTER 4. BASIC TECHNIQUES 138

iterations, and an operation that requires expanding the array performs n

loop iterations, where n is the size of the stack prior to the operation.
We now need an appropriate potential function. We first note that the
Pop operation not only is cheap, but it also improves the state of the stack
by making more array locations available. We therefore don’t need to focus
on this operation when looking for a potential function. Instead, we need
to focus on the Push operation. A Push that does not expand the array is
inexpensive, but degrades the future performance by reducing the number
of available array locations. We therefore want the potential function to
increase by at most a constant in this case. A Push that requires an array
expansion is expensive — requiring n iterations — but improves the perfor-
mance of the structure by creating additional array locations. We want the
potential function to decrease by roughly n in this case.
We mentioned earlier that we wanted the potential function to be a
measure of how full the array is. Perhaps the most natural measure is n/k,
where n is the number of elements in the stack and k is the size of the array.
This function is 0 when n is 0 and is always nonnegative. Furthermore,
because n ≤ k, n/k never exceeds 1; hence, no operation can increase this
function by more than 1. However, this also means that no operation causes
it to decrease by more than 1. Therefore, it does not fit the characteristics
we need for a tight amortized analysis.
In order to overcome this problem, let us try multiplying n/k by some
value in order to give it more of a range. Because we need for the function
to decrease by about n when we expand the array, it will need to have grown
by about n after we have done n Pushes; hence, it needs to exhibit at least
linear growth. n/k is bounded by a constant; hence, to cause it to be linear
in n, we would want to multiply it by a function that is linear in n. This
suggests that we might want to try some function of the form an2 /k, where
a is some positive real number to be determined later.
Using this potential function, consider the amortized cost of a Push
operation that expands the array. Prior to the operation, n = k. Therefore,
the change in potential is
a(n + 1)2 an2 an2 + 2an + a − 2an2
− =
2n n 2n
an a
=− +a+
2 2n
an a
≤− +a+
2 2
an 3a
=− + .
2 2
CHAPTER 4. BASIC TECHNIQUES 139

When we add the actual cost of n, we need the result to be bounded by

a fixed constant. We can accomplish this by setting a = 2. The potential
function 2n2 /k therefore results in an amortized cost of no more than 3 in
this case.
Now let us consider the amortized cost of a Push operation that does
not expand the array. Because n increases by 1 and k does not change, the
change in the potential function 2n2 /k is

2(n + 1)2 2n2 2n2 + 4n + 2 − 2n2

− =
k k k
4n + 2
=
k !
n + 12
=4
k
<4

because k must be strictly larger than n, and both are integers. Because no
loop iterations are performed in this case, the actual cost is 0; hence, the
amortized cost is less than 4.
In order to complete the analysis, we must consider the Pop operation.
Because n is initially positive and decreases by 1, and because k remains
the same, the change in potential is

2(n − 1)2 2n2 2n2 − 4n + 2 − 2n2

− =
k k k
2 − 4n
=
k
< 0.

The actual cost is 0. The amortized cost is therefore less than 0.

In each case, the amortized cost is in O(1). Because the time for each
loop iteration and the time required by each operation apart from the loop
iterations are both in O(1), we conclude that the amortized running time of
the stack operations is in O(1).

4.6 Summary
We have shown how the top-down design paradigm can be applied to the
design of data structures. In many cases, we can reduce the implementation
of an ADT to the implementation of simpler or lower-level ADTs. In other
CHAPTER 4. BASIC TECHNIQUES 140

cases, we can reduce the implementation to a common design pattern. The

algorithms we have used for implementing the operations of ADTs have been
quite simple. As we examine more advanced data structures in the following
chapters, we will see that the algorithms in the implementations also use the
top-down approach as presented in Chapter 1.
Applying the top-down approach yields clean techniques for proving that
implementations of ADTs meet their specifications. The techniques are simi-
lar to those presented in Chapter 2, but additionally require proving security
of the implementations. Borrowing some ideas from modular and object-
oriented languages, we have supplied a computational model that facilitates
security in a straightforward way. This model also facilitates the imple-
mentation of immutable structures, which in some cases yield performance
benefits by eliminating the need to copy data. However, use of immutable
structures tends to increase the amount of dynamic memory allocation and
requires the presence of an automatic garbage collector.
The analysis techniques of Chapter 3 can be applied to data structures
as well. In addition, amortized analysis is sometimes useful for analyzing
structures for which operations are occasionally expensive. By amortizing
the cost, we can see that sequences of operations may be less expensive than
a simple worst-case analysis would suggest. Potential functions provide a
general approach to amortized analysis.

4.7 Exercises
Exercise 4.1 Complete the proof of Theorem 4.2 by giving proofs of main-
tenance and correctness for the two missing cases.

Exercise 4.2 Prove that ConsListStack, shown in Figure 4.8 on page

126, meets its specification, given in Figure 4.1 on page 107.

* Exercise 4.3 Give an algorithm for Append, specified in Figure 4.15.

Your algorithm should run in O(n) time, where n is the number of elements
in x.

Exercise 4.4 Prove that IterBinCounter, shown in Figure 4.14 (page

135), meets the specification shown in Figure 4.13 (page 134).

Exercise 4.5 Let f (n) denote the number of 1s in value after n calls to
Increment on a new IterBinCounter. Prove by induction on n that the
CHAPTER 4. BASIC TECHNIQUES 141

Figure 4.15 Specification for Append

Precondition: x and y are ConsLists representing the sequences

hx1 , x2 , . . . , xn i and hy1 , y2 , . . . , ym i, respectively.
Postcondition: Returns a ConsList representing the sequence
hx1 , x2 , . . . , xn , y1 , y2 , . . . , ym i.
Append(x, y)

total number of iterations of the while loop in these n calls is

n − f (n) + 1.

Exercise 4.6 Figure 4.16 gives an alternative implementation of the In-

crement operation for the BinaryCounter ADT.

a. Prove that the implementation that uses this algorithm meets its spec-
ification.

b. It is easily seen that the running time of Increment is proportional

to the number of calls made (including recursive calls) to the internal
function Inc. Using a potential function, show that the amortized
number of calls to Inc is in O(1).

Exercise 4.7 Analyze the amortized cost of the ExpandableArrayStack

operations using the number of iterations performed as the actual cost and

4n3
3k 2
as the potential function, where n is the number of elements in the stack
and k is the size of the array.

Exercise 4.8 Let c > 1 be a fixed real number. Suppose we modify Figure
4.6 so that the new array is of size ⌈c · size⌉. Using the potential function
approach, show that the amortized running time of the stack operations is
in O(1).

Exercise 4.9 With ExpandableArrayStack, it is possible that the stack

reaches a state in which it is using much more space than it requires. This
CHAPTER 4. BASIC TECHNIQUES 142

Figure 4.16 Implementation of the Increment operation for Bina-

ryCounter, specified in Figure 4.13

RecBinCounter.Increment()
value ← Inc(value)

— Internal Functions Follow —

Precondition: c is a ConsList of 0s and 1s not ending in 0.
Postcondition: Returns a ConsList representing c + 1, where the Cons-
Lists are interpreted as the binary representation of natural numbers, least
significant bit first.
RecBinCounter.Inc(c)
if c.IsEmpty()
return new ConsList(1, c)
else if c.Head() = 0
return new ConsList(1, c.Tail())
else
return new ConsList(0, Inc(c.Tail()))

can happen if a large number of elements are pushed onto the stack, then
most are removed. One solution is to modify the Pop operation so that if
the number of elements drops below half the size of the array, then we copy
the elements to a new array of half the size. Give a convincing argument
that this solution would not result in O(1) amortized running time.

Exercise 4.10 An alternative to the solution sketched in the above exercise

is to reduce the size of the array by half whenever it becomes less than 1/4
full, but is still nonempty.

a. Give a modified Pop operation to implement this idea.

* b. Using the technique of Section 4.3, show that the stack operations
have an amortized running time in O(1) when this scheme is used.
You may assume that the array is initially of size 4.

** c. Repeat the above analysis using a potential function. [Hint: Your

potential function will need to increase as the size of the array diverges
CHAPTER 4. BASIC TECHNIQUES 143

from 2n, where n is the number of elements in the stack.]

Exercise 4.11 A queue is similar to a stack, but it provides first in first out
(FIFO) access to the data items. Instead of the operations Push and Pop,
it has operations Enqueue and Dequeue — Enqueue adds an item to the
end of the sequence, and Dequeue removes the item from the beginning of
the sequence.

a. Give an ADT for a queue.

b. Using the linked list design pattern, give an implementation of your

ADT for which all operations run in Θ(1) time.

c. Prove that your implementation meets its specification.

Exercise 4.12 A certain data structure contains operations that each con-
sists of a sequence of zero or more Pops from a stack, followed by a single
Push. The stack is initially empty, and no Pop is attempted when the stack
is empty.

a. Prove that in any sequence of n operations on an initialized structure,

there are at most 2n stack operations (i.e., Pushes and Pops).

b. Use a simple potential function to show that the amortized number of

stack operations is bounded by a constant.

Exercise 4.13 A String, as specified in Figure 4.17, represents a finite

sequence of Chars, or characters.

a. Give an implementation of String for which

• the constructor runs in O(n) amortized time;

• Append runs in O(m) amortized time, where m is the length of
x;
• Substring runs in O(len) amortized time; and
• GetCharacter and Length run in Θ(1) time in the worst
case.

For the purpose of defining amortized running times, think of the

constructor and the Substring operation as appending Chars to an
empty string. Prove the above running times for your implementation.
CHAPTER 4. BASIC TECHNIQUES 144

Figure 4.17 The String ADT

Precondition: A[0..n − 1] is an array of Chars, and n is a Nat.

Postcondition: Constructs a String representing the sequence of Chars
in A.
String(A[0..n − 1])
Precondition: x is a String.
Postcondition: Adds the sequence of Chars represented by x to the end
of this String.
Append(x)
Precondition: i and len are Nats such that i + len does not exceed the
length of this String.
Postcondition: Returns the String representing the sequence
hai , ai+1 , . . . , ai+len−1 i, where ha0 , . . . , an−1 i is the sequence represented by
this String.
Substring(i, len)
Precondition: i is a Nat strictly less than the length of this String.
Postcondition: Returns the Char ai , where ha0 , . . . , an−1 i is the sequence
represented by this String.
GetCharacter(i)
Precondition: true.
Postcondition: Returns the length of the represented sequence.
Length()
CHAPTER 4. BASIC TECHNIQUES 145

b. Prove that your implementation meets its specification.

Exercise 4.14 Figure 4.18 gives an ADT for an immutable arbitrary-preci-

sion natural number. Such an ADT is useful for defining algorithms for op-
erating on natural numbers which may not fit in a single machine word. We
can implement this ADT using a single representation variable, bits[0..n−1],
which is an array of 0s and 1s. The structural invariant is that all elements
of bits are either 0 or 1, and that if SizeOf(bits) 6= 0,

bits[SizeOf(bits) − 1] = 1.

If n = SizeOf(bits), the represented number is then

n−1
X
bits[i]2i .
i=0

Note that the least significant bit has the lowest index; hence, it might
be helpful to think of the array with index 0 at the far right, and indices
increasing from right to left.

a. Complete this implementation of BigNum such that

• NumBits runs in Θ(1) time;

• Shift and GetBits run in Θ(n) time, where n is the number of
bits in the result; and
• the constructor and the remaining operations run in Θ(n) time,
where n is the number of bits in the largest number involved in
the operation.

b. Prove that your implementation meets its specification.

4.8 Chapter Notes

The phenomenon that occurs when multiple copies are made of the same
reference is known in the literature as aliasing. The problem is thoroughly
discussed by, e.g., Aho, Sethi, and Ullman [3] and Muchnick [89].
Use of immutable structures has its roots in functional programming,
though it has carried over to some degree to languages from other paradigms.
CHAPTER 4. BASIC TECHNIQUES 146

Figure 4.18 BigNum ADT.

Precondition: A[0..n − 1] is an array whose values are all either 0 or 1.

Postcondition: Constructs a BigNum representing
n−1
X
A[i]2i .
i=0

BigNum(A[0..n − 1])
Precondition: v refers to a BigNum.
Postcondition: Returns 1 if the value of this BigNum is greater than v,
0 if it is equal to v, or −1 if it is less than v.
BigNum.CompareTo(v)
Precondition: v refers to a BigNum.
Postcondition: Returns a BigNum representing the sum of the value of
this BigNum and v.
BigNum.Add(v)
Precondition: v refers to a BigNum no greater than the value of this
BigNum.
Postcondition: Returns a BigNum representing the value of this BigNum
minus v.
BigNum.Subtract(v)
Precondition: i is an integer.
Postcondition: Returns the floor of the BigNum obtained by multiplying
this BigNum by 2i .
BigNum.Shift(i)
Precondition: true.
Postcondition: Returns the number of bits in the binary representation
of this BigNum with no leading zeros.
BigNum.NumBits()
Precondition: start and len are natural numbers.
Postcondition: Returns an array A[0..len − 1] containing the values of bit
positions start through start + len − 1; zeros are assigned to the high-order
positions if necessary.
GetBits(start, len)
CHAPTER 4. BASIC TECHNIQUES 147

Paulson [91] gives a nice introduction to functional programming using ML,

where immutable data types are the norm.
The search tree viewer posted on this textbook’s web site contains com-
plete Java implementations of ConsList and ConsListStack. Deep clon-
ing is simulated in this code because only immutable items are placed on
the stacks.
Exercise 4.12 is due to Tarjan [105], who gives an excellent survey of
amortized analysis. He credits D. Sleator for the potential function method
of amortized analysis.
Chapter 5

Priority Queues

In many applications, we need data structures which support the efficient

storage of data items and their retrieval in order of a pre-determined priority.
Consider priority-based scheduling, for example. Jobs become available for
execution at various times, and as jobs complete, we wish to schedule the
available job having highest priority. These priorities may be assigned, for
example, according to the relative importance of each job’s being executed
in a timely manner. In order to support this form of storage and retrieval,
we define a PriorityQueue as a set of items, each having an associated Strictly speaking,
we should use a
number giving its priority, together with the operations specified in Figure multiset, because
5.1. we do not
We sometimes wish to have operations MinPriority() and Remove- prohibit multiple
occurrences of
Min() instead of MaxPriority() and RemoveMax(). The specifications the same item.
of these operations are the same as those of MaxPriority and Remove- However,
because we
Max, respectively, except that minimum priorities are used instead of max- ordinarily would
imum priorities. We call the resulting ADT an InvertedPriorityQueue. not insert
multiple
It is a straightforward matter to convert any implementation of Priori- occurrences, we
tyQueue into an implementation of InvertedPriorityQueue. will call it a set.
In order to facilitate implementations of PriorityQueue, we will use a
data structure Keyed for pairing data items with their respective priorities.
This structure will consist of two readable representation variables, key and
data. We will use a rather general interpretation, namely, that key and data
are associated with each other. This generality will allow us to reuse the
structure in later chapters with somewhat different contexts. Its structural
invariant will simply be true. It will contain a constructor that takes two
inputs, x and k, and produces an association with k as the key and x as the
data. It will contain no additional operations.

148
CHAPTER 5. PRIORITY QUEUES 149

Figure 5.1 The PriorityQueue ADT

Precondition: true.
Postcondition: Constructs an empty PriorityQueue.
PriorityQueue()
Precondition: p is a Number.
Postcondition: Adds x to the set with priority p.
PriorityQueue.Put(x, p)
Precondition: The represented set is not empty.
Postcondition: Returns the maximum priority of any item in the set.
PriorityQueue.MaxPriority()
Precondition: The represented set is not empty.
Postcondition: An item with maximum priority is removed from the set
and returned.
PriorityQueue.RemoveMax()
Precondition: true.
Postcondition: Returns the number of items in the set.
PriorityQueue.Size()

5.1 Sorted Arrays

Our first implementation of PriorityQueue maintains the data in an ex-
pandable array, sorted in nondecreasing order of priorities. The representa-
tion consists of two variables:

• elements[0..M − 1]: an array of Keyed items, each containing a data

item with its associated priority as its key, in order of priorities; and

• size: an integer giving the number of data items.

Implementation of the RemoveMax operation is then trivial — after ver-

ifying that size is nonzero, we simply decrement size by 1 and then return
elements[size].Data(). Clearly, this can be done in Θ(1) time. Similarly,
the MaxPriority operation can be trivially implemented to run in Θ(1)
time.
CHAPTER 5. PRIORITY QUEUES 150

In order to implement Put(x, p), we must find the correct place to insert
x so that the order of the priorities is maintained. Let us therefore reduce
the Put operation to the problem of finding the correct location to insert a
given priority p. This location is the index i, 0 ≤ i ≤ size, such that

• if 0 ≤ j < i, then elements[j].Key() < p; and

• if i ≤ j < size, then p ≤ elements[j].Key().

Because the priorities are sorted, elements[i].Key() = p iff there is an item

in the array whose priority is p. Furthermore, if no such item exists, i gives
the location at which such an item should be inserted.
We can apply the top-down approach to derive a search technique called
binary search. Assume we are looking for the insertion point in an array
A[lo..hi − 1]; i.e., the insertion point i will be in the range lo ≤ i ≤ hi.
Further assume that lo < hi, for otherwise, we must have lo = i = hi. Recall
that the divide-and-conquer technique, introduced in Section 3.7, reduces
large instances to smaller instances that are a fixed fraction of the size of
the original instance. In order to apply this technique, we first look at the
priority of the middle data item — the item with index mid = ⌊(lo + hi)/2⌋.
If the key of this item is greater than or equal to p, then i can be no greater
than mid, which in turn is strictly less than hi. Otherwise, i must be
strictly greater than mid, which in turn is greater than or equal to lo. We
will therefore have reduced our search to a strictly smaller search containing
about half the elements from the original search.
Note that this reduction is actually a transformation — a reduction in
which the solution to the smaller problem is exactly the solution to the
original problem. Recall from Section 2.4 that a transformation can be
implemented as a loop in a fairly straightforward way. Specifically, each
iteration of the loop will reduce a large instance to a smaller instance. When
the loop terminates, the instance will be the base case, where lo = hi.
Prior to the loop, lo and hi must have values 0 and size, respectively.
Our invariant will be that 0 ≤ lo ≤ hi ≤ size, that items with indices less
than lo have a key less than p, and that elements with indices greater than
or equal to hi have a key greater than or equal to p. Thus, the index i to be
returned will always be in the range lo ≤ i ≤ hi. When the loop terminates,
we will have lo = hi; hence, we can return either lo or hi. This algorithm
is given as the Find function in Figure 5.2, where a partial implementation
of SortedArrayPriorityQueue is given. The Expand function copies
the contents of its argument into an array of twice the original size, as in
CHAPTER 5. PRIORITY QUEUES 151

Figure 5.2 SortedArrayPriorityQueue implementation (partial) of

the PriorityQueue ADT

Structural Invariant: 0 ≤ size ≤ SizeOf(elements), where elements is

an array of Keyed items whose keys are numbers in nondecreasing order.

SortedArrayPriorityQueue.Put(x, p : Number)
i ← Find(p)
if size = SizeOf(elements)
elements ← Expand(elements)
for j ← size − 1 to i by −1
elements[j + 1] ← elements[j]
elements[i] ← new Keyed(x, p); size ← size + 1

— Internal Functions Follow —

Precondition: The structural invariant holds, and p is a Number.
Postcondition: Returns the index i, 0 ≤ i ≤ size, such that if 0 ≤ j < i,
then elements[j].Key() < p and if i ≤ j < size, then
p ≤ elements[j].Key().
SortedArrayPriorityQueue.Find(p)
lo ← 0; hi ← size
// Invariant: 0 ≤ lo ≤ hi ≤ size,
// if 0 ≤ j < lo, then elements[j].Key() < p,
// and if hi ≤ j < size, then elements[j].Key() ≥ p.
while lo < hi
mid ← ⌊(lo + hi)/2⌋
if elements[mid].Key() ≥ p
hi ← mid
else
lo ← mid + 1
return lo
CHAPTER 5. PRIORITY QUEUES 152

Section 4.3. The remainder of the implementation and its correctness proof
are left as an exercise.
Let us now analyze the running time of Find. Clearly, each iteration of
the while loop runs in Θ(1) time, as does the code outside the loop. We
therefore only need to count the number of iterations of the loop.
Let f (n) denote the number of iterations, where n = hi − lo gives the
number of elements in the search range. One iteration reduces the number
of elements in the range to either ⌊n/2⌋ or ⌈n/2⌉ − 1. The former value
occurs whenever the key examined is greater than or equal to p. The worst
case therefore occurs whenever we are looking for a key smaller than any
key in the set. In the worst case, the number of iterations is therefore given
by the following recurrence:

f (n) = f (⌊n/2⌋) + 1

for n > 1. From Theorem 3.32, f (n) ∈ Θ(lg n). Therefore, Find runs in
Θ(lg n) time.
Let us now analyze the running time of Put. Let n be the value of
size. The first statement requires Θ(lg n) time, and based on our analysis
in Section 4.3, the Expand function should take O(n) time in the worst
case. Because we can amortize the time for Expand, let us ignore it for
now. Clearly, everything else outside the for loop and a single iteration of
the loop run in Θ(1) time. Furthermore, in the worst case (which occurs
when the new key has a value less than all other keys in the set), the loop
iterates n times. Thus, the entire algorithm runs in Θ(n) time in the worst
case, regardless of whether we count the time for Expand.

5.2 Heaps
The SortedArrayPriorityQueue has very efficient MaxPriority and
RemoveMax operations, but a rather slow Put operation. We could speed
up the Put operation considerably by dropping our requirement that the
array be sorted. In this case, we could simply add an element at the end of
the array, expanding it if necessary. This operation is essentially the same
as the ExpandableArrayStack.Push operation, which has an amortized
running time in Θ(1). However, we would no longer be able to take ad-
vantage of the ordering of the array in finding the maximum priority. As a
result, we would need to search the entire array. The running times for the
MaxPriority and RemoveMax operations would therefore be in Θ(n)
time, where n is the number of elements in the priority queue.
CHAPTER 5. PRIORITY QUEUES 153

In order to facilitate efficient implementations of all three operations,

let us try applying the top-down approach to designing an appropriate data
structure. Suppose we have a nonempty set of elements. Because we need
to be able to find and remove the maximum priority quickly, we should
keep track of it. When we remove it, we need to be able to locate the new
maximum quickly. We can therefore organize the remaining elements into
two (possibly empty) priority queues. (As we will see, using two priority
queues for these remaining elements can yield significant performance ad-
vantages over a single priority queue.) Assuming for the moment that both
of these priority queues are nonempty, the new overall maximum must be
the larger of the maximum priorities from each of these priority queues. We
can therefore find the new maximum by comparing these two priorities. The
cases in which one or both of the two priority queues are empty are likewise
straightforward.
We can implement the above idea by arranging the priorities into a heap,
as shown in Figure 5.3. This structure will be the basis of all of the remaining
PriorityQueue implementations presented in this chapter. In this figure,
integer priorities of several data items are shown inside circles, which we
will call nodes. The structure is referenced by its root node, containing the
priority 89. This value is the maximum of the priorities in the structure.
The remaining priorities are accessed via one of two references, one leading
to the left, and the other leading to the right. Each of these two groups
of priorities forms a priority queue structured in a similar way. Thus, as
we follow any path downward in the heap, the values of the priorities are
nonincreasing.
A heap is a special case of a more general structure known as a tree. Let
N be a finite set of nodes, each containing a data item. We define a rooted
tree comprised of N recursively as:
• a special object which we will call the empty tree if N = ∅; or
• a root node x ∈ N , together with a finite sequence hT1 , . . . , Tk i of
children, where
– each Ti is a rooted tree comprised of some (possibly empty) set
Ni ⊆ N \{x} (i.e., each element of each Ni is an element of N ,
but not the root node);
– ki=1 Ni = N \{x} (i.e., the elements in all of the sets Ni together
S
form the set N , without the root node); and
– for i 6= j, Ni ∩Nj = ∅ (i.e., no two of these sets have any elements
in common).
CHAPTER 5. PRIORITY QUEUES 154

Figure 5.3 A heap — each priority is no smaller than any of its children

89

53 32

48 53 27

24 32 13

17

Thus, the structure shown in Figure 5.3 is a rooted tree comprised of 10

nodes. Note that the data items contained in the nodes are not all distinct;
however, the nodes themselves are distinct. The root node contains 89. Its
first child is a rooted tree comprised of six nodes and having a root node
containing 53.
When a node c is a child of a node p, we say that p is the parent of c.
Also, two children of a given node are called siblings. Thus, in Figure 5.3,
the node containing 48 has one node containing 53 as its parent and another
as its sibling. We refer to a nonempty tree whose children are all empty as
a leaf. Thus, in Figure 5.3, the subtree whose root contains 13 is a leaf.
We define the size of a rooted tree to be the number of nodes which
comprise it. Another important measure of a nonempty rooted tree is its
height, which we define recursively to be one plus the maximum height of
its nonempty children; if it has no nonempty children, we say its height is
0. Thus, the height is the maximum of the distances from the root to the
leaves, where we define the distance as the number of steps we must take to
get from one node to the other. For example, the tree in Figure 5.3 has size
10 and height 4.
When we draw rooted trees, we usually do not draw empty trees. For
CHAPTER 5. PRIORITY QUEUES 155

example, in Figure 5.3, the subtree whose root contains 24 has two children,
but the first is empty. This practice can lead to ambiguity; for example, it is
not clear whether the subtree rooted at 13 contains any children, or if they
might all be empty. For this and other reasons, we often consider restricted
classes of rooted trees. Here, we wish to define a binary tree as a rooted tree
in which each nonempty subtree has exactly two children, either (or both)
of which may be empty. In a binary tree, the first child is called the left
child, and the other is called the right child. If we then state that the rooted
tree in Figure 5.3 is a binary tree, it is clear that the subtree rooted at 13,
because it is nonempty, has two empty children.
It is rather difficult to define an ADT for either trees or binary trees
in such a way that it can be implemented efficiently. The difficulty is in
enforcing as a structural invariant the fact that no two children have nodes
in common. In order for an operation to maintain this invariant when adding
a new node, it would apparently need to examine the entire structure to see
if the new node is already in the tree. As we will see, maintaining this
invariant becomes much easier for specific applications of trees. It therefore
seems best to think of a rooted tree as a mathematical object, and to mimic
its structure in defining a heap implementation of PriorityQueue.
In order to build a heap, we need to be able to implement a single
node. For this purpose, we will define a data type BinaryTreeNode. Its
representation will contain three variables:

• root: the Keyed data item stored in the node;

• leftChild: the BinaryTreeNode representing the root of the left

child; and

• rightChild: the BinaryTreeNode representing the root of the right

child.

We will provide read/write access to all three of these variables, and our
structural invariant is simply true. The only constructor is shown in Figure
5.4, and no additional operations are included. Clearly, BinaryTreeNode
meets its specification (there is very little specified), and each operation and
constructor runs in Θ(1) time.
We can now formally define a heap as a binary tree containing Keyed
elements such that if the tree is nonempty, then

• the item stored at the root has the maximum key in the tree; and

• both children are heaps.

CHAPTER 5. PRIORITY QUEUES 156

Figure 5.4 Constructor for BinaryTreeNode

Precondition: true.
Postcondition: Constructs a BinaryTreeNode with all three variables
nil.
BinaryTreeNode()
root ← nil; leftChild ← nil; rightChild ← nil

Based on the above definition, we can define a representation for Priori-

tyQueue using two variables:
• elements: a BinaryTreeNode; and
• size: a natural number.
Our structural invariant will be that elements is a heap whose size is given
by size. We interpret the contents of the nodes comprising this heap as
the set of items stored in the priority queue, together with their associated
priorities.
Implementation of MaxPriority is now trivial — we just return the
key of the root. To implement RemoveMax, we must remove the root
(provided the heap is nonempty) and return the data from its contents.
When we remove the root, we are left with the two children, which must
then be combined into one heap. We therefore will define an internal function
Merge, which takes as input two heaps h1 and h2 with no nodes in common
(i.e., the two heaps share no common structure, though they may have keys
in common), and returns a single heap containing all of the nodes from h1
and h2 . Note that we can also use the Merge function to implement Put
if we first construct a single-node heap from the element we wish to insert.
Let us consider how to implement Merge. If either of the two heaps h1
and h2 is nil (i.e., empty), we can simply return the other heap. Otherwise,
the root of the result must be the root of either h1 or h2 , whichever root
contains a Keyed item with larger key (a tie can be broken arbitrarily).
Let L denote the heap whose root contains the maximum key, and let S
denote the other heap. Then we must form a heap whose root is the root of
L and whose two children are heaps containing the nodes in the following
three heaps:
• the left child of L;
CHAPTER 5. PRIORITY QUEUES 157

• the right child of L; and

• S.

We can form these two children by recursively merging two of these three
heaps.
A simple implementation, which we call SimpleHeap, is shown in Figure
5.5. Note that we can maintain the structural invariant because we can
ensure that the precondition to Merge is always met (the details are left
as an exercise). Note also that the above discussion leaves some flexibility
in the implementation of Merge. In fact, we will see shortly that this
particular implementation performs rather poorly. As a result, we will need
to find a better way of choosing the two heaps to merge in the recursive call,
and/or a better way to decide which child the resulting heap will be.
Let us now analyze the running time of Merge. Suppose h1 and h2
together have n nodes. Clearly, the running time excluding the recursive
call is in Θ(1). In the recursive call, L.RightChild() has at least one fewer
node than does L; hence the total number of nodes in the two heaps in the
recursive call is no more than n − 1. The total running time is therefore
bounded above by

f (n) ∈ f (n − 1) + O(1)
⊆ O(n)

by Theorem 3.31.
At first it might seem that the bound of n − 1 on the number of nodes in
the two heaps in the recursive call is overly pessimistic. However, upon close
examination of the algorithm, we see that not only does this describe the
worst case, it actually describes every case. To see this, notice that nowhere
in the algorithm is the left child of a node changed after that node is created.
Because each left child is initially empty, no node ever has a nonempty left
child. Thus, each heap is single path of nodes going to the right.
The SimpleHeap implementation therefore amounts to a linked list in
which the keys are kept in nonincreasing order. The Put operation will
therefore require Θ(n) time in the worst case, which occurs when we add a
node whose key is smaller than any in the heap. In the remainder of this
chapter, we will examine various ways of taking advantage of the branching
potential of a heap in order to improve the performance.
CHAPTER 5. PRIORITY QUEUES 158

Figure 5.5 SimpleHeap implementation of PriorityQueue

Structural Invariant: elements is a heap whose size is given by size.

SimpleHeap()
elements ← nil; size ← 0

SimpleHeap.Put(x, p : Number)
h ← new BinaryTreeNode(); h.SetRoot(new Keyed(x, p))
elements ← Merge(elements, h); size ← size + 1

SimpleHeap.MaxPriority()
return elements.Root().Key()

SimpleHeap.RemoveMax()
x ← elements.Root().Data(); size ← size − 1
elements ← Merge(elements.LeftChild(), elements.RightChild())
return x

— Internal Functions Follow —

Precondition: h1 and h2 are (possibly nil) BinaryTreeNodes
representing heaps with no nodes in common.
Postcondition: Returns a heap containing all of the nodes in h1 and h2 .
SimpleHeap.Merge(h1 , h2 )
if h1 = nil
return h2
else if h2 = nil
return h1
else
if h1 .Root().Key() > h2 .Root().Key()
L ← h1 ; S ← h2
else
L ← h2 ; S ← h1
L.SetRightChild(Merge(L.RightChild(), S))
return L
CHAPTER 5. PRIORITY QUEUES 159

5.3 Leftist Heaps

In order to improve the performance of merging two heaps, it would make
sense to try to reach one of the base cases as quickly as possible. In Sim-
pleHeap.Merge, the base cases occur when one of the two heaps is empty.
In order to simplify the discussion, let us somewhat arbitrarily decide that
one of the two heaps to be merged in the recursive call will always be S.
We therefore need to decide which child of L to merge with S. In order to
reach a base case as quickly as possible, it would make sense to use the child
having an empty subtree nearest to its root.
Let us define, for a given binary tree T , the null path length to be the
length of the shortest path from the root to an empty subtree. Specifically, if
T is empty, then its null path length is 0; otherwise, it is 1 plus the minimum
of the null path lengths of its children. Now if, in the recursive call, we were
to merge S with the child of L having smaller null path length, then the
sum of the null path lengths of the two heaps would always be smaller for
the recursive call than for the original call. The running time is therefore
proportional to the sum of the null path lengths. This is advantageous due
to the following theorem.

Theorem 5.1 For any binary tree T with n nodes, the null path length of
T is at most lg(n + 1).
The proof of this theorem is typical of many proofs of properties of trees.
It proceeds by induction on n using the following general strategy:

• For the base case, prove that the property holds when n = 0 — i.e.,
for an empty tree.

• For the induction step, apply the induction hypothesis to one or more
of the children of a nonempty tree.

Proof of Theorem 5.1: By induction on n.

Base: n = 0. Then by definition, the null path length is 0 = lg 1.

Induction Hypothesis: Assume for some n > 0 that for 0 ≤ i < n, the
null path length of any tree with i nodes is at most lg(i + 1).

Induction Step: Let T be a binary tree with n nodes. Then because the
two children together contain n − 1 nodes, they cannot both contain more
CHAPTER 5. PRIORITY QUEUES 160

than (n − 1)/2 nodes; hence, one of the two children has no more than
⌊(n − 1)/2⌋ nodes. By the induction hypothesis, this child has a null path of
at most lg(⌊(n − 1)/2⌋ + 1). The null path length of T is therefore at most
1 + lg(⌊(n − 1)/2⌋ + 1) ≤ 1 + lg((n + 1)/2)
= lg(n + 1).

By the above theorem, if we can always choose the child with smaller
null path length for the recursive call, then the merge will operate in O(lg n)
time, where n is the number of nodes in the larger of the two heaps. We
can develop slightly simpler algorithms if we build our heaps so that the
right-hand child always has the smaller null path length, as in Figure 5.6(a).
We therefore define a leftist tree to be a binary tree which, if nonempty, has The term
“leftist” refers to
two leftist trees as children, with the right-hand child having a null path the tendency of
length no larger than that of the left-hand child. A leftist heap is then a these structures
leftist tree that is also a heap. to be heavier on
the left.
In order to implement a leftist heap, we will use an implementation of
a leftist tree. The leftist tree implementation will take care of maintaining
the proper shape of the tree. Because we will want to combine leftist trees
to form larger leftist trees, we must be able to handle the case in which
two given leftist trees have nodes in common. The simplest way to handle
this situation is to define the implementation to be an immutable structure.
Because no changes can be made to the structure, we can treat all nodes
as distinct, even if they are represented by the same storage (in which case
they are the roots of identical trees).
In order to facilitate fast computation of null path lengths, we will record
the null path length of a leftist tree in one of its representation variables.
Thus, when forming a new leftist tree from a root and two existing leftist
trees, we can simply compare the null path lengths to decide which tree
should be used as the right child. Furthermore, we can compute the null
path length of the new leftist tree by adding 1 to the null path length of its
right child.
For our representation of LeftistTree, we will therefore use four vari-
ables:
• root: a Keyed item;
• leftChild: a LeftistTree;
• rightChild: a LeftistTree; and
CHAPTER 5. PRIORITY QUEUES 161

Figure 5.6 Example of performing a LeftistHeap.RemoveMax opera-

tion

20

15 13 15

10 7 11 10 7 13

5 3 5 3 11

(a) The original heap (b) Remove the root (20) and merge
the smaller of its children (13)
with the right child of the larger
of its children (7)

15 15

10 13 13 10

5 11 7 11 7 5

3 3

(c) Make 13 the root of the (d) Because 13 has a larger null path
subtree and merge the tree length than 10, swap them
rooted at 7 with the empty
right child of 13
CHAPTER 5. PRIORITY QUEUES 162

• nullPathLength: a Nat.
We will allow read access to all variables. Our structural invariant will be
that this structure is a leftist tree such that
• nullPathLength gives its null path length; and

• root = nil iff nullPathLength = 0.

Specifically, we will allow the same node to occur more than once in the
structure — each occurrence will be viewed as a copy. Because the structure
is immutable, such sharing is safe. The implementation of LeftistTree is
shown in Figure 5.7. Clearly, each of these constructors runs in Θ(1) time.
We now represent our LeftistHeap implementation of PriorityQueue
using two variables:
• elements: a LeftistTree; and

• size: a natural number.

Our structural invariant is that elements is a leftist heap whose size is given
by size, and whose nodes are Keyed items. We interpret these Keyed
items as the represented set of elements with their associated priorities.
The implementation of LeftistHeap is shown in Figure 5.8.
Based on the discussion above, Merge runs in O(lg n) time, where n is
the number of nodes in the larger of the two leftist heaps. It follows that
Put and RemoveMax operate in O(lg n) time, where n is the number of
items in the priority queue. Though it requires some work, it can be shown
that the lower bound for each of these running times is in Ω(lg n).

Example 5.2 Consider the leftist heap shown in Figure 5.6(a). Suppose
we were to perform a RemoveMax on this heap. To obtain the resulting
heap, we must merge the two children of the root. The larger of the two
keys is 15; hence, it becomes the new root. We must then merge its right
child with the original right child of 20 (see Figure 5.6(b)). The larger of the
two roots is 13, so it becomes the root of this subtree. The subtree rooted at
7 is then merged with the empty right child of 13. Figure 5.6(c) shows the
result without considering the null path lengths. We must therefore make
sure that in each subtree that we’ve formed, the null path length of the right
child is no greater than the null path length of the left child. This is the
case for the subtree rooted at 13, but not for the subtree rooted at 15. We
therefore must swap the children of 15, yielding the final result shown in
Figure 5.6(d).
CHAPTER 5. PRIORITY QUEUES 163

Figure 5.7 The LeftistTree data structure.

Structural Invariant: This structure forms a leftist tree with

nullPathLength giving its null path length, and root = nil iff
nullPathLength = 0.
Precondition: true.
Postcondition: Constructs an empty LeftistTree.
LeftistTree()
root ← nil; leftChild ← nil; rightChild ← nil; nullPathLength ← 0

Precondition: x is a non-nil Keyed item.

Postcondition: Constructs a LeftistTree containing x at its root and
having two empty children.
LeftistTree(x : Keyed)
if x = nil
error
else
root ← x; nullPathLength ← 1
leftChild ← new LeftistTree()
rightChild ← new LeftistTree()

Precondition: x, t1 , and t2 are non-nil, x is Keyed, and t1 and t2 are

LeftistTrees.
Postcondition: Constructs a LeftistTree containing x at its root and
having children t1 and t2 .
LeftistTree(x : Keyed, t1 : LeftistTree, t2 : LeftistTree)
if x = nil or t1 = nil or t2 = nil
error
else if t1 .nullPathLength ≥ t2 .nullPathLength
leftChild ← t1 ; rightChild ← t2
else
leftChild ← t2 ; rightChild ← t1
root ← x; nullPathLength ← 1 + rightChild.nullPathLength
CHAPTER 5. PRIORITY QUEUES 164

Figure 5.8 LeftistHeap implementation of PriorityQueue

Structural Invariant: elements is a leftist heap whose size is given by

size and whose nodes are Keyed items.

LeftistHeap()
elements ← new LeftistTree(); size ← 0

LeftistHeap.Put(x, p : Number)
elements ← Merge(elements, new LeftistTree(new Keyed(x, p)))
size ← size + 1

LeftistHeap.MaxPriority()
return elements.Root().Key()

LeftistHeap.RemoveMax()
x ← elements.Root().Data()
elements ← Merge(elements.LeftChild(), elements.RightChild())
size ← size − 1
return x

— Internal Functions Follow —

Precondition: h1 and h2 are LeftistTrees storing heaps.
Postcondition: Returns a LeftistTree containing the elements of h1
and h2 in a heap.
LeftistHeap.Merge(h1 , h2 )
if h1 .Root() = nil
return h2
else if h2 .Root() = nil
return h1
else if h1 .Root().Key() > h2 .Root().Key()
L ← h1 ; S ← h2
else
L ← h2 ; S ← h1
t ← Merge(L.RightChild(), S)
return new LeftistTree(L.Root(), L.LeftChild(), t)
CHAPTER 5. PRIORITY QUEUES 165

A Put operation is performed by creating a single-node heap from the

element to be inserted, then merging the two heaps as in the above example.
The web site that accompanies this textbook contains a program for viewing
and manipulating various kinds of heaps, including leftist heaps and the
heaps discussed in the remainder of this chapter. This heap viewer can be
useful for generating other examples in order to understand the behavior of
heaps.
It turns out that in order to obtain O(lg n) worst-case performance, it
is not always necessary to follow the shortest path to a nonempty subtree.
For example, if we maintain a tree such that for each of its n nodes, the
left child has at least as many nodes as the right child, then the distance
from the root to the rightmost subtree is still no more than lg(n + 1). As a
result, we can use this strategy for obtaining O(lg n) worst-case performance
for the PriorityQueue operations (see Exercise 5.7 for details). However,
we really don’t gain anything from this strategy, as it is now necessary to
maintain the size of each subtree instead of each null path length. In the
next two sections, we will see that it is possible to achieve good performance
without maintaining any such auxiliary information.

5.4 Skew Heaps

In this section, we consider a simple modification to SimpleHeap that yields
good performance without the need to maintain auxiliary information such
as null path lengths. The idea is to avoid the bad performance of Sim-
pleHeap by modifying Merge to swap the children after the recursive
call. We call this modified structure a skew heap. The Merge function for
SkewHeap is shown in Figure 5.9; the remainder of the implementation of
SkewHeap is the same as for SimpleHeap.

Example 5.3 Consider again the heap shown in Figure 5.6(a), and suppose
it is a skew heap. Performing a RemoveMax on this heap proceeds as
shown in Figure 5.6 through part (c). At this point, however, for each node
at which a recursive Merge was performed, the children of this node are
swapped. These nodes are 13 and 15. The resulting heap is shown in Figure
5.10.
In order to understand why such a simple modification might be advan-
tageous, observe that in Merge, when S is merged with L.RightChild(),
we might expect the resulting heap to have a tendency to be larger than
L.LeftChild(). As we noted at the end of the previous section, good
CHAPTER 5. PRIORITY QUEUES 166

Figure 5.9 The SkewHeap.Merge internal function

Precondition: h1 and h2 are (possibly nil) BinaryTreeNodes represent-

ing heaps with no nodes in common.
Postcondition: Returns a heap containing all of the nodes in h1 and h2 .
SkewHeap.Merge(h1 , h2 )
if h1 = nil
return h2
else if h2 = nil
return h1
else
if h1 .Root().Key() > h2 .Root().Key()
L ← h1 ; S ← h2
else
L ← h2 ; S ← h1
t ← Merge(L.RightChild(), S)
L.SetRightChild(L.LeftChild()); L.SetLeftChild(t)
return L

worst-case behavior can be obtained by ensuring that the left child of each
node has at least as many nodes as the right child. Intuitively, we might
be able to approximate this behavior by swapping the children after ev-
ery recursive call. However, this swapping does not always avoid expensive
operations.
Suppose, for example, that we start with an empty skew heap, then
insert the sequence of keys 2, 1, 4, 3, . . . , 2i, 2i − 1, 0, for some i ≥ 1. Figure
5.11 shows this sequence of insertions for i = 3. Note that each time an
even key is inserted, because it is the largest in the heap, it becomes the Corrected
3/30/11.
new root and the original heap becomes its left child. Then when the next
key is inserted, because it is smaller than the root, it is merged with the
empty right child, then swapped with the other child. Thus, after each odd
key is inserted, the heap will contain all the even keys in the rightmost path
(i.e., the path beginning at the root and going to the right until it reaches
an empty subtree), and for i ≥ 1, key 2i will have key 2i − 1 as its left child.
Finally, when key 0 is inserted, because it is the smallest key in the heap,
it will successively be merged with each right child until it is merged with
the empty subtree at the far right. Each of the subtrees on this path to the
CHAPTER 5. PRIORITY QUEUES 167

Figure 5.10 The result of performing a RemoveMax on the skew heap

shown in Figure 5.6(a)

15

13 10

7 11 5

right is then swapped with its sibling. Clearly, this last insertion requires
Θ(i) running time, and i is proportional to the number of nodes in the heap.
The bad behavior described above results because a long rightmost path
is constructed. Note, however, that 2i Put operations were needed to con-
struct this path. Each of these operations required only Θ(1) time. Fur-
thermore, after the Θ(i) operation, no long rightmost paths exist from any
node in the heap (see Figure 5.11). This suggests that a skew heap might
have good amortized running time.
A good measure of the actual cost of the SkewHeap operations is the
number of calls to Merge, including recursive calls. In order to derive a
bound on the amortized cost, let us try to find a good potential function.
Based upon the above discussion, let us say that a node is good if its left
child has at least as many nodes as its right child; otherwise, it is bad. We
now make two key observations, whose proofs are left as exercises:

• In any binary tree with n nodes, the number of good nodes in the
rightmost path is no more than lg(n + 1).

• In the Merge function, if L is a bad node initially, it will be a good

node in the resulting heap.

Due to these observations, we use as our potential function the number

of bad nodes in the heap. Because the number of good nodes in each of
CHAPTER 5. PRIORITY QUEUES 168

Figure 5.11 Construction of a bad skew heap

2 2 4 4

1 2 3 2

1 1

6 6 6

4 5 4 4 5

3 2 3 2 2 3

1 1 0 1

the two rightmost paths is logarithmic, the potential function can increase
by only a logarithmic amount on any Merge. Furthermore, because any
bad node encountered becomes good, the resulting change in potential will
cancel the actual cost associated with this call, leaving only a logarithmic
number of calls whose actual costs are not canceled. As a result, we should
expect the amortized costs of the SkewHeap operations to be in O(lg n),
where n is the number of elements in the priority queue (the details of the
analysis are left as an exercise). Thus, a SkewHeap provides a simple, yet
efficient, implementation of PriorityQueue.
CHAPTER 5. PRIORITY QUEUES 169

Figure 5.12 The RandomizedHeap.Merge internal function

Precondition: h1 and h2 are (possibly nil) BinaryTreeNodes represent-

ing heaps with no nodes in common.
Postcondition: Returns a heap containing all of the nodes in h1 and h2 .
RandomizedHeap.Merge(h1 , h2 )
if h1 = nil
return h2
else if h2 = nil
return h1
else
if h1 .Root().Key() > h2 .Root().Key()
L ← h1 ; S ← h2
else
L ← h2 ; S ← h1
if FlipCoin() = heads
L.SetLeftChild(Merge(L.LeftChild(), S))
else
L.SetRightChild(Merge(L.RightChild(), S))
return L

5.5 Randomized Heaps

For all of the heap implementations we have seen so far, the merge uses
the right child in the recursive call. This choice is not necessary for the
correctness of any of the algorithms, but does impact their performance.
SimpleHeap.Merge performs badly because all recursive calls use right
children, and their results all form right children. Leftist heaps and skew
heaps avoid this bad performance by using the results of the recursive calls
as left children, at least part of the time. Another approach is to use different
children in different calls. Specifically, when we make a recursive call, we
can flip a coin to determine which child to use.
The resulting Merge function is shown in Figure 5.12; the remainder
of the implementation of RandomizedHeap is identical to the implemen-
tations of SimpleHeap and SkewHeap. We assume that the FlipCoin
function returns heads or tails randomly with uniform probability. Thus,
each call to FlipCoin returns heads with probability 1/2, regardless of the
CHAPTER 5. PRIORITY QUEUES 170

results of any prior calls. This function can typically be implemented using
a built-in random number generator. Most platforms provide a function
returning random values uniformly distributed over the range of signed in-
tegers on that platform. In a standard signed integer representation, the
negative values comprise exactly half the range. The FlipCoin function
can therefore generate a random integer and return heads iff that integer is
negative.
It usually makes no sense to analyze the worst-case running time for a
randomized algorithm, because the running time usually depends on random
events. For example, if a given heap consists of a single path with n nodes,
the algorithm could follow exactly that path. However, this could only
happen for one particular sequence of n coin flips. If any of the flips differ
from this sequence, the algorithm reaches a base case and terminates at that
point. Because the probability of flipping this exact sequence is very small
for large n, a worst-case analysis seems inappropriate. Perhaps more to the
point, a worst-case analysis would ignore the effect of randomization, and
so does not seem appropriate for a randomized algorithm.
Instead, we can analyze the expected running time of a randomized al-
gorithm. The goal of expected-case analysis is to bound the average perfor-
mance over all possible executions on a worst-case input. For an ordinary
deterministic algorithm, there is only one possible execution on any given
input, but for randomized algorithms, there can be many possible executions
depending on the random choices made.
Expected-case analysis is based on the expected values of random vari-
ables over discrete probability spaces. A discrete probability space is a count- A set is said to
countable if each
able set of elementary events, each having a probability. For an elementary element can be
event e in a discrete probability space S, we denote the probability of e by labeled with a
P (e). For any discrete probability space S, we require that 0 ≤ P (e) ≤ 1 unique natural
number.
and that X
P (e) = 1.
e∈S
As a simple example, consider the flipping of a fair coin. The probability
space is {heads, tails}, and each of these two elementary events has probabil-
ity 1/2. For a more involved example, let T be a binary tree, and consider
the probability space PathT consisting of paths from the root of T to empty
subtrees. We leave as an exercise to show that if T has n nodes, then it
has n + 1 empty subtrees; hence PathT has n + 1 elements. In order that
it be a probability space, we need to assign a probability to each path. The
probability of a given path of length k should be the same as the probability
of the sequence of k coin flips that yields this path in the Merge algorithm;
CHAPTER 5. PRIORITY QUEUES 171

thus, if the path corresponds to k flips, its probability should be 2−k . We

leave as an exercise to prove that the sum of these probabilities is 1 for any
binary tree.
An important element of expected-case analysis is the notion of a discrete
random variable, which is a function f : S → R, where S is a discrete
probability space. In this text, we will restrict our random variables to
nonnegative values. For an example of a random variable, let lenT (e) give
the length of a path e in the probability space PathT defined above. The More precisely,
lenT (e) is the
expected value of a random variable f over a probability space S is defined number of coin
to be X flips that are
E[f ] = f (e)P (e). needed to
generate e.
e∈S

Thus, by multiplying the value of the random variable for each elementary
event by the probability of that elementary event, we obtain an average
value for that variable. Note that it is possible for an expected value to
be infinite. If the summation converges, however, it converges to a unique
value, because all terms are nonnegative.

Example 5.4 Let T be a binary tree with n nodes, such that all paths from
the root to empty subtrees have the same length. Because the probability of
each path is determined solely by its length, all paths must have the same
probability. Because there are n + 1 paths and the sum of their probabilities
is 1, each path must have probability 1/(n + 1). In this case, E[lenT ] is
simply the arithmetic mean, or simple average, of all of the lengths:
X
E[lenT ] = lenT (e)P (e)
e∈PathT
1 X
= lenT (e).
n+1
e∈PathT

Furthermore, because the lengths of all of the paths are the same, E[lenT ]
must be this length, which we will denote by k.
We have defined the probability of a path of length k to be 2−k . Fur-
thermore, we have seen that all probabilities are 1/(n + 1). We therefore
have
2−k = 1/(n + 1).
Solving for k, we have
k = lg(n + 1).
Thus, E[lenT ] = lg(n + 1).
CHAPTER 5. PRIORITY QUEUES 172

The discrete random variable lenT is always a natural number. When

this is the case, its expected value is often easier to analyze. To show why,
we first need to define an event, which is any subset of the elementary events
in a discrete probability space. The probability of an event A is the sum of
the probabilities of its elementary events; i.e.,
X
P (A) = P (e).
e∈A

Note that because the sum of the probabilities of all elementary events in
a discrete probability space is 1, the probability of an event is never more
than 1.
The following theorem gives a technique for computing expected values
of discrete random variables that range over the natural numbers. It uses
predicates like “f = i” to describe events; e.g., the predicate “f = i” defines
the event in which f has the value i, and P (f = i) is the probability of this
event.

Theorem 5.5 Let f : S → N be a discrete random variable. Then

∞
X
E[f ] = P (f ≥ i).
i=1

The idea behind the proof is that P (f = i) = P (f ≥ i) − P (f ≥ i + 1).

The definition of E[f ] then yields
X
E[f ] = f (e)P (e)
e∈S
∞
X
= iP (f = i)
i=0
X∞
= i(P (f ≥ i) − P (f ≥ i + 1))
i=0
X∞
= (iP (f ≥ i) − iP (f ≥ i + 1)).
i=0

In the above sum, the negative portion iP (f ≥ i + 1) of the ith term cancels
most of the positive portion (i + 1)P (f ≥ i + 1) of the (i + 1)st term.
The result of this cancellation is the desired sum. However, in order for
this reasoning to be valid, it must be the case that the “leftover” term,
−iP (f ≥ i + 1), converges to 0 as i approaches infinity if E[f ] is finite. We
leave the details as an exercise.
CHAPTER 5. PRIORITY QUEUES 173

Example 5.6 Let T be a binary tree in which each of the n nodes has
an empty left child; i.e., the nodes form a single path going to the right.
Again, the size of PathT is n + 1, but now the probabilities are not all the
same. The length of the path to the rightmost empty subtree is n; hence,
its probability is 2−n . For 1 ≤ i ≤ n, there is exactly one path that goes
right i − 1 times and left once. The probabilities for these paths are given
by 2−i . We therefore have
X
E[lenT ] = lenT (e)P (e)
e∈PathT
n
X
= n2−n + i2−i .
i=1

Because we have no formula to evaluate the above summation, let us

instead apply Theorem 5.5. The probability that a given path has length at
least i, for 1 ≤ i ≤ n, is the probability that i − 1 coin flips all yield tails.
This probability is 21−i . The probability that a given path has length at
least i for i > n is 0. By Theorem 5.5, we therefore have
∞
X
E[lenT ] = P (lenT ≥ i)
i=1
n
X
= 21−i
i=1
n−1
X
= (1/2)i
i=0
(1/2)n − 1
= (by (2.2))
(1/2) − 1
= 2 − 21−n .

Thus, E[lenT ] < 2.

In order to be able to analyze the expected running time of Random-
izedHeap.Merge, we need to know E[lenT ] for a worst-case binary tree T
with n nodes. Examples 5.4 and 5.6 give two extreme cases — a completely
balanced tree and a completely unbalanced tree. We might guess that the
worst case would be one of these extremes. Because lg(n + 1) ≥ 2 − 21−n for
all n ∈ N, a good guess would be that lg(n + 1) is an upper bound for the
worst case. We can show that this is indeed the case, but we need to use
CHAPTER 5. PRIORITY QUEUES 174

the following theorem relating the sum of logarithms to the logarithm of a

sum.

Theorem 5.7 If x and y are positive real numbers, then

lg x + lg y ≤ 2 lg(x + y) − 2.

Proof: We first note that lg x + lg y = lg xy. We will therefore show that

the right-hand side of the inequality is at least lg xy. Using the fact that
lg 4 = 2, we have

2 lg(x + y) − 2 = lg((x + y)2 ) − lg 4

2
x + 2xy + y 2

= lg .
4

In order to isolate lg xy, let us now subtract xy from the fraction in the
above equation. This yields
2
x + 2xy + y 2

2 lg(x + y) − 2 = lg
4
x2 − 2xy + y 2

= lg xy +
4
(x − y)2

= lg xy +
4
≥ lg xy,

because (x − y)2 /4 is always nonnegative and the lg function is nondecreas-

ing.

We can now show that lg(n + 1) is an upper bound for E[lenT ] when T
is a binary tree with n nodes.

Theorem 5.8 Let T be any binary tree with size n, where n ∈ N. Then
E[lenT ] ≤ lg(n + 1).

Proof: By induction on n.

Base: n = 0. Then only one path to an empty tree exists, and its length is
0. Hence, E[lenT ] = 0 = lg 1.
CHAPTER 5. PRIORITY QUEUES 175

Induction Hypothesis: Assume that for some n > 0, if S is any binary

tree with size i < n, then E[lenS ] ≤ lg(i + 1).

Induction Step: Suppose T has size n. Because n > 0, T is nonempty.

Let L and R be the left and right children, respectively, of T . We then have
X
E[lenT ] = lenT (e)P (e)
e∈PathT
X P (e) X P (e)
= (lenL (e) + 1) + (lenR (e) + 1) , (5.1)
2 2
e∈PathL e∈PathR

because the probability of any path from the root of a child of T to any
empty subtree is twice the probability of the path from the root of T to the
same empty subtree, and its length is one less.
Because the two sums in (5.1) are similar, we will simplify just the first
one. Thus,
 
X P (e) 1 X X
(lenL (e) + 1) =  lenL (e)P (e) + P (e)
2 2
e∈PathL e∈PathL e∈PathL
 
1 X
= lenL (e)P (e) + 1 ,
2
e∈PathL

because in PathL , the sum of the probabilities is 1. We now observe that

X
lenL (e)P (e) = E[lenL ].
e∈PathL

Applying a similar simplification to the second sum in 5.1, we have

E[lenT ] = 1 + (E[lenL ] + E[lenR ])/2.

Suppose L has size i. Then R has size n − i − 1. Because 0 ≤ i < n, the

Induction Hypothesis applies to both L and R. Thus,

E[lenT ] ≤ 1 + (lg(i + 1) + lg(n − i))/2

≤ 1 + (2 lg(n + 1) − 2)/2 (by Theorem 5.7)
= lg(n + 1).

CHAPTER 5. PRIORITY QUEUES 176

The fact that the expected length of a randomly chosen path in a binary
tree of size n is never more than lg(n + 1) gives us reason to believe that the
expected running time of RandomizedHeap.Merge is in O(lg n). How-
ever, Merge operates on two binary trees. We therefore need a bound on
the expected sum of the lengths of two randomly chosen paths, one from
each of two binary trees. Hence, we will combine two probability spaces
PathS and PathT to form a new discrete probability space Paths S,T . The
elementary events of this space will be pairs consisting of an elementary
event from PathS and an elementary event from PathT .
We need to assign probabilities to the elementary events in Paths S,T . In
so doing, we need to reflect the fact that the lengths of any two paths from
S and T are independent of each other; i.e., knowing the length of one path
tells us nothing about the length of the other path. Let e1 and e2 be events
over a discrete probability space S. We say that e1 and e2 are independent
if P (e1 ∩ e2 ) = P (e1 )P (e2 ).
Suppose we were to define a new discrete probability space Se2 including
only those elementary events in the event e2 . The sum of the probabilities
of these elementary events is P (e2 ). If we were to scale all of these proba-
bilities by dividing by P (e2 ), we would achieve a total probability of 1 while
preserving the ratio of any two probabilities. The probability of event e1
within Se2 would be given by

P (e1 ∩ e2 )
P (e1 | e2 ) = , (5.2)
P (e2 )

where the probabilities on the right-hand side are with respect to S. We call
P (e1 | e2 ) the conditional probability of e1 given e2 . Note that if P (e2 ) 6=
0, independence of e1 and e2 is equivalent to P (e1 ) = P (e1 | e2 ). Thus,
two events are independent if knowledge of one event does not affect the
probability of the other.
The definition of independence tells us how to assign the probabilities
in Paths S,T . Let e1 be the event such that the path in S is s, and let e2
be the event such that the path in T is t. Then e1 ∩ e2 is the elementary
event consisting of paths s and t. We need P (e1 ∩ e2 ) = P (e1 )P (e2 ) in
order to achieve independence. However, P (e1 ) should be the probability
of s in PathS , and P (e2 ) should be the probability of t in PathT . Thus the
probability of an elementary event in Paths S,T must be the product of the
probabilities of the constituent elementary events from PathS and PathT .
It is then not hard to verify that P (e1 ) and P (e2 ) are the probabilities of s
in PathS and of t in PathT , respectively.
CHAPTER 5. PRIORITY QUEUES 177

We now extend the discrete random variables lenS and lenT to the space
Paths S,T so that lenS gives the length of the path in S and lenT gives
the length of the path in T . Because neither the lengths of the paths nor
their probabilities change when we make this extension, it is clear that their
expected values do not change either.
The running time of RandomizedHeap.Merge is clearly proportional
to the lengths of the paths followed in the two heaps S and T . These paths
may or may not go all the way to an empty subtree, but if not, we can extend
them to obtain elementary events s and t in PathS and PathT , respectively.
The running time is then bounded above by c(lenS (s) + lenT (t)), where
c is some fixed positive constant. The expected running time of Merge
is therefore bounded above by E[c(lenS + lenT )]. In order to bound this
expression, we need the following theorem.

Theorem 5.9 (Linearity of Expectation) Let f , g, and hi be discrete

random variables for all i ∈ N, and let a ∈ R≥0 . Then

E[af + g] = aE[f ] + E[g];

and " #
∞
X ∞
X
E hi = E[hi ].
i=0 i=0

The proof of this theorem is straightforward and left as an exercise. It is

important to realize not only what this theorem says, but also what it doesn’t
say. For example, it is not necessarily the case that E[f g] = E[f ]E[g], or
that E[2f ] = 2E[f ] — see Exercise 5.17 for specific counterexamples. We
must therefore be very careful in working with expected values, as they do
not always behave as our intuition might suggest.
Applying Theorems 5.9 and 5.7 to our analysis, we now see that

E[c(lenS + lenT )] = c(E[lenS ] + E[lenT ])

≤ c(lg(|S| + 1) + lg(|T | + 1))
≤ 2c lg(|S| + |T | + 2),

where |S| and |T | denote the sizes of S and T , respectively. Thus, the
expected running time of Merge is in O(lg n), where n is the total number
of nodes in the two heaps. It follows that the expected running times of
Put and RemoveMax are also in O(lg n).
A close examination of Example 5.4 reveals that the bound of lg(n + 1)
on E[lenT ] is reached when n + 1 is a power of 2. Using the fact that lg is
CHAPTER 5. PRIORITY QUEUES 178

smooth, we can then show that the expected running time of Merge is in
Ω(lg n); the details are left as an exercise. Thus, the expected running times
of Put and RemoveMax are in Θ(lg n).

5.6 Sorting and Binary Heaps

In Section 3.6, we saw how to sort an array in Θ(n2 ) time. A priority queue
can be used to improve this performance. Using either a LeftistHeap or
a SkewHeap, we can insert n elements in Θ(n lg n) time, by Theorem 3.28.
We can then sort the items in the heap by removing the maximum in Θ(lg n)
time and sorting the remainder. It is easily seen that this entire algorithm
runs in Θ(n lg n) time.
In order to improve further the performance of sorting, we would like
to avoid the need to use an auxiliary data structure. Specifically, we would
like to keep the data items in a single array, which is partitioned into an
unsorted part followed by a sorted part, as illustrated in Figure 5.13 (a).
The unsorted part will, in essence, be a representation of a priority queue
— we will explain the details of this representation in what follows. This
priority queue will contain keys that are no larger than any of the keys in
the sorted part. When we remove the maximum element from the priority
queue, this frees up an array location, as shown in Figure 5.13 (b). We can
put the element that we removed from the priority queue into this location.
Because this key is at least as large as any key in the priority queue, but
no larger than any key in the sorted part, we can extend the sorted part to
include this location (see Figure 5.13 (c)).
We therefore need to be able to represent a heap using an array. One way
to accomplish this is to number the nodes left-to-right by levels, as shown
in Figure 5.14. The numbers we have assigned to the nodes can be used
as array indices. In order to avoid ambiguity, there should be no “missing”
nodes; i.e., each level except possibly the last should be completely full, and
all of the nodes in the last level should be as far to the left as possible. This
scheme for storing a heap is known as a binary heap.
Notice that a binary heap is very nearly balanced. We saw in Example
5.4 that in a completely balanced binary tree with n nodes, the length of any
path to an empty subtree is lg(n+1). This result holds only for tree sizes that
can be completely balanced. However, it is not hard to show that for any
n, if a binary tree with n nodes is balanced as nearly as possible, the length
of the longest path to an empty subtree is ⌈lg(n + 1)⌉ (or equivalently, the
height is ⌈lg(n + 1)⌉ − 1). We will show that this fact allows us to implement
CHAPTER 5. PRIORITY QUEUES 179

Figure 5.13 Illustration of sorting using a priority queue represented in an

array

(a)
55 48 52 37 41 50 70 75 85 89 94
unsorted sorted
(priority queue)

(b)
52 48 50 37 41 70 75 85 89 94
unsorted sorted
(priority queue)
55

(c)
52 48 50 37 41 55 70 75 85 89 94
unsorted sorted
(priority queue)

Figure 5.14 A binary heap

1
89

2 3
53 32

4 5 6 7
48 53 17 27

8 9 10
24 13 32
CHAPTER 5. PRIORITY QUEUES 180

both Put and RemoveMax for a binary heap in Θ(lg n) time.

Note that each level of a binary heap, except the first and possibly the
last, contains exactly twice as many nodes as the level above it. Thus, if we
were to number the levels starting with 0 for the top level, then each level
i (except possibly the last) contains exactly 2i nodes. It follows from (2.2)
that levels 0 through i − 1, where i is strictly less than the total number
of levels, have a total of 2i − 1 nodes. Let x be the jth node on level i. x
would then have index 2i − 1 + j. Suppose x has a left child, y. In order
to compute its index, we observe that level i has j − 1 nodes to the left of
x. Each of these nodes has two children on level i + 1 to the left of node y.
Therefore, the index of y is

2i+1 − 1 + 2(j − 1) + 1 = 2i+1 + 2j − 2,

or exactly twice the index of its parent. Likewise, if x has a right child, its
index is 1 greater than that of y.
As a result of these relationships, we can use simple calculations to find
either child or the parent of a node at a given location. Specifically, the left
and right children of the element at location i are the elements at locations
2i and 2i + 1, respectively, provided they exist. Furthermore, the parent of
the element at location i > 1 is at location ⌊i/2⌋.
Let us consider how we can implement a binary heap as a data structure.
We will use two representation variables:

• elements[0..m]: an array of Keyed items; and

• size: a Nat.

We allow read access to size. For reasons that will become clear shortly,
elements[0] will act as a sentinel element, and will have as its key the max- A sentinel
element is an
imum allowable value. For convenience, we will use a constant sentinel to extra element
represent such a data item. Note because ⌊1/2⌋ = 0, we can treat elements[0] added to a data
as if it were the parent of elements[1]. structure in order
to indicate when
The structural invariant will be: a traversal of
that structure
• size ≤ SizeOf(elements); has reached the
end.
• elements[0] = sentinel; and

• for 1 ≤ i ≤ size, elements[i].Key() ≤ elements[⌊i/2⌋].Key().

We interpret elements[1..size] as the elements of the set being represented,

together with their associated priorities.
CHAPTER 5. PRIORITY QUEUES 181

Unfortunately, the algorithms for merging heaps don’t work for binary
heaps because they don’t maintain the balanced shape. Therefore, let us
consider how to insert an element x into a binary heap. If size is 0, then we
can simply make x the root. Otherwise, we need to compare x.Key() with
the key of the root. The larger of the two will be the new root, and we can
then insert the other into one of the children. We select which child based
on where we need the new leaf.
In this insertion algorithm, unless the tree is empty, there will always be
a recursive call. This recursive call will always be on the child in the path
that leads to the location at which we want to add the new node. Note
that the keys along this path from the root to the leaf are in nonincreasing
order. As long as the key to be inserted is smaller than the key to which it
is compared, it will be the inserted element in the recursive call. When it is
compared with a smaller key, that smaller key is used in the recursive call.
When this happens, the key passed to the recursive call will always be at
least as large as the root of the subtree in which it is being inserted; thus,
it will become the new root, and the old root will be used in the recursive
call. Thus, the entire process results in inserting the new key at the proper
point in the path from the root to the desired insertion location.
For example, suppose we wish to insert the priority 35 into the binary
heap shown in Figure 5.15(a). We first find the path to the next insertion
point. This path is h89, 32, 17i. The proper position of 35 in this path
is between 89 and 32. We insert 35 at this point, pushing the following
priorities downward. The result is shown in Figure 5.15(b).
Because we can easily find the parent of a node in a BinaryHeap, we
can implement this algorithm bottom-up by starting at the location of the
new leaf and shifting elements downward one level until we reach a location
where the new element will fit. This is where having a sentinel element is
convenient — we know we will eventually find some element whose key is at
least as large as that of x. The resulting algorithm is shown in Figure 5.16.
We assume that Expand(A) returns an array of twice the size of A, with
the elements of A copied to the first half of the returned array.
The RemoveMax operation is a bit more difficult. We need to remove
the root because it contains the element with maximum priority, but in or-
der to preserve the proper shape of the heap, we need to remove a specific
leaf. We therefore first save the value of the root, then remove the proper
leaf. We need to form a new heap by replacing the root with the removed
leaf. In order to accomplish this, we use the MakeHeap algorithm shown
in Figure 5.17. For ease of presentation, we assume t is formed with Bi-
naryTreeNodes, rather than with an array. If the key of x is at least as
CHAPTER 5. PRIORITY QUEUES 182

Figure 5.15 Example of inserting 35 into a binary heap

89 89

65 32 65 35

48 53 17 27 48 53 32 27

24 13 27 41 24 13 27 41 17

(a) The original heap (b) 35 is inserted into the path

h89, 32, 17i

Figure 5.16 The BinaryHeap.Put operation

BinaryHeap.Put(x, p : Number)
size ← size + 1
if size > SizeOf(elements)
elements ← Expand(elements)
i ← size; elements[i] ← elements[⌊i/2⌋]
// Invariant: 1 ≤ i ≤ size, elements[1..size] forms a heap,
// elements[0..size] contains the elements originally in
// elements[0..size − 1], with elements[i] and elements[⌊i/2⌋]
// being duplicates, and p > elements[j].Key() for
// 2i ≤ j ≤ max(2i + 1, size).
while p > elements[i].Key()
i ← ⌊i/2⌋; elements[i] ← elements[⌊i/2⌋]
elements[i] ← new Keyed(x, p)
CHAPTER 5. PRIORITY QUEUES 183

Figure 5.17 The MakeHeap algorithm

Precondition: x is a Keyed element and t is a BinaryTreeNode whose

children are both heaps.
Postcondition: Forms a heap from t containing x and the elements of t’s
children without changing the shape of t.
MakeHeap(x, t)
L ← t.LeftChild(); R ← t.RightChild()
if L = nil and R = nil
t.SetRoot(x)
else
if L 6= nil
if R 6= nil and L.Root().Key() < R.Root().Key()
largerChild ← R
else
largerChild ← L
else
largerChild ← R
if x.Key() ≥ largerChild.Root().Key()
t.SetRoot(x)
else
t.SetRoot(largerChild.Root())
MakeHeap(x, largerChild)

large as the keys of the roots of all children of t, we can simply replace the
root of t with x, and we are finished. Otherwise, we need to move the root
of the child with larger key to the root of t and make a heap from this child
and x. This is just a smaller instance of the original problem.
We can simplify MakeHeap somewhat when we use it with a binary
heap. First, we observe that once we have determined that at least one child
is nonempty, we can conclude that the left child must be nonempty. We also
observe that the reduction is a transformation to a smaller instance; i.e.,
MakeHeap is tail recursive. We can therefore implement it using a loop.
In order to simplify the statement of the loop invariant, we make use of the
fact that the entire tree is initially a heap, so that the precondition of Make-
Heap could be strengthened to specify that t is a heap. (Later we will use
CHAPTER 5. PRIORITY QUEUES 184

Figure 5.18 The BinaryHeap.RemoveMax operation.

BinaryHeap.RemoveMax()
if size = 0
error
else
m ← elements[1].Data(); size ← size − 1; i ← 1
// Invariant: elements[1..size] forms a heap; 1 ≤ i ≤ size + 1;
// elements[1..i − 1], elements[i + 1..size + 1], and m are
// the elements in the original set;
// elements[size + 1].Key() ≤ elements[⌊i/2⌋].Key();
// and m has maximum key.
while elements[i] 6= elements[size + 1]
j ← 2i
if j > size
elements[i] ← elements[size + 1]
else
if j < size and elements[j].Key() < elements[j + 1].Key()
j ←j+1
if elements[j].Key() ≤ elements[size + 1].Key()
elements[i] ← elements[size + 1]
else
elements[i] ← elements[j]; i ← j
return m

MakeHeap in a context in which we need the weaker precondition.) Figure

5.18 gives the entire RemoveMax operation without a separate MakeHeap
function. Note that elements[size + 1] in Figure 5.18 corresponds to x in
Figure 5.17, elements[i] corresponds to t, and j corresponds to largerChild.
Notice that in RemoveMax, i is initialized to 1, the root of the heap,
and on each iteration that does not cause the while condition to be false,
i is set to j, the index of its larger child. Furthermore, on each iteration,
elements[i] is set to either elements[size + 1] or elements[j]. In the latter
case, the larger child of elements[i] is copied to elements[i], and in the former
case, the removed leaf is placed in its proper location. Thus, as in the Put
operation, an element is inserted into a path in the heap; however, in this
case, the path follows the larger of the children of a node, and the elements
CHAPTER 5. PRIORITY QUEUES 185

Figure 5.19 Example of the RemoveMax operation for a binary heap

89 65

65 32 48 32

48 43 17 27 41 43 17 27

24 33 27 41 24 33 27

(a) The original heap (b) 89 is removed, and 41 is inserted

into the path h65, 48, 33i

preceding the insertion location are moved upward.

For example, suppose we were to perform a RemoveMax on the binary
heap shown in Figure 5.19(a). We would remove 89 and find the path that
follows the larger child of each node. This path is h65, 48, 33i. We would
then insert 41, the last leaf, into this path between 48 and 33, moving the
preceding priorities upward. The result is shown in Figure 5.19(b).
It is easily seen that both Put and RemoveMax operate in Θ(lg n)
time, excluding any time needed to expand the array. Furthermore, as we
saw in Section 4.3, we can amortize the cost of array expansion to constant
time per insertion. The amortized running time for Put is therefore in
Θ(lg n), and the worst-case time for RemoveMax is in Θ(lg n).
We now return to the sorting problem. In order to sort an array A, we
first need to arrange it into a binary heap. One approach is first to make
A[1..n − 1] into a heap, then to insert A[n]. We can easily implement this
bottom-up. The resulting algorithm does n − 1 insertions into heaps of sizes
ranging from 1 to n − 1. The total running time is therefore in
n−1
X
Θ(lg i) ⊆ Θ((n − 1) lg(n − 1)) (from Theorem 3.28)
i=1
= Θ(n lg n).
We can do better, however, by viewing the array A[1..n] as a binary tree
CHAPTER 5. PRIORITY QUEUES 186

in which the parent of A[i] is A[⌊i/2⌋] for i > 1. With this view in mind, the
natural approach seems to be to make the children into heaps first, then use
MakeHeap to make the entire tree into a heap. The resulting algorithm is
easiest to analyze when the tree is completely balanced — i.e., when n + 1
is a power of 2. Let N = n + 1, and let f (N ) give the worst-case running
time for this algorithm. When N is a power of 2, we have

f (N ) ∈ 2f (N/2) + Θ(lg N ).

From Theorem 3.32, f (N ) ∈ Θ(N ) = Θ(n).

This implementation of MakeHeap must be more general than the im-
plementation used for BinaryHeap. Specifically, we must be able to apply
MakeHeap to arbitrary subtrees in order to be able to use it to form the
heap initially. In order to allow us to express the specification of Figure
5.17 in terms of a binary heap, we introduce the notation Tree(A, k, n) to
denote the binary tree formed by using A[k] as the root, Tree(A, 2k, n) as
the left child, and Tree(A, 2k + 1, n) as the right child, provided k ≤ n. If
k > n, Tree(A, k, n) denotes an empty tree. Thus, Tree(A, 1, n) denotes
the binary tree T implied by the array A[1..n], and for k ≤ n, Tree(A, k, n)
denotes the subtree of T rooted at A[k]. The full implementation of Heap-
Sort is shown in Figure 5.20.
It is not hard to show that MakeHeap operates in Θ(lg(n/k)) time in
the worst case. It is easily seen that the first for loop in HeapSort operates
in O(n lg n) time, though in fact a careful analysis shows that it runs in Θ(n)
time, as suggested by the above discussion. It is not hard to show, using
Theorem 3.28, that the second for loop operates in Θ(n lg n) time in the
worst case. Therefore, HeapSort runs in Θ(n lg n) time in the worst case.

5.7 Summary
A heap provides a clean framework for implementing a priority queue. Al-
though LeftistHeaps yield Θ(lg n) worst-case performance for the opera-
tions Put and RemoveMax, the simpler SkewHeaps and Randomized-
Heaps yield O(lg n) amortized and Θ(lg n) expected costs, respectively, for
these operations. BinaryHeaps, while providing no asymptotic improve-
ments over LeftistHeaps, nevertheless tend to be more efficient in practice
because they require less dynamic memory allocation. They also provide the
basis for HeapSort, a Θ(n lg n) in-place sorting algorithm. A summary of
the running times of the PriorityQueue operations for the various imple-
mentations is shown in Figure 5.21.
CHAPTER 5. PRIORITY QUEUES 187

Figure 5.20 HeapSort implementation of Sort, specified in Figure 1.1

HeapSort(A[1..n])
// Invariant: A[1..n] is a permutation of its original elements such
// that for 2(i + 1) ≤ j ≤ n, A[⌊j/2⌋] ≥ A[j].
for i ← ⌊n/2⌋ to 1 by −1
MakeHeap(A[1..n], i, A[i])
// Invariant: A[1..n] is a permutation of its original elements such
// that for 2 ≤ j ≤ i, A[⌊j/2⌋] ≥ A[j], and
// A[1] ≤ A[i + 1] ≤ A[i + 2] ≤ · · · ≤ A[n].
for i ← n to 2 by −1
t ← A[i]; A[i] ← A[1]; MakeHeap(A[1..i − 1], 1, t)

Precondition: A[1..n] is an array of Numbers such that Tree(A, 2k, n)

and Tree(A, 2k + 1, n) form heaps, 1 ≤ k ≤ n, and x is a Number.
Postcondition: Tree(A, k, n) is a heap containing a permutation of the
original values of Tree(A, 2k, n), Tree(A, 2k + 1, n), and x, and no other
elements of A have changed.
MakeHeap(A[1..n], k, x)
A[k] ← sentinel; i ← k
// Invariant: Tree(A, k, n) forms a heap; k ≤ i ≤ n;
// A[i] belongs to Tree(A, k, n);
// the elements of Tree(A, k, n), excluding A[i], are the elements initially
// in Tree(A, 2k, n) and Tree(A, 2k + 1, n);
// all other elements of A have their initial values;
// and if i ≥ 2k, then x < A[⌊i/2⌋].
while A[i] 6= x
j ← 2i
if j > n
A[i] ← x
else
if j < n and A[j] < A[j + 1]
j ←j+1
if A[j] ≤ x
A[i] ← x
else
A[i] ← A[j]; i ← j
CHAPTER 5. PRIORITY QUEUES 188

Figure 5.21 Running times for the PriorityQueue operations for various
implementations.

Put RemoveMax
SortedArrayPriorityQueue Θ(n) Θ(1)
SimpleHeap Θ(n) Θ(1)
LeftistHeap Θ(lg n) Θ(lg n)
SkewHeap O(lg n) O(lg n)
amortized amortized
RandomizedHeap Θ(lg n) Θ(lg n)
expected expected
BinaryHeap Θ(lg n) Θ(lg n)
amortized

Notes:

• n is the number of elements in the priority queue.

• Unless otherwise noted, all running times are worst-case.

• The constructor and the MaxPriority and Size operations all run
is Θ(1) worst-case time for all implementations.

For the implementations that use a Merge function, it is possible to

provide Merge as an operation. However, this operation is not very ap-
propriate for the PriorityQueue ADT because we may need to require
the two priority queues to be of the same type. For example, if we added
a Merge operation to LeftistHeap, we would need to require that the
parameter is also a LeftistHeap — Merge(PriorityQueue) would be
insufficient. Furthermore, we would need to be concerned with security be-
cause the resulting heap would share storage with the original heaps.
Using an immutable structure, as we did for LeftistHeap, would take
care of the security issue. With such implementations, the Merge operation
could be done in Θ(lg n) worst-case time for a LeftistHeap, or in in Θ(lg n)
expected time for a RandomizedHeap, where n is the sum of the sizes of the
two priority queues. The amortized time for SkewHeap.Merge, however,
is not in O(lg n) unless we restrict the sequences of operations so that after
two priority queues are merged, the original priority queues are not used in
CHAPTER 5. PRIORITY QUEUES 189

any subsequent operations; otherwise, we can repeatedly perform the same

expensive Merge.
In Section 5.5, we introduced the basics of expected-case analysis for
randomized algorithms. Specifically, we showed how discrete random vari-
ables can be defined and manipulated in order to analyze expected running
time. In Section 6.4, we will develop this theory more fully.

5.8 Exercises
Exercise 5.1 Complete the implementation of SortedArrayPriority-
Queue shown in Figure 5.2 by adding a constructor and implementations
of the MaxPriority and RemoveMax operations. Prove that your im-
plementation meets its specification.

Exercise 5.2 Prove that SimpleHeap, shown in Figure 5.5, meets its spec-
ification.

Exercise 5.3 Show the result of first inserting the sequence of priorities
below into a leftist heap, then executing one RemoveMax.

34, 12, 72, 15, 37, 49, 17, 55, 45

Exercise 5.4 Prove that LeftistTree, shown in Figure 5.7, meets its
specification.

Exercise 5.5 Prove that LeftistHeap, shown in Figure 5.8, meets its
specification.

* Exercise 5.6 Prove that for any n ∈ N, if we insert a sequence of n

strictly decreasing priorities into an initially empty leftist heap, we obtain a
leftist heap with null path length ⌊lg(n + 1)⌋.

Exercise 5.7 Instead of keeping track of the null path lengths of each node,
a variation on LeftistTree keeps track of the number of nodes in each
subtree, and ensures that the left child has as many nodes as the right child.
We call this variation a LeftHeavyTree.

a. Give an implementation of LeftHeavyTree. The structure must be

immutable, and each constructor must require only Θ(1) time.
CHAPTER 5. PRIORITY QUEUES 190

Figure 5.22 The HasPriority interface

Precondition: true.
Postcondition: Returns a number representing a priority.
HasPriority.Priority()

b. Prove by induction on the number of nodes n in the tree that in any

LeftHeavyTree, the distance from the root to the rightmost empty
subtree is no more than lg(n + 1).
c. Using the result of part b, show that if we use LeftHeavyTrees
instead of LeftistTrees in the implementation of LeftistHeap,
the running times of the operations are still in O(lg n), where n is the
number of elements in the priority queue.

Exercise 5.8 Repeat Exercise 5.3 using a skew heap instead of a leftist
heap.

Exercise 5.9 Prove that SkewHeap, obtained by replacing the Merge

function in SimpleHeap (Figure 5.5) with the function shown in Figure
5.9, meets its specification.

* Exercise 5.10 Another way of specifying a priority queue is to define an

interface HasPriority, as shown in Figure 5.22. Rather than supplying
two arguments to the Put operation, we could instead specify that it takes
a single argument of type HasPriority, where the priority of the item is
given by its Priority operation. Discuss the potential security problems for
this approach. How could these problems be avoided if such a specification
were adopted?

Exercise 5.11 The goal of this exercise is to complete the analysis of the
amortized running times of the SkewHeap operations.
a. Prove by induction on n that in any binary tree T with n nodes, the
number of good nodes on its rightmost path is no more than lg(n + 1),
where the definition of a good node is as in Section 5.4.
b. Prove that in the SkewHeap.Merge operation (shown in Figure 5.9
on page 166) if L is initially a bad node, then it is a good node in the
resulting heap.
CHAPTER 5. PRIORITY QUEUES 191

c. Given two skew heaps to be merged, let us define the potential of each
node to be 0 if the node is good, or 1 if the node is bad. Using the
results from parts a and b above, prove that the actual cost of the
Merge operation, plus the sum of the potentials of the nodes in the
resulting heap, minus the sum of potentials of the nodes in the two
original heaps, is in O(lg n) where n is the number of keys in the two
heaps together.
d. Using the result of part c, prove that the amortized running times of
the SkewHeap operations are in O(lg n), where n is the number of
nodes in the heap.

Exercise 5.12 Prove that RandomizedHeap, obtained by replacing the

Merge function in SimpleHeap (Figure 5.5) with the function shown in
Figure 5.12, meets its specification.

Exercise 5.13 Prove by induction on n that any binary tree with n nodes
has exactly n + 1 empty subtrees.

Exercise 5.14 Prove by induction on the number of nodes in a binary tree

T , that the sum of the probabilities in PathT is 1.

Exercise 5.15 The goal of this exercise is to prove Theorem 5.5. Let f :
S → N be a discrete random variable.
a. Prove by induction on n that
n
X n
X
iP (f = i) = P (f ≥ i) − nP (f ≥ n + 1).
i=0 i=1

b. Prove that for every n ∈ N ,

∞
X n
X
iP (f = i) ≥ iP (f = i) + nP (f ≥ n + 1).
i=0 i=0

c. Using the fact that if g(i) ≥ 0 for all i, then

∞
X n
X
g(i) = lim g(i),
n→∞
i=0 i=0

prove that if E[f ] is finite, then

lim nP (f ≥ n + 1) = 0.
n→∞
CHAPTER 5. PRIORITY QUEUES 192

d. Prove Theorem 5.5.

Exercise 5.16 Prove Theorem 5.9.

Exercise 5.17 Let S be the set of all sequences of four flips of a fair coin,
where each sequence has probability 1/16. Let h be the discrete random
variable giving the number of heads in the sequence.

a. Compute E[h].

b. Compute E[h2 ], and show that E[h2 ] 6= (E[h])2 .

c. Compute E[2h ], and show that E[2h ] 6= 2E[h] .

Exercise 5.18 Use Example 5.4 to show that the expected running time of
RandomizedHeap.Merge, shown in Figure 5.12, is in Ω(lg n) in the worst
case, where n is the number of elements in the two heaps combined.

Exercise 5.19 Complete the implementation of BinaryHeap by adding

a constructor and a MaxPriority operation to the operations shown in
Figures 5.16 and 5.18. Prove that the resulting implementation meets its
specification.

Exercise 5.20 Repeat Exercise 5.3 using a binary heap instead of a leftist
heap. Show the result as both a tree and an array.

Exercise 5.21 Prove that HeapSort, shown in Figure 5.20, meets its spec-
ification.

Exercise 5.22 Prove that the first loop in HeapSort runs in Θ(n) time
in the worst case.

Exercise 5.23 Prove that HeapSort runs in Θ(n lg n) time in the worst
case.

Exercise 5.24 We can easily modify the Sort specification (Figure 1.2 on
page 6) so that instead of sorting numbers, we are sorting Keyed items in
nondecreasing order of their keys. HeapSort can be trivially modified to
meet this specification. Any sorting algorithm meeting this specification is
said to be stable if the resulting sorted array always has elements with equal
keys in the same order as they were initially. Show that HeapSort, when
modified to sort Keyed items, is not stable.
CHAPTER 5. PRIORITY QUEUES 193

Exercise 5.25 Consider the following scheduling problem. We have a col-

lection of jobs, each having a natural number ready time ri , a positive integer
execution time ei , and a positive integer deadline di , such that di ≥ ri + ei .
At each natural number time instant t, we wish to schedule the job with
minimum deadline satisfying the following conditions:

• t ≥ ri (i.e., the job is ready); and

• if the job has already been executed for a < ei time units, then t +
ei − a ≤ di (i.e., the job can meet its deadline).

Note that this scheduling strategy may preempt jobs, and that it will discard
jobs that have been delayed so long that they can no longer meet their
deadlines. Give an algorithm to produce such a schedule, when given a
sequence of jobs ordered by ready time. Your algorithm should store the
ready jobs in an InvertedPriorityQueue. (You do not need to give an
implementation of InvertedPriorityQueue.) Show that your algorithm
operates in O(k lg n) time, where k is length of the schedule and n is the
number of jobs. You may assume that k ≥ n and that Put and RemoveMin
both operate in Θ(lg n) time in the worst case.

Exercise 5.26 The game of craps consists of a sequence of rolls of two six-
sided dice with faces numbered 1 through 6. The first roll is known as the
come-out roll. If the come-out roll is a 7 or 11 (the sum of the top faces
of the two dice), the shooter wins. If the come-out roll is a 2, 3, or 12,
the shooter loses. Otherwise, the result is known as the point. The shooter
continues to roll until the result is either the point (in which case the shooter
wins) or a 7 (in which case the shooter loses).

a. For each of the values 2 through 12, compute the probability that any
single roll is that value.

b. A field bet can be made on any roll. For each dollar bet, the payout
is determined by the roll as follows:

• 2 or 12: $3 (i.e., the bettor pays $1 and receives $3, netting $2);
• 3, 4, 9, 10 or 11: $2;
• 5, 6, 7, or 8: 0.

Calculate the expected payout for a field bet.

CHAPTER 5. PRIORITY QUEUES 194

c. A pass-line bet is a bet, placed prior to the come-out roll, that the
shooter will win. For each dollar bet, the payout for a win is $2,
whereas the payout for a loss is 0. Compute the expected payout for a
pass-line bet. [Hint: The problem is much easier if you define a finite
probability space, ignoring those rolls that don’t affect the outcome.
In order to do this you will need to use conditional probabilities (e.g.,
given that the roll is either a 5 or a 7, the probability that it is a 5).]

Exercise 5.27 Let S be a discrete probability space, and let f be a discrete

random variable over S. Let a be any positive real number. Prove Markov’s
Inequality:
P (f ≥ a) ≤ E[f ]/a. (5.3)

5.9 Chapter Notes

Both heaps and heap sort were introduced by Williams [113]. The linear-
time construction of a binary heap is due to Floyd [39]. Leftist heaps were
introduced by Crane [26]; see also Knuth [80]. Skew heaps were introduced
by Sleator and Tarjan [100]. Randomized heaps were introduced by Gambin
and Malinowski [46].
Other implementations of priority queues have been defined based on the
idea of a heap. For example, binomial queues were introduced by Vuillemin
[108]. Lazy binomial queues and Fibonacci heaps, each of which provide
Put and RemoveMax operations with amortized running times in O(1)
and O(lg n), respectively, were introduced by Fredman and Tarjan [44].
The information on craps in Exercise 5.26 is taken from Silberstang [98].
Chapter 6

Storage/Retrieval I: Ordered
Keys

In this chapter, we begin an examination of data structures for general

storage and retrieval. We will assume that with each data item is associated
a key that uniquely identifies the data item. An example of this kind of key
might be a bank account number. Thus, if we provide a bank’s database with
a customer’s account number, we should be able to retrieve that customer’s
account information.
We will let Key denote the type to which the keys belong. We will
assume that, even though the Key may not be a numeric type, it is still
possible to sort elements of this type using a comparison operator ≤. More-
over, we assume that we need an efficient way to obtain data items in order
of their keys. In the next chapter, we will examine data structures that do
not require these two assumptions.
In order to facilitate arbitrary processing of all of the items in the data
structure, we provide the Visitor interface, shown in Figure 6.1. We can
then define an implementation of Visitor so that its Visit() operation does
whatever processing we wish on a data item. (The use of this interface is
known as the visitor pattern.)
We then define a Dictionary as a finite set of data items, each having
a unique key of type Key, together with the operations shown in Figure
6.2. In the next chapter, we will examine implementations of this ADT. In
this chapter, we will explore several implementations of the OrderedDic-
tionary ADT, which is the extension of the Dictionary ADT obtained
by adding the interface shown in Figure 6.3.

195
CHAPTER 6. STORAGE/RETRIEVAL I: ORDERED KEYS 196

Figure 6.1 The Visitor interface

Precondition: true.
Postcondition: Completes without an error. May modify the state of x.
Visitor.Visit(x)

Figure 6.2 The Dictionary ADT

Precondition: true.
Postcondition: Constructs an empty Dictionary.
Dictionary()
Precondition: k is a Key.
Postcondition: Returns the element with key k, or nil if no item with key
k is contained in the set.
Dictionary.Get(k)
Precondition: x 6= nil, and k is a Key that is not associated with any
item in the set.
Postcondition: Adds x to the set with key k.
Dictionary.Put(x, k)
Precondition: k is a Key.
Postcondition: If there is an item with key k in the set, this item is
removed.
Dictionary.Remove(k)
Precondition: true.
Postcondition: Returns the number of items in the set.
Dictionary.Size()
CHAPTER 6. STORAGE/RETRIEVAL I: ORDERED KEYS 197

Figure 6.3 Interface extending Dictionary to OrderedDictionary

Precondition: v is a Visitor.
Postcondition: Applies v.Visit(x) to every item x in the set in order of
their keys.
OrderedDictionary.VisitInOrder(v)

Figure 6.4 Printer implementation of Visitor

Structural Invariant: true.

Precondition: true.
Postcondition: Prints x.
Printer.Visit(x)
print x

Example 6.1 Suppose we would like to print all of the data items in an in-
stance of OrderedDictionary in order of keys. We can accomplish this by
implementing Visitor so that its Visit operation prints its argument. Such
an implementation, Printer, is shown in Figure 6.4. Note that we have
strengthened the postcondition over what is specified in Figure 6.1. This is
allowable because an implementation with a stronger postcondition and/or
a weaker precondition is still consistent with the specification. Having de-
fined Printer, we can print the contents of the OrderedDictionary d
with the statement,
d.VisitInOrder(new Printer())
In order to implement OrderedDictionary, it is possible to store the
data items in a sorted array, as we did with SortedArrayPriorityQueue
in Section 5.1. Such an implementation has similar advantages and disad-
vantages to those of SortedArrayPriorityQueue. Using binary search,
we can find an arbitrary data item in Θ(lg n) time, where n is the number of
items in the dictionary. Thus, the Get operation can be implemented to run
in Θ(lg n) time. However, to add or remove an item requires Θ(n) time in
the worst case. Thus, Put and Remove are inefficient using such an imple-
mentation. The fact that the elements of an array are located contiguously
CHAPTER 6. STORAGE/RETRIEVAL I: ORDERED KEYS 198

Figure 6.5 A binary search tree

54

23 77

13 35 64 92

17 28 41 71

allows random access, which, together with the fact that the elements are
sorted, facilitates the fast binary search algorithm. However, it is exactly
this contiguity that causes updates to be slow, because in order to maintain
sorted order, elements must be moved to make room for new elements or to
take the place of those removed.
If we were to use a linked list instead of an array, we would be able
to change the structure without moving elements around, but we would no
longer be able to use binary search. The “shape” of a linked list demands
a sequential search; hence, look-ups will be slow. In order to provide fast
updates and retrievals, we need a linked structure on which we can approx-
imate a binary search.

6.1 Binary Search Trees

Consider the binary tree structure shown in Figure 6.5. Integer keys of
several data items are shown in the nodes. Note that the value of the
root is roughly the median of the keys in the structure; i.e., about half of
the remaining keys are smaller. The smaller keys are all in the left child,
whereas the larger keys are all in the right child. The two children are then
structured in a similar way. Thus, if our search target is smaller than the
CHAPTER 6. STORAGE/RETRIEVAL I: ORDERED KEYS 199

key at the current node, we look in the left child, and if it is larger, we look
in the right child. Because this type of search approximates a binary search,
this structure is called a binary search tree.
More formally we define a binary search tree (BST) to be a binary tree
satisfying the following properties:

• Each node contains a data item together with its key.

• If the BST is nonempty, then both of its children are BSTs, and the
key in its root node is

– strictly larger than all keys in its left child; and

– strictly smaller than all keys in its right child.

We can represent a binary search tree with the following variables:

• elements, which refers to a BinaryTreeNode (see Section 5.2); and

• size, which is an integer giving the number of data items in the set.

We interpret elements as representing a binary search tree as follows. If its

root variable is nil, then the represented BST is empty. Otherwise, its root
variable is a Keyed item (see page 148) containing the data item stored in
the root with its associated key, and its leftChild and rightChild variables
represent the left and right children, respectively. Our structural invariant
is that elements refers to a binary search tree according to the above inter-
pretation, and that size gives the number of nodes in the represented binary
search tree. Note that this invariant implies that all non-nil root variables
in the BinaryTreeNodes refer to Keyed items.
Because both Get and Remove require that we find a node with a given
key, we will design an internal function Find(k, t) to find the node in the
BST rooted at t with key k. If no such key exists in the tree, Find(k, t) will
return a reference to the empty subtree in which it could be inserted; thus,
we can also use Find in implementing Put.
Consider how we might implement Find(k, t). If t is empty, then obvi-
ously it does not contain the key. Otherwise, we should first check the root.
If we don’t find our key, we should see how it compares with the key at the
root. Proceeding top-down, we can then find the key in either the left or
right child, depending on the outcome of the comparison. This algorithm
is a transformation, and so can be implemented as a loop, as is shown in
Figure 6.6.
CHAPTER 6. STORAGE/RETRIEVAL I: ORDERED KEYS 200

Figure 6.6 BSTDictionary implementation of OrderedDictionary,

part 1

Structural Invariant: elements refers to a BST, and size gives the

number of nodes in the represented BST.

BSTDictionary()
elements ← new BinaryTreeNode(); size ← 0

BSTDictionary.Get(k)
return Find(k, elements).Root().Data()

BSTDictionary.Put(x, k)
if x = nil
error
else
t ← Find(k, elements)
if t.Root() = nil
size ← size + 1
t.SetRoot(new Keyed(x, k))
t.SetLeftChild(new BinaryTreeNode())
t.SetRightChild(new BinaryTreeNode())

— Internal Functions Follow —

Precondition: k is a key and t is a BinaryTreeNode representing a
BST.
Postcondition: Returns the BinaryTreeNode in t with key k, or in
which k could be inserted if k is not in t.
BSTDictionary.Find(k, t)
// Invariant: k belongs in the subtree represented by t
while t.Root() 6= nil and k 6= t.Root().Key()
if k < t.Root().Key()
t ← t.LeftChild()
else
t ← t.RightChild()
return t
CHAPTER 6. STORAGE/RETRIEVAL I: ORDERED KEYS 201

Complete implementations of Get and Put are shown in Figure 6.6.

The implementation of Remove requires a bit more work, however. Once
we find the node containing the key k, we need to be able to remove that
node t while maintaining a binary search tree. If t has no nonempty children,
all we need to do is remove it (i.e., make the subtree empty). If t has only
one nonempty child, we can safely replace the node to which t refers by
that child. The difficulty comes when t has two nonempty children. In
order to take care of this case, we replace t with the node in its right child
having smallest key. That node must have an empty left child, so we can
easily remove it. Furthermore, because it is in the right child of t, its key is
greater than any key in the left child of t; hence, moving it to the root of t
maintains the BST structure.
We can find the smallest key in a nonempty BST by first looking at the
left child of the root. If it is empty, then the root contains the smallest key.
Otherwise, the smallest key is in the left child. This is a transformation,
and so can be implemented using a loop. The complete implementation of
Remove is shown in Figure 6.7. Figure 6.8 shows the result of deleting 54
from the BST shown in Figure 6.5. Specifically, because 54 has two children,
it is replaced by the smallest key (64) in its right child, and 64 is replaced
by its right child (71).
The VisitInOrder operation requires us to apply v.Visit to each data
item in the BST, in order of keys. If the BST is empty, then there is
nothing to do. Otherwise, we must visit all of the data items in the left
child prior to visiting the root, then we must visit all of the data items in
the right child. Because the left and right children are themselves BSTs,
they comprise smaller instances of this problem. Applying the top-down
approach in a straightforward way, we obtain the recursive internal function
TraverseInOrder shown in Figure 6.7.
The above algorithm implemented by TraverseInOrder is known as
an inorder traversal. Inorder traversal applies strictly to binary trees, but
two other traversals apply to rooted trees in general. A preorder traversal
visits the root prior to recursively visiting all of its children, whereas a
postorder traversal visits the root after recursively visiting all of its children.
Let us now analyze the running time of Find. Let n be the number
of data items in the BST. Clearly, the time required outside the loop and
the time for a single iteration of the loop are each in Θ(1). We therefore
need to analyze the worst-case number of iterations of the loop. Initially, t
refers to a BST with n nodes. A single iteration has the effect of resetting
t to refer to one of its children. In the worst case, this child may contain
all nodes except the root. Thus, in the worst case, the loop may iterate n
CHAPTER 6. STORAGE/RETRIEVAL I: ORDERED KEYS 202

Figure 6.7 BSTDictionary implementation of OrderedDictionary,

part 2

BSTDictionary.Remove(k)
t ← Find(k, elements)
if t.Root() 6= nil
if t.LeftChild().Root() = nil
Copy(t.RightChild(), t)
else if t.RightChild().Root() = nil
Copy(t.LeftChild(), t)
else
m ← t.RightChild()
// Invariant: The smallest key in the right child of t is in the
// subtree rooted at m.
while m.LeftChild().Root() 6= nil
m ← m.LeftChild()
t.SetRoot(m.Root()); Copy(m.RightChild(), m)

BSTDictionary.VisitInOrder(v)
TraverseInOrder(elements, v)

— Internal Functions Follow —

Precondition: source and dest are BinaryTreeNodes.
Postcondition: Copies the contents of source to dest.
BSTDictionary.Copy(source, dest)
dest.SetRoot(source.Root())
dest.SetLeftChild(source.LeftChild())
dest.SetRightChild(source.RightChild())

Precondition: t is a BinaryTreeNode representing a BST, and v is a

Visitor
Postcondition: Applies v.Visit to every node in t in order of their keys.
BSTDictionary.TraverseInOrder(t, v)
if t.Root() 6= nil
TraverseInOrder(t.LeftChild(), v)
v.Visit(t.Root().Data())
TraverseInOrder(t.RightChild(), v)
CHAPTER 6. STORAGE/RETRIEVAL I: ORDERED KEYS 203

Figure 6.8 The result of deleting 54 from the BST shown in Figure 6.5 —
54 is replaced by 64, which in turn is replaced by 71.

64

23 77

13 35 71 92

17 28 41

times. This can happen, for example, if all left children are empty, so that
elements refers to a BST that consists of a single chain of nodes going to
the right (see Figure 6.9). The worst-case running time is therefore in Θ(n).

Example 6.2 Suppose we build a BSTDictionary by inserting n items

with integer keys 1, 2, . . . , n, in that order. As each key is inserted, it is
larger than any key already in the BST. It is therefore inserted to the right
of every key already in the BST. The result is shown in Figure 6.9. It is
easily seen that to insert key i requires Θ(i) time. The total time to build
the BSTDictionary is therefore in Θ(n2 ), by Theorem 3.28.
The worst-case performance of a binary search tree is no better than
either a sorted array or a sorted linked list for implementing OrderedDic-
tionary. In fact, in the worst case, a binary search tree degenerates into
a sorted linked list, as Figure 6.9 shows. However, the performance of a
binary search tree is not nearly as bad as the worst-case analysis suggests.
In practice, binary search trees often yield very good performance.
In order to help us to understand better the performance of a binary
search tree, let us analyze the worst-case running time of Find in a some-
what different way. Instead of analyzing the running time in terms of the
CHAPTER 6. STORAGE/RETRIEVAL I: ORDERED KEYS 204

Figure 6.9 A worst-case binary search tree

..
.
n

number of data items, let us instead analyze it in terms of the height of

the tree. The analysis is similar, because again in the worst case the child
selected in the loop has height one less than the entire subtree. Thus, the
worst-case running time of Find is in Θ(h), where h is the height of the
tree.
What this analysis tells us is that in order to achieve good performance
from a binary search tree, we need a tree whose height is not too large.
In practice, provided the pattern of insertions and deletions is somewhat
random, the height of a binary search tree tends to be in Θ(lg n). It is not
hard to see that the worst-case running time of Get, Put, and Remove
are all in Θ(h). Thus, if the height of the tree is logarithmic, we can achieve
good performance. In the next section, we will show how we can modify
our implementation to guarantee logarithmic height in the worst case, and
thereby achieve logarithmic performance from these three operations.
Before we leave ordinary binary search trees, however, let us analyze
the worst-case running time of TraverseInOrder, which does not use the
Find function. We cannot analyze this operation completely because we
cannot analyze v.Visit without knowing how it is implemented. Assuming
the correctness of TraverseInOrder (whose proof we leave as an exercise),
we can nevertheless conclude that v.Visit is called exactly once for each data
item, and so must take at least Ω(n) time. What we would like to analyze
is the time required for everything else. This amounts to analyzing the
overhead involved in applying v.Visit to every data item.
Ignoring the call to v.Visit and the recursive calls, it is easily seen that
CHAPTER 6. STORAGE/RETRIEVAL I: ORDERED KEYS 205

the remaining code runs in Θ(1) time. However, setting up a recurrence de-
scribing the worst-case running time, including the recursion but excluding
calls to v.Visit, is not easy. We must make two recursive calls, but all we
know about the sizes of the trees in these calls is that their sum is one less
than the size of the entire tree.
Let us therefore take a different approach to the analysis. As we have
already argued, v.Visit is called exactly once for each data item. Further-
more, it is easily seen that, excluding the calls made on empty trees, v.Visit
is called exactly once in each call to TraverseInOrder. A total of exactly
n calls are therefore made on nonempty trees. The calls made on empty trees
make no further recursive calls. We can therefore obtain the total number
of recursive calls (excluding the initial call made from VisitInOrder) by
counting the recursive calls made by each of the calls on nonempty trees.
Because each of these calls makes two recursive calls, the total number of
recursive calls is exactly 2n. Including the initial call the total number of
calls made to TraverseInOrder is 2n + 1. Because each of these calls
runs in Θ(1) time (excluding the time taken by v.Visit), the total time is
in Θ(n). Note that we cannot hope to do any better than this because the
specification requires that v.Visit be called n times.

6.2 AVL Trees

In this section, we present a variant of binary search trees that guarantees
logarithmic height. As a result, we obtain an implementation of Ordered-
Dictionary for which the Get, Put, and Remove operations all run in
Θ(lg n) time.
The key to keeping the height of a binary tree relatively small is in
maintaining balance. The trick is to be able to achieve balance without too
much overhead; otherwise the overhead in maintaining balance may result in
poor overall performance. Consequently, we need to be careful how we define
“balance”, so that our balance criterion is not too difficult to maintain.
The balance criterion that we choose is that in any subtree, the heights
of the two children differ by at most 1. For the purpose of this definition, we
consider an empty tree to have height −1, or one less than the height of a tree
containing a single node. A binary search tree obeying this balance criterion
is known as an AVL tree; “AVL” stands for the names of its inventors,
Adel’son-Vel’skiı̆ and Landis.
Figure 6.10 shows an AVL tree of height 4 containing integer keys. Note
that its balance is not perfect – it is not hard to construct a binary tree of
CHAPTER 6. STORAGE/RETRIEVAL I: ORDERED KEYS 206

Figure 6.10 An AVL tree

55

42 79

31 53 74 86

25 34 47 61

12

height 3 with even more nodes. Nevertheless, the children of each nonempty
subtree have heights differing by at most 1, so it is an AVL tree.
Before we begin designing an AVL tree implementation of Ordered-
Dictionary, let us first derive an upper bound on the height of an AVL
tree with n nodes. We will not derive this bound directly. Instead, we will
first derive a lower bound on the number of nodes in an AVL tree of height
h. We will then transform this lower bound into our desired upper bound.
Consider an AVL tree with height h having a minimum number of nodes.
By definition, both children of a nonempty AVL tree must also be AVL trees.
By definition of the height of a tree, at least one child must have height h−1.
By definition of an AVL tree, the other child must have height at least h − 2.
In order to minimize the number of nodes in this child, its height must be
exactly h − 2, provided h ≥ 1. Thus, the two children are AVL trees of
heights h − 1 and h − 2, each having a minimum number of nodes.
The above discussion suggests a recurrence giving the minimum number
of nodes in an AVL tree of height h. Let g(h) give this number. Then for
h ≥ 1, the number of nodes in the two children are g(h − 1) and g(h − 2).
Then for h ≥ 1,
g(h) = g(h − 1) + g(h − 2) + 1, (6.1)
where g(−1) = 0 (the number of nodes in an empty tree) and g(0) = 1 (the
number of nodes in a tree of height 0).
CHAPTER 6. STORAGE/RETRIEVAL I: ORDERED KEYS 207

Example 6.3 Consider the subtree rooted at 25 in Figure 6.10. It is of

height 1, and its two children are minimum-sized subtrees of height 0 and
−1, respectively. It is therefore a minimum-sized AVL tree of height 1, so
g(1) = 2. Likewise, the subtrees rooted at 53 and 74 are also minimum-sized
AVL trees of height 1. The subtrees rooted at 31 and 79 are then easily seen
to be minimum-sized AVL trees of height 2, so g(2) = 4. In like manner it
can be seen that g(3) = 7, g(4) = 12, and the entire tree is a minimum-sized
AVL tree of height 4.
Recurrence (6.1) does not fit any of the forms we saw in Chapter 3.
However, it is somewhat similar to the form of Theorem 3.31. Recall that
we only need a lower bound for g(n). In what follows, we will derive a
recurrence that fits the form of Theorem 3.31 and gives a lower bound for
g(n).
We first observe that g must be a nondecreasing function. Thus, for
h ≥ 1, if
g1 (h) = 2g1 (h − 2) + 1
where g1 (h) = g(h) for h < 1, then

g1 (h) ≤ g(h) (6.2)

for all h ≥ −1.

Now if we let g2 (h) = g1 (2h) for all h, we obtain

g2 (h) = g1 (2h)
= 2g1 (2h − 2) + 1
= 2g1 (2(h − 1)) + 1
= 2g2 (h − 1) + 1.

g2 then fits the form of Theorem 3.31. Applying this theorem, we obtain

g2 (h) ∈ Θ(2h ).

Thus, for sufficiently large h, there is a positive real number c1 such that

g1 (2h) = g2 (h)
≥ c1 2h .

Then for sufficiently large even h,

g1 (h) ≥ c1 2h/2 .
CHAPTER 6. STORAGE/RETRIEVAL I: ORDERED KEYS 208

For sufficiently large odd h, we have

g1 (h) ≥ g1 (h − 1)
≥ c1 2(h−1)/2 (because h − 1 is even)
c1
= √ 2h/2 ,
2
so that for some positive real number c2 and all sufficiently large h,

g1 (h) ≥ c2 2h/2 . (6.3)

Combining (6.3) with (6.2), we obtain

c2 2h/2 ≤ g(h)

for sufficiently large h. Applying lg to both sides and rearranging terms, we

obtain

h ≤ 2(lg g(h) − lg c2 )
∈ O(lg g(h)).

Because g(h) is the minimum number of nodes in an AVL tree of height

h, it follows that the height of an AVL tree is in O(lg n), where n is the
number of nodes. By a similar argument, it can be shown that the height is
in Ω(lg n) as well. We therefore have the following theorem.

Theorem 6.4 The worst-case height of an AVL tree is in Θ(lg n), where n
is the number of nodes.
By Theorem 6.4, if we can design operations that run in time linear in
the height of an AVL tree, these operations will run in time logarithmic in
the size of the data set. Certainly, adding or deleting a node will change the
heights of some of the subtrees in an AVL tree; hence, these operations must
re-establish balance. Computing the height of a binary tree involves finding
the longest path, which apparently requires examining the entire tree. How-
ever, we can avoid recomputing heights from scratch if we record the height
of each subtree. If the heights of both children are known, computing the
height of the tree is straightforward.
We therefore define the data type AVLNode, which is just like Binary-
TreeNode, except that it has an additional representation variable, height.
This variable is used to record the height of the tree as an integer. As for the
other three variables, we allow read/write access to height. The constructor
CHAPTER 6. STORAGE/RETRIEVAL I: ORDERED KEYS 209

for AVLNode is just like the constructor for BinaryTreeNode, except

that it also initializes height to −1.
To represent an OrderedDictionary using an AVL tree, we again
use two variables, elements and size, as we did for BSTDictionary. In
this representation, however, elements will refer to an AVLNode. Our
structural invariant is that elements represents an AVL tree. We interpret
this statement as implying that each height variable gives the height of the
subtree at which it is rooted, or −1 if that subtree is empty.
We could define a Find function for this implementation as we did for
BSTDictionary; in fact, because an AVL tree is a binary search tree, the
same function would work. However, this function would not be useful in
implementing the Put or Remove operations because we might need to
change the shape of the tree at some other location in order to maintain the
balance criterion.
Let us therefore consider how Put might be implemented. More gener-
ally, let us consider how a data item x might be inserted into an arbitrary
AVL tree t, which may be a subtree of a larger AVL tree. To a certain
extent, we need to proceed as in BSTDictionary.Find. Specifically, if t
is empty, we can replace it with a single-node AVL tree containing x. Oth-
erwise, we’ll need to compare keys and insert into the appropriate child.
However, we are not yet finished, because the insertion into the child will
have changed its shape; hence, we need to compare the heights of the two
children and restore balance if necessary. Note that this reduction is not a
transformation, due to the additional work required following the insertion
into the child.
In order to complete the insertion function, we need to be able to restore
the balance criterion after an insertion into one of the children. Clearly,
if we insert into one particular child, the other child will be unchanged.
Furthermore, if we specify the insertion function to cause the result to be
an AVL tree, we know that both children will be AVL trees; hence, we only
need to worry about restoring balance at the root. Before we can talk about
how to restore balance at the root, we should consider how much difference
there might be in the heights of the children. It stands to reason that an
insertion should either leave the height unchanged or increase it by 1. We
will therefore include this condition in the postcondition of our insertion
function.
Based on the above discussion, it suffices to show how to balance a binary
search tree whose children are AVL trees having a height difference of exactly
2. This restoration of balance is accomplished via rotations. Consider, for
example, the rotation shown in Figure 6.11. In this figure, circles denote
CHAPTER 6. STORAGE/RETRIEVAL I: ORDERED KEYS 210

Figure 6.11 A single rotate right

d b

b e a d

a c c e

single nodes, and triangles denote arbitrary subtrees, which in some cases
may be empty. All nodes and subtrees are labeled in a way that corresponds
to the order of nodes in a BST (e.g., subtree c is to the right of node b and
to the left of node d). The rotation shown is known as a single rotate
right. It is accomplished by promoting node b to the root, then filling in the
remaining pieces in the only way that maintains the ordering of keys in the
BST. Suppose that in the “before” picture, the right child (e) has height h
and the left child has height h + 2. Because the left child is an AVL tree,
one of its two children has a height of h + 1 and the other has a height of
either h or h + 1. Suppose subtree a has a height of h + 1. Then it is easily
seen that this rotation results in an AVL tree.
The rotation shown in Figure 6.11 does not restore balance, however, if
subtree a has height h. Because the left child in the “before” picture has
height h+2, subtree c must have height h+1 in this case. After the rotation,
the left child has height h, but the right child has height h + 2. To take care
of this case, we need another kind of rotation called a double rotate right,
shown in Figure 6.12. It is accomplished by promoting node d to the root
and again filling in the remaining pieces in the only way that maintains the
ordering of keys. Suppose that subtrees a and g have height h and that the
subtree rooted at d in the “before” picture has height h + 1. This is then the
case for which a single rotate fails to restore balance. Subtrees c and e may
have heights of either h or h − 1 (though at least one must have height h).
It is therefore easily seen that following the rotation, balance is restored.
These two rotations handle the cases in which the left child has height
2 greater than the right child. When the right child has height 2 greater
than the left child a single rotate left or a double rotate left may be applied.
These rotations are simply mirror images of the rotations shown in Figures
CHAPTER 6. STORAGE/RETRIEVAL I: ORDERED KEYS 211

Figure 6.12 A double rotate right

f d

b g b f

a d a c e g

c e

6.11 and 6.12.

To complete our discussion of the insertion function, we must convince
ourselves that if it changes the height of the tree, then it increases it by
exactly 1. This is clearly the case if no rotation is done. Let us then
consider the rotations shown in Figures 6.11 and 6.12. If either of these
rotations is applied, then the data item must have been inserted into the
left child, causing its height to increase from h + 1 to h + 2. The overall
height of the tree had to have been h + 2 prior to the insertion. Following a
single rotate right, it is easily seen that the height is either h + 2 or h + 3.
Likewise, following a double rotate right, it is easily seen that the height is
h+2. Thus, the result of the insertion is to either leave the height unchanged
or increase it by 1.
The insertion algorithm is shown as AVLDictionary.Insert in Figure
6.13. The Balance function can also be used by the deletion algorithm.
The remainder of the AVLDictionary implementation, including the ro-
tations, is left as an exercise. Note that the rotations must ensure that all
height values except that of the root are correct. Specifically, the heights of
node d in Figure 6.11 and nodes b and f in Figure 6.12 must be recomputed.

Example 6.5 Suppose we were to insert the key 39 into the AVL tree shown
in Figure 6.14(a). Using the ordinary BST insertion algorithm, 39 should
be made the right child of 35, as shown in Figure 6.14(b). To complete
the insertion, we must check the balance along the path to 39, starting at
the bottom. Both 35 and 23 satisfy the the balance criterion; however,
the left child of 42 has height 2, whereas the right child has height 0. We
CHAPTER 6. STORAGE/RETRIEVAL I: ORDERED KEYS 212

Figure 6.13 Some internal functions for AVLDictionary implementation

of OrderedDictionary

Precondition: t represents an AVL tree, and x is a Keyed item.

Postcondition: Inserts x into t if its key is not already there, resulting in
an AVL tree whose height is either unchanged or increased by 1.
AVLDictionary.Insert(x, t)
if t.Height() = −1
t.SetRoot(x)
t.SetLeftChild(new AVLNode())
t.SetRightChild(new AVLNode())
t.SetHeight(0)
else if x.Key() < t.Root().Key()
Insert(x, t.LeftChild())
Balance(t)
else if x.Key() > t.Root().Key()
Insert(x, t.RightChild())
Balance(t)

Precondition: t refers to a nonempty AVLNode representing a BST. The

children of t are AVL trees whose heights differ by at most 2.
Postcondition: Arranges t into an AVL tree.
AVLDictionary.Balance(t)
l ← t.LeftChild()
r ← t.RightChild()
if l.Height() = r.Height() + 2
if l.LeftChild().Height() > r.Height()
SingleRotateRight(t)
else
DoubleRotateRight(t)
else if r.Height() = l.Height() + 2
if r.RightChild().Height() > l.Height()
SingleRotateLeft(t)
else
DoubleRotateLeft(t)
t.SetHeight(Max(l.Height(), r.Height()) + 1)
CHAPTER 6. STORAGE/RETRIEVAL I: ORDERED KEYS 213

Figure 6.14 Example of inserting 39 into an AVL tree

54 54 54

42 67 42 67 35 67

23 50 73 23 50 73 23 42 73

11 35 11 35 11 39 50

39
(a) The original tree (b) 39 is inserted as (c) A double rotate
into an ordinary right is done at
BST 42

therefore need to perform a rotation at 42. To determine which rotation

is appropriate, we compare the height of the left child of the left child of
42 (i.e., the subtree rooted at 11) with the right child of 42. Because both
of these subtrees have height 0, a double rotate right is required at 42.
To accomplish this rotation, we promote 35 to the root of the subtree (i.e.,
where 42 currently is), and place the nodes 23 and 42, along with the subtrees
rooted at 11, 39, and 50, at the only locations that preserve the order of
the BST. The result of this rotation is shown in Figure 6.14(c). Because the
balance criterion is satisfied at 54, this tree is the final result.
It is not hard to see that each of the rotations can be implemented to
run in Θ(1) time, and that Balance therefore runs in Θ(1) time. Let us
now analyze Insert. Excluding the recursion, this function clearly runs in
Θ(1) time. At most one recursive call is made, and its second parameter
has height strictly less than the height of t; in the worst case, it is 1 less. If
h is the height of t, then the worst-case running time of Insert is given by

f (h) ∈ f (h − 1) + Θ(1)
CHAPTER 6. STORAGE/RETRIEVAL I: ORDERED KEYS 214

for h > 0. By Theorem 3.31, f (h) ∈ Θ(h). By Theorem 6.4, Insert

therefore runs in Θ(lg n) time, where n is the size of the data set. Clearly,
Put can be written to operate in Θ(lg n) time. We leave as an exercise the
design and analysis of logarithmic algorithms for Get and Remove.

6.3 Splay Trees

While a worst-case bound of Θ(lg n) for OrderedDictionary accesses is
good, bounding the worst case does not always result in the best perfor-
mance. For example, in many applications, the so-called “80-20” rule holds;
i.e., 80% of the accesses involve roughly 20% of the data items. This rule
then applies recursively, so that 64% of the accesses involve roughly 4% of
the data items. In order to get good performance in such an environment,
we would like to structure the data so that the most commonly-accessed
data items can be accessed more quickly.
One variation of a binary search tree that attempts to achieve this kind
of performance is a splay tree. Structurally, a splay tree is simply a binary
search tree. Operationally, however, it is self-adjusting — when it accesses
an item, it brings that item to the root of the tree via a series of rotations.
(The VisitInOrder operation is an exception to this rule, as it accesses all
of the data items.) As a result, the more frequently accessed items tend to
remain closer to the root.
No attempt is made to ensure any sort of balance in a splay tree. As a
result, the operations run in Θ(n) time in the worst case. However, when
a long path is traversed, the rotations have the effect of shortening it by
roughly half. Thus, an expensive operation improves future performance.
As a result, the amortized running times of Get, Put, and Remove are
all in O(lg n).
To get a rough idea of how rotations improve the data structure, suppose
we have a long zig-zag path from the root to some node b; i.e., by starting
from the root, then taking first the left child, then the right child, and
continuing to alternate, we eventually reach b. We could then bring b to
the root of the tree by a series of double rotations, each promoting b by two
levels. Now referring to Figure 6.12, notice that the distance between the
root and any descendant of d decreases by 1 for each rotation. The number
of rotations is half the distance from the root to d, so each descendant of
d ends up closer to the root by half the original distance between the root
and d.
Unfortunately, single rotations are not as effective in improving the struc-
CHAPTER 6. STORAGE/RETRIEVAL I: ORDERED KEYS 215

Figure 6.15 A zig-zig right rotation

c a

b D A b

a C B c

A B C D

ture. Notice that in Figure 6.11, nodes in subtree c do not get any closer
to the root as a result of the rotation. As a result, we need a new kind of
double rotation that can be applied when the node to be promoted is not a
“zig-zag” from its grandparent. So that we might distinguish between the
various double rotations, we will refer to the rotation of Figure 6.12 as a
zig-zag right, and to its mirror image as a zig-zag left. A zig-zig right is
shown in Figure 6.15. Note that by this rotation, the distance between the
root and any descendant of a is decreased by at least 1.
Our representation, interpretation, and structural invariant will be the
same as for BSTDictionary. The only differences will occur in the actual
implementations of the operations. In fact, the implementation of VisitIn-
Order will also be the same as for BSTDictionary.
Let us consider how we can implement a Find function. First, we observe
that no value needs to be returned, because if the key we are looking for
exists, we will bring it to the root of the tree. Hence, after invoking the
Find function, the Get operation only needs to look in the root to see if
the desired key is there. Second, we don’t want to bring a node representing
an empty subtree to the root. For this reason, we will need to verify that a
node is nonempty at some point before rotating it to the root. It therefore
seems reasonable to include as part of the precondition that the tree is
nonempty.
We therefore begin by comparing the given key k to the key at the root
of the given tree t. If the keys don’t match, we will need to look in the
appropriate child, after verifying that it is nonempty. However, we want to
CHAPTER 6. STORAGE/RETRIEVAL I: ORDERED KEYS 216

do a double rotation whenever possible, so rather than using a recursive call

at this point, we should go ahead and make another comparison. If we find
the key k, or if the appropriate grandchild is empty, we do a single rotation.
Otherwise, we recursively look for k in the appropriate grandchild and do a
double rotation. The algorithm is shown in Figure 6.16.

Example 6.6 Suppose we were to do a Find on 60 in the splay tree shown

in Figure 6.17(a). Because the length of the path to 60 is odd, we must
begin with a single rotation. Figure 6.17(b) shows the result of doing a
single rotate left at 53. We then proceed with double rotations to bring 60
to the root. In this case, only one rotation — a zig-zag left — is required.
The result is shown in Figure 6.17(c).
The insertion algorithm cannot use Find because it must insert a new
data item when an empty subtree is found. However, it can be patterned
after the Find algorithm. The main difference is that because a data item
is inserted into an empty tree, we will always rotate that node to the root.
We therefore do not need to restrict its use to nonempty trees. The details
are left as an exercise.
The deletion algorithm can, however, use Find. Suppose we want to
delete key k. We can use Find(k, elements) to move k to the root if it
is present. If the right child is empty, we can simply make the left child
the new root. Otherwise, we can use another internal function, Find-
Min(elements.RightChild()), to move the minimum key m in the right
child to the root of the right child. At this point, the right child has an
empty left child, because there are no keys with values between k and its
right child. The result is shown in Figure 6.18. We can therefore complete
the deletion by making A the left child of m and making m the root (see
Figure 6.18). The algorithm is given in Figure 6.19.
Let us now analyze the amortized running times of Get, Put, and Re-
move for SplayDictionary. It is not hard to see that all of the recursive
algorithms have constant running time, excluding recursive calls. Further-
more, each time a recursive call is made, a rotation is done. It is therefore
sufficient to analyze the total number of rotations. Each rotation, therefore,
will have an actual cost of 1.
In order to amortize the number of rotations, we need to find an appro-
priate potential function. Intuitively, an operation involving many rotations
should improve the overall balance of the tree. The potential function should
in some way measure this balance, decreasing as the balance increases. If
the tree is very unbalanced, as in Figure 6.9, many of the subtrees have a
CHAPTER 6. STORAGE/RETRIEVAL I: ORDERED KEYS 217

Figure 6.16 The Find internal function for the SplayDictionary imple-
mentation of OrderedDictionary.

Precondition: k is a Key and t is a reference to a BinaryTreeNode

representing a nonempty BST.
Postcondition: If a data item with key k is in t, then t is rearranged into
a BST with k at the root.
SplayDictionary.Find(k, t)
if k < t.Root().Key()
if t.LeftChild().Root() 6= nil
if k < t.LeftChild().Root().Key()
if t.LeftChild().LeftChild().Root() 6= nil
Find(k, t.LeftChild().LeftChild()); ZigZigRight(t)
else
SingleRotateRight(t)
else if k > t.LeftChild().Root().Key()
if t.LeftChild().RightChild().Root() 6= nil
Find(k, t.LeftChild().RightChild()); ZigZagRight(t)
else
SingleRotateRight(t)
else
SingleRotateRight(t)
else if k > t.Root().Key()
if t.RightChild().Root() 6= nil
if k < t.RightChild().Root().Key()
if t.RightChild().LeftChild().Root() 6= nil
Find(k, t.RightChild().LeftChild()); ZigZagLeft(t)
else
SingleRotateLeft(t)
else if k > t.RightChild().Root().Key()
if t.RightChild().RightChild().Root() 6= nil
Find(k, t.RightChild().RightChild()); ZigZigLeft(t)
else
SingleRotateLeft(t)
else
SingleRotateLeft(t)
CHAPTER 6. STORAGE/RETRIEVAL I: ORDERED KEYS 218

Figure 6.17 Example of doing a Find on 60 in a splay tree

50 50

47 63 47 63

43 53 70 43 60 70

34 51 60 65 76 34 53 65 76

51

(a) The original tree (b) A single rotate left is done

at 53

60

50 63

47 53 70

43 51 65 76

34

(c) A zig-zag left is done at 50

CHAPTER 6. STORAGE/RETRIEVAL I: ORDERED KEYS 219

Figure 6.18 The splay tree after the calls to Find and FindMin in Remove

A m

Figure 6.19 The Remove operation for the SplayDictionary implemen-

tation of OrderedDictionary

SplayDictionary.Remove(k)
if elements.Root() 6= nil
Find(k, elements)
if elements.Root().Key() = k
l ← elements.LeftChild()
r ← elements.RightChild()
if r.Root() = nil
elements = l
else
FindMin(r); r.SetLeftChild(l); elements ← r

— Internal Functions Follow —

Precondition: t is a BinaryTreeNode representing a nonempty BST.
Postcondition: Arranges t into a BST whose smallest key is at the root.
SplayDictionary.FindMin(t)
if t.LeftChild().Root() 6= nil
if t.LeftChild().LeftChild().Root() = nil
SingleRotateRight(t)
else
FindMin(t.LeftChild().LeftChild())
ZigZigRight(t)
CHAPTER 6. STORAGE/RETRIEVAL I: ORDERED KEYS 220

Figure 6.20 The subtrees that are modified by a zig-zag rotation

Tc
c Tb′
Ta
b
Ta′ Tc′
a D
a c
Tb
A b
A B C D
B C

comparatively large number of nodes, whereas in a balanced tree, most of

the subtrees have only a few nodes. It would therefore make sense to define
the potential function to depend on the number of nodes in each subtree.
An example of such a potential function is the sum of the sizes of all of
the subtrees. However, this potential function will not work. Consider what
happens when 0 is inserted into the tree in Figure 6.9. It is inserted to the
left of 1, then rotated to the root via a single rotate right. The original tree
therefore ends up as the right child of the result. The potential function
therefore increases by the number of nodes in the result. With an increase
this large, we cannot achieve a logarithmic amortized cost.
In order to scale back the growth of the potential function, let us try
applying the lg function to the size of each nonempty subtree. Specifically,
let |t| denote the number of nodes in a subtree t. We then define our potential
function Φ(T ) to be the sum of all lg |t| such that t is a nonempty subtree
of the entire tree T . In what follows, we will show that for each of the three
operations, the amortized cost with respect to Φ is in O(lg n).
Because most of the rotations will be double rotations, let us begin by
analyzing a zig-zag rotation. We will be focusing on the tree subtrees that
are changed by the zig-zag rotation, as shown in Figure 6.20; thus Ta , Tb ,
and Tc denote the subtrees rooted at a, b, and c, respectively, prior to
the rotation, and Ta′ , Tb′ , and Tc′ denote the subtrees rooted at these nodes
following the rotation. The amortized cost of the rotation will be the actual
CHAPTER 6. STORAGE/RETRIEVAL I: ORDERED KEYS 221

Figure 6.21 The subtrees that are modified by a zig-zig rotation

Tc Ta′

c a
Tb Tb′

Ta b D A b Tc′

a C B c

A B C D

cost (i.e., 1) plus the change in the potential function Φ. Noting that |Tb′ | =
|Tc |, we conclude that the change in Φ is

lg |Ta′ | + lg |Tc′ | − lg |Ta | − lg |Tb |. (6.4)

We need to simplify the above expression. It will be much easier to use

if we can bound it in terms of subtrees of the original tree. In particular, we
would like to get an expression involving only |Tb | and |Tc | so that, when the
amortized costs of all rotations in one operation are added together, perhaps
most terms will cancel out.
Let us therefore apply Theorem 5.7 (page 174) to lg |Ta′ | + lg |Tc′ | in (6.4).
We know that |Ta′ | + |Tc′ | ≤ |Tc |. By Theorem 5.7, we have lg |Ta′ | + lg |Tc′ | ≤
2 lg |Tc | − 2. Using the fact that |Ta | > |Tb | and adding in the actual cost,
we obtain the following upper bound on the amortized cost of a zig-zag
rotation:

2 lg |Tc | − 2 − 2 lg |Tb | + 1 = 2(lg |Tc | − lg |Tb |) − 1 (6.5)

Let us now analyze the amortized cost of a zig-zig rotation. Referring to

Figure 6.21 and adopting the same notational conventions as above, we see
that the change in Φ is

lg |Tb′ | + lg |Tc′ | − lg |Ta | − lg |Tb |. (6.6)

CHAPTER 6. STORAGE/RETRIEVAL I: ORDERED KEYS 222

Figure 6.22 The subtrees that are modified by a single rotation

Tb Ta′

b a
Ta Tb′
a C A b

A B B C

In order to get a tight bound for this expression in terms of lg |Tc | − lg |Ta |,
we need to be a bit more clever. We would again like to use Theorem 5.7.
Note that |Ta | + |Tc′ | ≤ |Tc |; however, lg |Ta | + lg |Tc′ | does not occur in (6.6).
Let us therefore both add and subtract lg |Ta | to (6.6). Adding in the actual
cost, applying Theorem 5.7, and simplifying, we obtain the following bound
on the amortized cost of a zig-zig rotation:
lg |Tb′ | + lg |Tc′ | + lg |Ta | − 2 lg |Ta | − lg |Tb | + 1
≤ lg |Tb′ | + 2 lg |Tc | − 2 − 2 lg |Ta | − lg |Tb | + 1
≤ 3 lg |Tc | − 3 lg |Ta | − 1
= 3(lg |Tc | − lg |Ta |) − 1. (6.7)
Finally, let us analyze the amortized cost of a single rotate. We refer to
Figure 6.22 for this analysis. Clearly, the amortized cost is bounded by
lg |Tb′ | − lg |Ta | + 1 ≤ lg |Tb | − lg |Ta | + 1. (6.8)
Because each operation will do at most two single rotations (recall that a
deletion can do a single rotation in both the Find and the FindMin), the
“+ 1” in this bound will not cause problems.
We can now analyze the amortized cost of a Find. We first combine
bounds (6.5), (6.7), and (6.8) into a single recurrence defining a function
f (k, t) bounding the amortized cost of Find(k, t). Suppose Find(k, t) makes
a recursive call on a subtree s and performs a double rotation. We can then
combine (6.5) and (6.7) to define:
f (k, t) = 3(lg |t| − lg |s|) + f (k, s).
CHAPTER 6. STORAGE/RETRIEVAL I: ORDERED KEYS 223

For the base of the recurrence, suppose that either no rotation or a single
rotate is done. Using (6.8), we can define

f (k, t) = 3(lg |t| − lg |s|) + 1,

where s is the child rotated upward or t if no rotation is done.

It is easily seen that the above recurrence telescopes; i.e., when the value
for f (k, s) is substituted into the value for f (k, t), the lg |s| terms cancel.
The entire recurrence therefore simplifies to

f (k, t) = 3(lg |t| − lg |s|) + 1

where s is the subtree whose root is rotated to the root of t. Clearly, f (k, t) ∈
O(lg n), where n is the number of nodes in t. The amortized cost of Find,
and hence of Get, is therefore in O(lg n).
The analysis of Put is identical to the analysis of Find, except that we
must also account for the change in Φ when the new node is added to the
tree. When the new node is added, prior to any subsequent rotations, it is
a leaf. Let s denote the empty subtree into which the new leaf is inserted.
The insertion causes each of the ancestors of s, including s itself, to increase
in size by 1. Let t be one of these ancestors other than the root, and let t′ be
the same subtree after the new node is inserted. Note that t′ has no more
nodes than does the parent of t. If we think of the insertion as replacing the
parent of t by t′ , then this replacement causes no increase in Φ. The only
node for which this argument does not apply is the root. Therefore, the
increase in Φ is no more than lg(n + 1), where n is the number of nodes in
the tree prior to the insertion. The entire amortized cost of Put is therefore
in O(lg n).
Finally, let us consider the Remove operation. The Find has an amor-
tized cost in O(lg n). Furthermore, the amortized analysis of Find also
applies to FindMin, so that it is also in O(lg n). Finally, it is easily seen
that the actual removal of the node does not increase Φ. The amortized cost
of Remove is therefore in O(lg n) as well.

6.4 Skip Lists

We conclude this chapter by returning to the idea of using an ordered linked
list to implement OrderedDictionary. Recall that the difficulty with
this idea is that items must be accessed sequentially, so that a binary search
cannot be used to find an item. A skip list overcomes this difficulty by using
CHAPTER 6. STORAGE/RETRIEVAL I: ORDERED KEYS 224

Figure 6.23 A skip list

−∞ 12 26 35 48 63 66 73 ∞

Note: Arrows to the same vertical line represent identical references.

additional references to skip over portions of the list (see Figure 6.23). Using
these additional references, a binary search can be approximated.
The main building block for a skip list is the data type SkipListNode,
which represents a data item, its key, a level n ≥ 1, and a sequence of n val-
ues, each of which is either a SkipListNode or empty. The representation
consists of three variables:

• data: a data item; and

• key: a Key;

• links[1..n]: an array of (possibly nil) SkipListNodes.

We interpret data as the represented data item, key as its associated key,
SizeOf(links) as the level of the SkipListNode, and links[i] as the ith
element of the sequence, where empty is represented by nil. We allow read
access to key and data. The complete implementation is shown in Figure
6.24.
We represent the OrderedDictionary with four variables:

• start: a non-nil SkipListNode;

• end: a non-nil SkipListNode;

• maxLevel: a Nat; and

• size: a Nat.
CHAPTER 6. STORAGE/RETRIEVAL I: ORDERED KEYS 225

Figure 6.24 The data type SkipListNode

Structural Invariant: true.

Precondition: x is a data item, k is a Key, and n is a non-zero Nat.
Postcondition: Constructs a SkipListNode containing data x with key
k and level n. All n elements of the sequence are empty.
SkipListNode(x, k, n)
data ← x; key ← k; links ← new Array[1..n]
for i ← 1 to n
links[i] ← nil

Precondition: true.
Postcondition: Returns the level.
SkipListNode.Level()
return SizeOf(links)

Precondition: i is a non-zero Nat no greater than the level.

Postcondition: Returns the ith element of the sequence, or nil if that
element is empty.
SkipListNode.Link(i)
return links[i]

Precondition: i is a non-zero Nat no greater than the level, and s is a

(possibly nil) reference to a SkipListNode.
Postcondition: Sets the ith element of the sequence to s, or to empty if s
is nil.
SkipListNode.SetLink(i, s : SkipListNode)
links[i] ← s
CHAPTER 6. STORAGE/RETRIEVAL I: ORDERED KEYS 226

We interpret the represented set to be the data items in the linked list
beginning with start and ending with end, using the variables links[1] to
obtain the next element in the list; the data items in start and end are
excluded from the set.
Our structural invariant is:
• Both start and end have a level of M ≥ maxLevel.

• start.key = minKey, which is the smallest possible key.

• end.key = maxKey, which is the largest possible key.

• There is a sequence of SkipListNodes with length size + 2 obtained

by starting with start and following the links[1] reference in each
SkipListNode until end is reached. We will refer to this sequence
as the level-1 sequence.

• For 1 < i ≤ M , there is a sequence obtained in the same way as

above, but using the links[i] variables instead of the links[1] variables.
We will refer to this sequence as the level-i sequence. The level-i
sequence is the subsequence of the level-(i − 1) sequence containing
those SkipListNodes having level i or greater.

• The keys in each sequence are strictly increasing in value.

• maxLevel is the maximum level of any SkipListNode in the above

sequences, excluding start and end. If start and end are the only
SkipListNodes in the sequences, maxLevel = 1.
In order to be able to approximate a binary search with this data struc-
ture, the level-i sequence should include roughly every second node from
the level-(i − 1) sequence, for 1 < i ≤ maxLevel. However, as is suggested
by Figure 6.23, we will not explicitly maintain this property. Instead, we
will use randomization to produce a structure that we expect, on average,
to approximate this property.
When we insert a new data item, we first determine the level of its
SkipListNode via a series of flips of a fair coin. As long as the outcome of
a coin flip is heads, we continue flipping. We stop flipping when the outcome
is tails. The level of the SkipListNode is the total number of flips. Because
we flip the coin at least once, every level will be at least 1. Because the coin
is fair, the probability of tails is 1/2; hence, we would expect about half of
the SkipListNodes to have level 1. The probability of flipping heads then
tails is (1/2)2 = 1/4, so we would expect about 1/4 of the SkipListNodes
CHAPTER 6. STORAGE/RETRIEVAL I: ORDERED KEYS 227

to have level 2. In general, the probability of flipping i − 1 heads followed by

1 tails is 2−i . We would therefore expect the fraction of nodes having level
i to be about 2−i . Because the coin is fair, these levels should be randomly
distributed over the level-1 sequence.
Suppose a given SkipListNode has level k. In order to insert it into the
skip list, we need to insert it into its proper location in the level-i sequence
for each i, 1 ≤ i ≤ k. For the purpose of finding these insertion points, we
will design a function Find(k, l), which will return an array of references
such that at index i, 1 ≤ i ≤ l, is the reference to the SkipListNode of
level at least i having the largest key strictly less than k. We will then be
able to use this function not only in the Put operation, but also in the Get
and Remove operations. The Put operation is shown in Figure 6.25.
The partial implementation shown in Figure 6.25 does not explicitly
handle the case in which the level of a new node exceeds the level of start
and end. In what follows, we will assume that these arrays are expanded as
needed using the expandable array design pattern. Later, we will argue that
for all practical purposes, a fixed-sized array can be used. In the meantime,
we note that the time needed to expand the array is proportional to the
number of iterations of the while loop in Put, so we need not amortize this
cost.
We will devote the remainder of this section to analyzing the expected
running time of Put. This analysis is rather involved, but it uses some
important new tools for analyzing expected running times. Furthermore,
much of this analysis can be applied directly to the analyses of Get and
Remove; we therefore leave these analyses as exercises. In view of the
sometimes counter-intuitive nature of expected values, we will proceed with
care.
We will partition the algorithm into five parts:

• the while loop;

• the call to Find;

• the construction of a new SkipListNode;

• the for loop; and

• the remainder of the algorithm for the case in which x 6= nil.

At this point, let us observe that a worst-case input must have x 6= nil and k
as a new key, not already in the set. On any such input, the overall running
time is the sum of the running times of the above five parts. By the linearity
CHAPTER 6. STORAGE/RETRIEVAL I: ORDERED KEYS 228

Figure 6.25 SkipListDictionary implementation (partial) of Ordered-

Dictionary

SkipListDictionary()
start ← new SkipListNode(nil, minKey, 100)
end ← new SkipListNode(nil, maxKey, 100)
for i ← 1 to 100
start.SetLink(i, end)
size ← 0; maxLevel ← 1

SkipListDictionary.Put(x, k)
if x = nil
error
else
l←1
while FlipCoin() = heads
l ←l+1
p ← Find(k, l)
if p[1].Link(1).Key() 6= k
maxLevel ← Max(maxLevel, l); q ← new SkipListNode(x, k, l)
for i ← 1 to l
q.SetLink(i, p[i].Link(i)); p[i].SetLink(i, q)

— Internal Functions Follow —

Precondition: k is a Key and l is a non-zero Nat, no larger than the
levels of start or end.
Postcondition: Returns an array A[1..l] such that A[i] refers to the
SkipListNode of level i or larger having the largest key strictly less than
k.
SkipListDictionary.Find(k, l)
A ← new Array[1..l]; p ← start
for i ← Max(maxLevel, l) to 1 by −1
while p.Link(i).Key() < k
p ← p.Link(i)
if i ≤ l
A[i] ← p
return A
CHAPTER 6. STORAGE/RETRIEVAL I: ORDERED KEYS 229

of expectation, the expected running time of the algorithm on a worst-case

input is therefore the sum of the expected running times of these five parts
on a worst-case input.
We begin by analyzing the while loop. We define the discrete probability
space Seq to be the set of all finite sequences of flips containing zero or more
heads, followed by exactly one tails. Note that for each positive integer i,
there is exactly one sequence in Seq with length i; hence, Seq is countable.
As we have already argued, the probability of achieving a sequence of length
i is 2−i . In order to conclude that Seq is a discrete probability space, we
must show that
X∞
2−i = 1.
i=1
This fact follows from the following theorem, using c = 2 and a = −1.

Theorem 6.7 For any real numbers a and c such that c > 1,
∞
X ca+1
ca−i = .
c−1
i=0

Proof:
∞
X n
X
ca−i = lim ca−i
n→∞
i=0 i=0
n
X
= ca lim (1/c)i
n→∞
i=0
(1/c)n+1 − 1
= ca lim 1 from (2.2)
c −1
n→∞

c − (1/c)n
= ca lim
n→∞ c−1
ca+1
=
c−1
because 1/c < 1.

We now define the discrete random variable len over Seq such that len(e)
is the length of the sequence of flips. Note that E[len] gives us the expected
number of times the while loop condition is tested, as well as the expected
final value of l. As a result, it also gives us the expected number of iterations
of the for loop, provided k is not already in the set.
CHAPTER 6. STORAGE/RETRIEVAL I: ORDERED KEYS 230

Because len(e) is always a natural number, we can apply Theorem 5.5

(page 172) to obtain E[len]. The probability that a sequence has length at
least i is the probability that i − 1 flips all result in heads, or 21−i . Thus,
∞
X
E[len] = 21−i
i=1
=2
from Theorem 6.7. We can therefore expect the while loop in Put to iterate
once, yielding an expected value of 2 for l, on average. Hence, the for loop,
if it executes, iterates twice on average. The expected running times of both
loops are therefore in Θ(1) for a worst-case input.
In order to determine the expected running time of the SkipListNode Corrected
4/7/11.
constructor, we need to analyze it again, but this time doing an expected-
case analysis using len as its third parameter. Using the same analysis as
we did for the for loop in Put, we see that its expected running time is in
Θ(1).
In order to complete the expected-case analysis of Put, we need to
analyze Find. We will begin by defining an appropriate discrete probability
space. Let Seqn be the set of all n-tuples of elementary events from Seq; i.e.,
each elementary event in Seqn is an n-tuple he1 , . . . , en i such that each ei is
a sequence of coin flips containing zero or more heads, followed by exactly
one tails. Such an n-tuple describes the “shape” of a skip list by recording,
for each of the n data elements, the sequence of coin flips which generated
the level of the SkipListNode containing it.
In order to show that Seqn is countable, we can label each n-tuple
he1 , . . . , en i ∈ Seqn with the natural number
len(e1 ) len(e2 ) n)
p1 p2 · · · plen(e
n ,
where pi is the ith prime. Because each elementary event in Seq is uniquely
identified by its length, and because each positive integer has a unique prime
factorization, each tuple has a unique label; hence, Seqn is countable.
We need to define the probabilities of elements in Seqn . In order to
do this properly, we need to extend the definition of independence given
in Section 5.5 to more than two events. We say that a set S of events is
pairwise independent if for every pair of events e1 , e2 ∈ S, e1 and e2 are
independent. If for every subset T ⊆ S containing at least two events, This -notation
Q

! denotes the
\ Y product of the
P e = P (e), probabilities
P (e) for all
e∈T e∈T events e in T .
CHAPTER 6. STORAGE/RETRIEVAL I: ORDERED KEYS 231

then we say the events in S are mutually independent. We leave as an exer- In other words,
the probability
cise to show that pairwise independence does not necessarily imply mutual that all of the
independence, even for 3-element sets of events. events in T occur
Returning to Seqn , let lenij denote the event that component i has length is the product of
the probabilities
j, for 1 ≤ i ≤ n, j > 1. In any set {leni1 j1 , . . . , lenim jm } with 2 ≤ m ≤ n for each of these
and all the ik s different, the events should be mutually independent. Fur- events.
thermore, in order to be consistent with Seq, we want P (lenij ) = 2−j for
1 ≤ i ≤ n and j > 1. We can satisfy these constraints by setting the proba-
bility of elementary event he1 , . . . , en i to the product of the probabilities in
Seq of e1 , . . . , en ; i.e.,
n
Y
P (he1 , . . . , en i) = 2−len(ei ) .
i=1

It can be shown by a straightforward induction on n that the sum of the

probabilities of the elementary events in Seqn is 1. Seqn is therefore a
discrete probability space.
We now need to determine what comprises a worst-case input to Find.
As we suggested earlier, the components of an elementary event in Seqn
correspond to the n data elements in a skip list. The length of a component
gives the level of the skip list element. Apart from the number of elements in
the structure, the shape of a skip list is determined completely at random,
independent of the data elements inserted or the order in which they are
inserted. Specifically, the keys in the data elements determine their order,
but their levels are determined solely by coin flips. Thus, in order to de-
termine a worst-case input, we needn’t worry about how the structure was
constructed — we need only concern ourselves with the parameters to Find.
Both of these values can affect the running time.
In order to determine the worst-case input, we need to consider the
behavior of the while loop. For a given value of i, the while loop iterates
once for each key at level i that is

• less than k; and

• greater than the largest key less than k at any level j > i (or −∞ if
there is no such key).

It is easily seen that at any level i, the expected number of iterations is

maximized when the number of keys less than k is maximized, because the
levels of these keys are determined randomly. The worst-case input therefore
has k greater than any key in the data set.
CHAPTER 6. STORAGE/RETRIEVAL I: ORDERED KEYS 232

We therefore define tail i (e), where e = he1 , . . . , en i ∈ Seqn , to be the

largest natural number j such that some suffix of e contains j components
with length exactly i and no components longer than i. Thus, if e contains
at least one component strictly longer than i, then tail i (e) is the number
of components with length i that follow the last component strictly longer
than i. Otherwise, tail i (e) is simply the number of components with length
i.

Example 6.8 Let e represent the skip list shown in Figure 6.23 on page Corrected
4/7/11.
224. Then
• tail 1 (e) = 0 because there are no level-1 nodes following the last node
with level greater than 1;

• tail 2 (e) = 2 because there are 2 level-2 nodes following the last node
with level greater than 2; and

• tail 3 (e) = 1 because there is 1 level-3 node, and there are no nodes
with level greater than 3.
Suppose e describes some skip list with n elements, and suppose this
skip list’s Find function is called with a key larger than any in the list. The
running time of Find is then proportional to the number of times the while
loop condition is tested. On iteration i of the for loop, the while loop will
iterate exactly tail i (e) times, but will be tested tail i (e) + 1 times, including
the test that causes the loop to terminate. The expected running time of
Find on a worst-case input is therefore proportional to:
 
max(maxLevel,l)
X
E (tail i + 1)
i=1
  
max(maxLevel,l)
X
= E  tail i  + max(maxLevel, l)
i=1
 
max(maxLevel,l)
X
=E tail i  + E[max(maxLevel, l)]. (6.9)
i=1

Let us first consider the first term in (6.9). It is tempting to apply lin-
earity of expectation to this term; however, note that maxLevel is a random
variable, as its value depends on the levels of the nodes in the skip list. The-
orem 5.9 therefore does not apply to this term. In particular, note that for
CHAPTER 6. STORAGE/RETRIEVAL I: ORDERED KEYS 233

any positive n and i, there is a non-zero probability that there is at least one
node at level i; hence, there is a non-zero probability that tail i is positive.
The proper way to handle this kind of a summation, therefore, is to
convert it to an infinite sum. The term inside the summation should be
equal to tail i when i ≤ max(maxLevel, l), but should be 0 for all larger i. In
this case, it is easy to derive such a term, as tail i = 0 when i > maxLevel.
We therefore have:
 
max(maxLevel,l)
"∞ #
X X
E tail i  = E tail i
i=1 i=1
∞
X
= E[tail i ]. (6.10)
i=1

By Theorem 5.5,
∞
X
E[tail i ] = P (tail i ≥ j).
j=1

Suppose that there are at least j components with length at least i. In order
for tail i ≥ j, the ith coin flip in each of the last j of these components must
be tails. The probability that j independent coin flips are all tails is 2−j .
However, this is not the probability that tail i ≥ j, but rather the conditional
probability given that there are at least j components with length at least
i. Let numi denote the number of components whose length is at least i.
We then have
P (tail i ≥ j | numi ≥ j) = 2−j .
Fortunately, this conditional probability is closely related to P (tail i ≥ j).
Specifically, in order for tail i ≥ j, it must be the case that numi ≥ j. Thus,
the event tail i ≥ j is a subset of the event numi ≥ j. Therefore, from (5.2)
we have

P (tail i ≥ j) = P ((tail i ≥ j) ∩ (numi ≥ j))

= P (numi ≥ j)P (tail i ≥ j | numi ≥ j)
= P (numi ≥ j)2−j .

Unfortunately, computing the exact value of P (numi ≥ j) is rather dif-

ficult. We will therefore content ourselves with observing that because it is
CHAPTER 6. STORAGE/RETRIEVAL I: ORDERED KEYS 234

a probability, it can be no more than 1. We therefore have,

∞
X
E[tail i ] = P (tail i ≥ j)
j=1
X∞
= P (numi ≥ j)2−j
j=1
X∞
≤ 2−j
j=1

=1

from Theorem 6.7.

This bound seems quite good, perhaps even surprisingly so. It tells us
that on any iteration of the for loop, we can expect the while loop to
iterate no more than once, on average. Still, this bound does not give a
finite bound for (6.10). However, we have already observed that for any
e ∈ Seqn , tail i (e) will be 0 for all but finitely many i. This follows because
there are only finitely many nonempty levels. Consequently, we might want
to use the fact that tail i (e) ≤ numi (e); hence, E[tail i ] ≤ E[numi ].
While this bound would yield a finite bound for (6.10), it unfortunately
is still too loose, as num1 (e) = n for every e ∈ Seqn . We would like to derive
a logarithmic upper bound, if possible. However, we can use a combination
of the two bounds. In particular, the bound of 1 seems to be a good up-
per bound as long as it is less than E[numi ]. Once i is large enough that
E[numi ] ≤ 1, E[numi ] would be a better bound. If we can determine the
smallest value of i such that E[numi ] ≤ 1, we should be able to break the
infinite sum into two sums and derive tight bounds for each of them.
In order to analyze E[numi ], we observe that for e ∈ Seqn , numi (e) is a
count of the number of components whose lengths are at least i. Further-
more, we can express the fact that a component has a length of at least i
as an event in Seq. The standard technique for counting events is to use
an indicator random variable. Specifically, consider the event in Seq that
len ≥ i; i.e., this event is the set of sequences of coin flips consisting of at
least i − 1 heads, followed by exactly one tails. The indicator for this event
is then defined to be
(
1 if len(ej ) ≥ i
I(len ≥ i)(ej ) =
0 otherwise.
CHAPTER 6. STORAGE/RETRIEVAL I: ORDERED KEYS 235

We can then express numi as follows:

n
X
numi (he1 , . . . , en i) = I(len ≥ i)(ej ).
j=1

The utility of indicator random variables is shown by the following the-

orem, whose proof follows immediately from Theorem 5.5.

Theorem 6.9 Let e be any event in a discrete probability space. Then Corrected
4/7/11.
E[I(e)] = P (e).
Applying the above theorem, we obtain
 
Xn
E[numi ] = E  I(len ≥ i)
j=1

= E[nI(len ≥ i)]
= nE[I(len ≥ i)]
= nP (len ≥ i)
= n21−i .
Clearly, E[numi ] > 1 iff i < 1+lg n. This suggests that maxLevel should
typically be about lg n (however, this is not a proof of the expected value
of maxLevel). Because we already know that the while loop is expected
to iterate no more than once for each level, this suggests that the overall
running time is logarithmic in n (assuming l is sufficiently small). While we
don’t have quite enough yet to show this, we can now show a logarithmic
bound on the first term in (6.9):
∞
X ∞
X
E[tail i ] ≤ (min(1, E[numi ]))
i=1 i=1
⌈lg n⌉ ∞
X X
= 1+ n21−i
i=1 i=⌈lg n⌉+1
∞
X
−⌈lg n⌉−i
= ⌈lg n⌉ + n 2
i=0
1−⌈lg n⌉
= ⌈lg n⌉ + n2
≤ ⌈lg n⌉ + n21−lg n
= ⌈lg n⌉ + 2. (6.11)
CHAPTER 6. STORAGE/RETRIEVAL I: ORDERED KEYS 236

In order to complete the analysis of Find, we must consider the second

term in (6.9), namely, E[max(maxLevel, l)]. It would be nice if we could use
a property like the linearity of expectation to conclude that this value is equal
to max(E[maxLevel], E[l]); however, such a property does not necessarily
hold (see Exercise 6.16). On the other hand, because maxLevel and l are
nonnegative, we can use the fact that max(maxLevel, l) ≤ maxLevel + l.
Therefore,

E[max(maxLevel, l)] ≤ E[maxLevel + l]

= E[maxLevel] + E[l]. (6.12)

For the case in which Find is called from Put, we know that E[l] = 2. Corrected
4/7/11.
We therefore need to evaluate E[maxLevel]. Note that maxLevel is the
number of nonempty levels. We can therefore use another indicator random
variable to express maxLevel — specifically,
∞
X
maxLevel = I(numi > 0).
i=1

We therefore have
∞
" #
X
E[maxLevel] = E I(numi > 0)
i=1
∞
X
= E[I(numi > 0)]. (6.13)
i=1

Clearly, I(numi > 0)(e) ≤ 1 for all e ∈ Seqn , so that E[I(numi > 0)] ≤
1. Furthermore, I(numi > 0)(e) ≤ numi (e), so that E[I(numi > 0)] ≤
E[numi ]. We therefore have E[I(numi > 0)] ≤ min(1, E[numi ]), which
is the same upper bound we showed for E[tail i ]. Therefore, following the
derivation of (6.11), we have

E[maxLevel] ≤ ⌈lg n⌉ + 2. (6.14)

Now combining (6.9), (6.10), (6.11), (6.12), and (6.14), it follows that
the expected number of tests of the while loop condition is no more than

2(⌈lg n⌉ + 2) + 2 ∈ O(lg n)

for a worst-case input when Find is called by Put. The expected running Corrected
4/7/11.
time of Find in this context is therefore in O(lg n).
CHAPTER 6. STORAGE/RETRIEVAL I: ORDERED KEYS 237

A matching lower bound for the expected running time of Find can also
be shown — the details are outlined in Exercise 6.18. We can therefore
conclude that the expected running time of Find when called from Put on
a worst-case input is in Θ(lg n).
We can now complete the analysis of Put. We have shown that the
expected running times for both loops and the constructor for SkipList-
Node are all in Θ(1). The expected running time of Find(k, l) is in Θ(lg n). Corrected
4/7/11.
The remainder of the algorithm clearly runs in Θ(1) time. The total time
is therefore expected to be in Θ(lg n) for a worst-case input. We leave as
exercises to design Get and Remove to run in Θ(lg n) expected time, as
well.
Earlier, we suggested that for all practical purposes, fixed-sized arrays
could be used for both start.elements and end.elements. We can now justify
that claim by observing that

P (numi > 0) = E[I(numi > 0)]

≤ E[numi ]
= n21−i .

Thus, the probability that some element has a level strictly greater than 100
is at most n2−100 . Because 2−20 < 10−6 , this means that for n ≤ 280 ≈ 1024 ,
the probability that a level higher than 100 is reached is less than one in a
million. Such a small probability of error can safely be considered negligible.

6.5 Summary
A summary of the running times of the operations for the various imple-
mentations of OrderedDictionary is given in Figure 6.26. Θ(lg n)-time
implementations of the Get, Put, and Remove operations for the Or-
deredDictionary interface can be achieved in three ways:
• A balanced binary search tree, such as an AVL tree, guarantees Θ(lg n)
performance in the worst case.

• A splay tree is a binary search tree that guarantees O(lg n) amortized

performance by rotating the items accessed to the root. This has an
additional benefit of leaving frequently accessed items near the root,
so that they are accessed more quickly.

• A skip list uses randomization to achieve Θ(lg n) expected performance

for worst-case inputs.
CHAPTER 6. STORAGE/RETRIEVAL I: ORDERED KEYS 238

Figure 6.26 Running times of the OrderedDictionary operations for

various implementations

Get Put Remove

BSTDictionary Θ(n) Θ(n) Θ(n)
AVLDictionary Θ(lg n) Θ(lg n) Θ(lg n)
SplayDictionary O(lg n) O(lg n) O(lg n)
amortized amortized amortized
SkipListDictionary Θ(lg n) Θ(lg n) Θ(lg n)
expected expected expected

Notes:

• n is the number of elements in the dictionary.

• The constructor and the Size operation each run in Θ(1) worst-case
time for each implementation.

• The VisitInOrder operation runs in Θ(n) worst-case time for each

implementation, assuming that the Visit operation for the given Vis-
itor runs in Θ(1) time.

• Unless otherwise noted, all running times are worst-case.

Section 6.4 introduced the use of indicator random variables for ana-
lyzing randomized algorithms. The application of this technique involves
converting the expected value of a random variable to the expected values
of indicator random variables and ultimately to probabilities. Theorems 5.5,
5.9, and 6.9 are useful in performing this conversion. The probabilities are Corrected
4/7/11.
then computed using the probabilities of the elementary events and the laws
of probability theory. Because we are only interested in asymptotic bounds,
probabilities which are difficult to compute exactly can often be bounded
by probabilities that are easier to compute.

6.6 Exercises
Exercise 6.1 Prove the correctness of BSTDictionary.TraverseInOr-
der, shown in Figure 6.7.
CHAPTER 6. STORAGE/RETRIEVAL I: ORDERED KEYS 239

Exercise 6.2 Draw the result of inserting the following keys in the order
given into an initially empty binary search tree:

34, 65, 75, 54, 19, 45, 11, 23, 90, 15

Exercise 6.3 Draw the result of deleting each of the following keys from
the tree shown in Figure 6.10, assuming that it is an ordinary binary search
tree. The deletions are not cumulative; i.e., each deletion operates on the
original tree.

a. 55

b. 74

c. 34

Exercise 6.4 Repeat Exercise 6.2 for an AVL tree.

Exercise 6.5 Repeat Exercise 6.3 assuming the tree is an AVL tree.

Exercise 6.6 Repeat Exercise 6.2 for a splay tree.

Exercise 6.7 Repeat Exercise 6.3 assuming the tree is a splay tree.

Exercise 6.8 Complete the implementation of AVLDictionary, shown

in Figure 6.13, so that Get, Put, and Remove run in Θ(lg n) time in
the worst case. Prove the correctness and running time of the resulting
implementation.

Exercise 6.9 The depth of a node in a tree is its distance from the root;
specifically the root has depth 0 and the depth of any other node is 1 plus
the depth of its parent. Prove by induction on the height h of any AVL tree
that every leaf has depth at least h/2.

* Exercise 6.10 Prove that when a node is inserted into an AVL tree, at
most one rotation is performed.

** Exercise 6.11 Prove that if 2m − 1 keys are inserted into an AVL tree
in increasing order, the result is a perfectly balanced tree. [Hint: You will
need to describe the shape of the tree after n insertions for arbitrary n, and
prove this by induction on n.]
CHAPTER 6. STORAGE/RETRIEVAL I: ORDERED KEYS 240

Exercise 6.12 A red-black tree is a binary search tree whose nodes are
colored either red or black such that
• if a node is red, then the roots of its nonempty children are black; and

• from any given node, every path to any empty subtree has the same
number of black nodes.
We call the number of black nodes on a path from a node to an empty
subtree to be the black-height of that node. In calculating the black-height
of a node, we consider that the node itself is on the path to the empty
subtree.
a. Prove by induction on the height of a red-black tree that if the black-
height of the root is b, then the tree has at least 2b − 1 black nodes.

b. Prove that if a red-black tree has height h, then it has at least 2h/2 − 1
nodes.

c. Prove that if a red-black tree has n nodes, then its height is at most
2 lg(n + 1).

Exercise 6.13 Give a splay-tree implementation of Put based on Splay-

Dictionary.Find, shown in Figure 6.16. You do not need to include algo-
rithms for the rotations. Prove its correctness, assuming the rotations are
correct.

Exercise 6.14 Exercise removed, 4/7/11.

Exercise 6.15 Prove by induction on n that the sum of the probabilities

of the elementary events in Seqn is 1.

Exercise 6.16 Let S = {heads, tails} be the discrete probability space in

which P (heads) = P (tails) = 1/2.
a. Using the definition of expected value, compute

E[max(I(heads), I(tails))].

b. Using Theorem 6.9, compute Corrected

4/7/11.
max(E[I(heads)], E[I(tails)]).

Your answer should be different from your answer in part a.

CHAPTER 6. STORAGE/RETRIEVAL I: ORDERED KEYS 241

Exercise 6.17 Prove that if f and g are discrete random variables in a

discrete probability space, then

E[max(f, g)] ≥ max(E[f ], E[g]).

* Exercise 6.18 The goal of this exercise is to show a lower bound on the
expected running time of SkipListDictionary.Find. Corrected
4/7/11.
a. Prove that P (numi > 0) = 1 − (1 − 21−i )n . [Hint: First compute
P (numi = 0).]

b. Prove the binomial theorem, namely, for any real a, b, and natural
number n,
n
n
X n n−j j
(a + b) = a b , (6.15)
j
j=0

where
n n!
=
j j!(n − j)!
are the binomial coefficients for 0 ≤ j ≤ n. [Hint: Use induction on
n.]

c. Using the results of parts a and b, prove that for i ≤ lg n + 1,

P (numi > 0) > 1/2.

d. Using the result of part c, Exercise 6.17, and (6.13), prove that

E[max(maxLevel, l)] ∈ Ω(lg n),

and hence, the expected running time of Find is in Ω(lg n). Corrected
4/7/11.
Exercise 6.19 Give algorithms for SkipListDictionary.Get and Skip-
ListDictionary.Remove. Prove that they meet their specifications and
run in expected Θ(lg n) time for worst-case input. Note that in both cases, Corrected
4/7/11.
you will need to modify the analysis of SkipListDictionary.Find to use
the appropriate value for E[l]. You may use the result of Exercise 6.18 for
the lower bounds.

Exercise 6.20 Suppose we define a discrete probability space consisting of

all ordered pairs of flips of a fair coin. This probability space contains four
elementary events, each having probability 1/4. We define the following
three events:
CHAPTER 6. STORAGE/RETRIEVAL I: ORDERED KEYS 242

• e1 : the first flip is heads;

• e2 : the second flip is heads; and

• e3 : the two flips are different.

Show that the three events are pairwise independent, but not mutually in-
dependent.

* Exercise 6.21 Let len be as defined in Section 6.4. For each of the
following, either find the expected value or show that it diverges (i.e., that
it is infinite).

a. E[2len ].
√
b. E[ 2len ].

Exercise 6.22 Let A[1..n] be a random permutation of the positive integers

less than or equal to n, such that each permutation is equally likely. Recall
from Exercise 3.28 (page 103) that an inversion is a pair of indices 1 ≤ i <
j ≤ n such that A[i] > A[j]. Determine the expected number of inversions
in A. [Hint: Use an indicator random variable for the event that (i, j) is
an inversion.]

Exercise 6.23 As in the above exercise, let A[1..n] be a random permuta-

tion of the positive integers less than or equal to n, such that each permu-
tation is equally likely. What is the expected number of indices i such that
A[i] = i?

6.7 Chapter Notes

AVL trees, which comprise the first balanced binary search tree scheme, were
introduced by Adel’son-Vel’skiı̆ and Landis [1]. Splay trees were introduced
by Sleator and Tarjan [99]. Red-black trees, mentioned in Exercise 6.12,
were introduced by Bayer [8] (see also Gubias and Sedgewick [57]). Balance
in red-black trees is maintained using the same rotations as for splay trees.
As a result, keys can be accessed in Θ(lg n) time in the worst case. Because
heights don’t need to be calculated, they tend to perform better than AVL
trees and are widely used in practice. A somewhat simpler version of red-
black trees, known as AA-trees, was introduced by Andersson [5].
CHAPTER 6. STORAGE/RETRIEVAL I: ORDERED KEYS 243

All of the above trees can be manipulated by the tree viewer on this
textbook’s web site. The implementations of these trees within this package
are all immutable.
Another important balanced search tree scheme is the B-tree, introduced
by Bayer and McCreight [9]. A B-tree is a data structure designed for
accessing keyed data from an external storage device such as a disk drive.
B-trees therefore have high branching factor in order to minimize the number
of disk accesses needed.
Skip lists were introduced by Pugh [93].
Chapter 7

Storage/Retrieval II:
Unordered Keys

In the last chapter, we considered the problem of storage and retrieval, as-
suming that we also need to be able to access keys in a predefined order.
In this chapter, we drop this assumption; i.e., we will be considering imple-
mentations of Dictionary (see Figure 6.2, p. 196) rather than Ordered-
Dictionary. The structures we defined in the last chapter all utilized the
ordering on the keys to guide the searches. Hence, it might seem that there
is nothing to be gained by neglecting to keep the keys in order. However,
we will see that disorder can actually be more beneficial when it comes to
locating keys quickly.

7.1 Arrays with Virtual Initialization

A simple implementation of Dictionary is to store all of the elements in
an array indexed by keys. Though this approach is simple, it has several
difficulties. The first difficulty is in using a key as an array index. For
example, if our keys are strings, we must somehow be able to interpret them
as natural numbers. Another difficulty is that we may have no fixed bound
on the size of our keys. In this case, we would not know how large an array
to construct. An expandable array would not yield a satisfactory solution
because however we determine the size of the array, the next key can be so
large that a new array must be constructed. Thus, we would have to expand
the array each time a new key is inserted. Such an approach is clearly too
expensive.
In spite of these difficulties, there is still a theoretically interesting ap-

244
CHAPTER 7. STORAGE/RETRIEVAL II: UNORDERED KEYS 245

proach using keys as array indices, provided we are willing to make some
assumptions. First, we assume that each key is a natural number (or equiv-
alently, each key can be treated as a natural number). Second, we assume
that there is a known upper bound on the values of all of the keys. Even
with these assumptions, it can still be the case that the range of the keys
is much larger than the number of keys. For example, suppose our data
set consists of 5,000 items keyed by 9-digit natural numbers (e.g., Social
Security Numbers). An array of 1 billion elements is required to store these
5,000 items. Initializing such an array would be very expensive.
Note, however, that once an array is initialized, storage and retrieval can
both be done in Θ(1) time in the worst case. What we need is a technique for
initializing an array in Θ(1) time while maintaining constant-time accesses
to elements. We will now present such a technique, known as virtual initial-
ization. This technique involves keeping track of which array elements have
been initialized in a way that facilitates making this determination quickly.
We assume that the environment provides a facility for allocating an array
in Θ(1) time without initializing its locations. We will call the resulting data
structure a VArray.
In addition to an array elements[0..n − 1] to store the data, we also
need an array used[0..n − 1] of Nats to keep track of which locations of
elements are used to store data. We use a Nat num to keep track of how
many locations of elements store data items. Thus, used[0..num − 1] will
be indices at which data items are stored in elements. Finally, in order
to facilitate a quick determination of whether elements[i] contains a data
element, we use a third array loc[0..n − 1] such that loc[i] stores the index in
used at which i is stored, if indeed i is in used[0..num − 1]. The structural
invariant is that 0 ≤ num ≤ n, and for 0 ≤ i < num, loc[used[i]] = i. We
interpret elements[i] as giving the data item at location i if 0 ≤ loc[i] < num
and used[loc[i]] = i; otherwise, we interpret the value stored at location i as
nil.
For example, Figure 7.1 shows a VArray with 10 locations, storing 35
at location 4, 17 at location 7, and nil at all other locations. Note that for
i = 4 or i = 7, 0 ≤ loc[i] < num and used[loc[i]] = i. For other values of i,
it is possible that loc[i] stores a natural number less than num; however, if
this is the case, then used[loc[i]] is either 4 or 7, so that used[loc[i]] 6= i.
To initialize all locations of the VArray to nil, we simply set num to 0.
In this way, there is no possible value of loc[i] such that 0 ≤ loc[i] < num,
so we interpret all locations as being nil. To retrieve the value at location
i, we first determine whether 0 ≤ loc[i] < num and used[loc[i]] = i. Note,
however, that loc[i] may not yet have been initialized, so that it may not
CHAPTER 7. STORAGE/RETRIEVAL II: UNORDERED KEYS 246

Figure 7.1 An example of a VArray

0 1 2 3 4 5 6 7 8 9

elements: ? ? ? ? 35 ? ? 17 ? ?

used: 7 4 ? ? ? ? ? ? ? ?

num = 2

loc: ? ? ? ? 1 ? ? 0 ? ?

even refer to a Number. Therefore, we must first verify that it is a Nat. If

all these tests are passed, we return elements[i]; otherwise, we return nil.
To store x at location i of the VArray, we must first determine as
above whether elements[i] is currently being used to store a data item —
i.e., whether 0 ≤ loc[i] < num and used[loc[i]] = i. If so, we can simply store
x in elements[i]. Otherwise, we must also update the other representation
variables to reflect the fact that we are using location i to store a data
item. In particular, we must store i in used[num], store num in loc[i], and
increment num. As a result, we interpret elements[i] as storing x, and the
structural invariant is maintained. The entire implementation of VArray
is shown in Figure 7.2.
It is easily seen that the constructor and all operations of VArray
operate in Θ(1) time. However, it uses Θ(n) space, where n is the size of
the range of indices, not the number of elements stored. Thus, if a VArray
is used to implement a Dictionary, its space usage will be proportional
to the number of possible keys. This number may be much larger than the
number of keys actually stored in the Dictionary.
In the remainder of this chapter, we will examine techniques for improv-
ing the space usage while maintaining fast storage and retrieval. Though
we cannot guarantee Θ(1) worst-case access time when we reduce the space
usage, we can achieve amortized expected access time proportional to the
CHAPTER 7. STORAGE/RETRIEVAL II: UNORDERED KEYS 247

Figure 7.2 Implementation of VArray

Structural Invariant: 0 ≤ num ≤ n, and for 0 ≤ i < num,

loc[used[i]] = i.
Precondition: n is a Nat.
Postcondition: Constructs a VArray with locations 0, . . . , n − 1, all
initialized to nil.
VArray(n)
elements ← new Array[0..n − 1]; used ← new Array[0..n − 1]
loc ← new Array[0..n − 1]; num ← 0

Precondition: true.
Postcondition: Sets all locations to nil.
VArray.Clear()
num ← 0

Precondition: i is a Nat less than the number of locations.

Postcondition: Returns the value stored at location i.
VArray.Get(i)
if loc[i] isType Nat and loc[i] < num and used[loc[i]] = i
return elements[i]
else
return nil

Precondition: i is a Nat less than the number of locations.

Postcondition: Sets the value stored at location i to x.
VArray.Put(x, i)
if not IsNat(loc[i]) or loc[i] ≥ num or used[loc[i]] 6= i
loc[i] ← num; used[num] ← i; num ← num + 1
elements[i] ← x
CHAPTER 7. STORAGE/RETRIEVAL II: UNORDERED KEYS 248

length of the key, even if the keys are not natural numbers and have an
unbounded range. Consequently, if the keys do have a fixed range, the
amortized expected access time is in Θ(1).

7.2 Hashing
The technique we will develop over the remainder of this chapter is known
as hashing. The basic idea behind hashing is to convert each key k to an
index h(k) using a hash function h, so that for all k, 0 ≤ h(k) < m for some
positive integer m. h(k) is then used as an index into a hash table, which is
an array T [0..m − 1]. We then store the data item at that index.
Typically, the universe of keys is much larger than m, the size of the hash
table. By choosing our array size m to be close to the number of elements
we need to store, we eliminate the space usage problem discussed in Section
7.1. However, because the number of possible keys will now be greater than
m, we must deal with the problem that h must map more than one potential
key to the same index. When two actual keys map to the same index, it is
known as a collision.
The potential for collisions is not just a theoretical issue unlikely to
occur in practice. Suppose, for example, that we were to randomly and
independently assign indices to n keys, so that for any given key k and
index i, 0 ≤ i < m, the probability that k is assigned i is 1/m. We can
model this scenario with a discrete probability space consisting of the mn
n-tuples of natural numbers less than m. Each tuple is equally likely, and
so has probability m−n . We can then define the random variable coll as
the number of collisions; i.e., coll(hi1 , . . . , in i) is the number of ordered pairs
(ij , ik ) such that ij = ik and j < k.
coll can be expressed as the sum of indicator random variables as follows:
n−1
X n
X
coll(hi1 , . . . , in i) = I(ij = ik ).
j=1 k=j+1
CHAPTER 7. STORAGE/RETRIEVAL II: UNORDERED KEYS 249

Therefore,
 
n−1
X n
X
E[coll] = E  I(ij = ik )
j=1 k=j+1
n−1
X n
X
= E[I(ij = ik )]
j=1 k=j+1
n−1
X n
X
= P (ij = ik ).
j=1 k=j+1

For each choice of i, j, and ij , ik can take on m possible values, one of which
is ij . Because the probabilities of all elementary events are equal, it is easily
seen that P (ij = ik ) = 1/m for j < k. Hence,
n−1
X n
X
E[coll] = 1/m
j=1 k=j+1
n−1
1 X
= (n − j)
m
j=1
n−1
1 X
= j (reversing the sum)
m
j=1
n(n − 1)
=
2m
by (2.1).
For example, if our hash table has 500,000 locations and we have more
than a thousand data elements, we should expect at least one collision, on
average. In general, it requires too much space to make the table large
enough so that we can reasonably expect to have no collisions.
Several solutions to the collision problem exist, but the most common is
to use a linked list to store all data elements that are mapped to the same
location. The approach we take here is similar, but we will use a ConsList
instead of a linked list. Using a ConsList results in somewhat simpler code,
and likely would not result in any significant performance degradation. This
approach is illustrated in Figure 7.3.
In the remainder of this section, we will ignore the details of specific hash
functions and instead focus on the other implementation details of a hash
table. In order to approach the use of hash functions in a general way, we
CHAPTER 7. STORAGE/RETRIEVAL II: UNORDERED KEYS 250

Figure 7.3 Illustration of a hash table.

0 14

1 8 29

4 53 11 32

use the HashFunction ADT, shown in Figure 7.4. Note that because there
are no operations to change the hash function, the HashFunction ADT
specifies an immutable data type. In remaining sections of this chapter,
we will consider various ways of implementing a HashFunction. As we
will see in the next section, not all hash table sizes are appropriate for
every HashFunction implementation. For this reason, we allow the user
to select an approximate table size, but leave it up to the HashFunction
to determine the exact table size.
Our HashTable representation of Dictionary then consists of three
variables:

• hash: a HashFunction whose associated table size is some positive

integer m;

• table[0..m − 1]: an array of ConsLists; and

• size: a readable Nat.

Our structural invariant is that:

• for 0 ≤ i < hash.Size(), table[i] is a ConsList containing only Keyed

items;
CHAPTER 7. STORAGE/RETRIEVAL II: UNORDERED KEYS 251

Figure 7.4 The HashFunction ADT

Precondition: n ≥ 1 is an Int.
Postcondition: Constructs a HashFunction for some table size that is
at least n and strictly less than 3n.
HashFunction(n)
Precondition: k refers to a Key.
Postcondition: Returns the index i associated with k by this HashFunc-
tion. i is a Nat strictly less than the table size.
HashFunction.Index(k)
Precondition: true.
Postcondition: Returns the table size for this HashFunction.
HashFunction.Size()

• for each Keyed item x in table[i], 0 ≤ i < m, hash.Index(x.Key()) =

i; and

• the total number of Keyed items in the ConsLists is given by size.

We interpret the Keyed items in the ConsLists to contain the elements of

the data set together with their associated keys.
Accessing a data item given a key k is now straightforward — we simply
compute hash.Index(k) and search the ConsList at this location in table;
see Figure 7.5. The worst-case time for any such operation is easily seen
to be proportional to the time to compute the index plus the length of the
ConsList.
Let us now consider the worst-case length of a ConsList in table. Un-
fortunately, it can be quite bad. The following theorem shows that under
reasonable assumptions, all keys can map to the same index; hence, in the
worst case, the running time of a hash table access is in Ω(n).

Theorem 7.1 Let T be a hash table with m locations, and suppose the
universe U of possible keys contains more than m(n − 1) elements. Then for
any function h mapping U to natural numbers less than m, there is some
natural number i < m such that h maps at least n keys in U to i.
The proof of the above theorem is simply the observation that if it were
CHAPTER 7. STORAGE/RETRIEVAL II: UNORDERED KEYS 252

Figure 7.5 The HashTable implementation of Dictionary (partial)

Structural Invariant: For 0 ≤ i < hash.Size(), table[i] is a ConsList

containing only Keyed items; for each Keyed item x in table[i],
0 ≤ i < m, hash.Index(x.Key()) = i; and the total number of Keyed
items in the ConsLists is given by size.

HashTable()
size ← 0; hash ← new HashFunction(100)
table ← new Array[0..hash.Size() − 1]
for i ← 0 to SizeOf(table) − 1
table[i] ← new ConsList()

HashTable.Get(k)
i ← hash.Index(k); L ← table[i]
while not L.IsEmpty()
if L.Head().Key() = k
return L.Head().Data()
L ← L.Tail()
return nil

not true — i.e., if h maps at most n − 1 elements to each i — then the size
of U could be at most m(n − 1). Though this result looks bad, what it tells
us is that we really want h to produce a random distribution of the keys so
that the list lengths are more evenly distributed throughout the table.
For the remainder of this section, therefore, we will assume that the
key distribution is modeled by a discrete probability space hashDist. The
elementary events in hashDist are the same as those in the probability dis-
tribution defined above: all n-tuples of natural numbers less than m. Again,
the n positions in the tuple correspond to n keys, and their values give their
indices in the hash table. Regarding probabilities, however, we will make
a weaker assumption, namely, the probability that any two given distinct
positions are equal is at most ǫ, where 0 < ǫ < 1. Our earlier probability
space satisfies this property for ǫ = 1/m, but we will see in Sections 7.4 and
7.5 that other spaces do as well.
In what follows, we will analyze the expected length of the ConsList
CHAPTER 7. STORAGE/RETRIEVAL II: UNORDERED KEYS 253

searched for an arbitrary key, assuming a distribution modeled by hashDist.

In the next section we will show how to define deterministic hash functions
that approximate this distribution well enough to work very well in practice.
Then in Sections 7.4 and 7.5, we will show how to guarantee this behavior
using randomization.
For a given search in the hash table, suppose there are a total of n
keys in the table together with the key for which we are searching. Thus,
if the given key is in the hash table, there are n keys in the hash table;
otherwise, there are n − 1. We will use hashDist to model the distribution
of these n keys, where the nth key is the one for which we are searching.
Let len be the discrete random variable giving the number of positions equal
to position n in a given element of hashDist. Then if the given key is in
the hash table, E[len] gives the expected length of the ConsList searched;
otherwise, E[len] − 1 gives this expected length.
We can express len as the sum of indicator random variables as follows:
n
X
len = I(ij = in ),
j=1

where ij denotes position j of an elementary event. Applying linearity of

expectation, we have
n
X
E[len] = E[I(ij = in )]
j=1
Xn
= P (ij = in ).
j=1

Now using the fact that P (ij = in ) ≤ ǫ when j 6= n, we have

n
X
E[len] = P (ij = in )
j=1
n−1
X
≤ P (in = in ) + ǫ
j=1

= 1 + ǫ(n − 1).

The above value is the expected length of the ConsList searched when
the key is found in a table containing n keys. If the key is not in the table,
n − 1 gives the number of keys in the table, and E[len] is one greater than
CHAPTER 7. STORAGE/RETRIEVAL II: UNORDERED KEYS 254

the expected length of the ConsList. Thus, if we let n denote the number
of keys in the table, the length of the ConsList searched is expected to be
nǫ.
In either case, the length of the ConsList is linear in n if ǫ is a fixed
constant. However, ǫ may depend upon m. Thus, if ǫ ≤ c/m for some
positive constant c and we use an expandable array for the table, we can
keep the expected length bounded by a constant. Let λ = n/m be known
as the load factor of the hash table. Using the expandable array design
pattern, we can ensure that λ ≤ d, where d is a fixed positive real number
of our choosing. Thus, the expected list length is bounded by

1 + ǫn ≤ 1 + cn/m
= 1 + cλ
≤ 1 + cd
∈ O(1).

In order to implement the expandable array pattern for a hash table,

we will need to change the hash function to take advantage of the larger
range of indices. In copying elements to the new table, we therefore need
to apply the new hash function to each element in order to find its proper
location. This technique is called rehashing. We leave it as an exercise to
show that as long as the size of the table increases by at least a factor of 2
and at most a factor of 6, the amortized cost of rehashing is proportional
to the cost of hashing a single key. The HashTable.Put operation, which
employs rehashing, is shown in Figure 7.6.
We conclude that as long as the hash function is designed so that the
probability of two arbitrary keys colliding is no more than c/m, where m is
the number of locations in the hash table and c is a positive real number,
the amortized expected running time of a hash table access is in Θ(1) plus
the time needed to compute the hash function. We can keep the constant
bounding the look-up time quite small by bounding λ by a small constant,
provided c is not much larger than 1. A bound of 3/4 on λ, for example,
gives a nice trade-off between space and time.

7.3 Deterministic Hash Functions

In this section, we will consider the design of a deterministic hash function.
Theorem 7.1 guarantees that any realistic deterministic hash function will
result in linear-time accesses in the worst case. However, it is possible to
CHAPTER 7. STORAGE/RETRIEVAL II: UNORDERED KEYS 255

Figure 7.6 The Put operation for HashTable

HashTable.Put(x, k)
i ← hash.Index(k); L ← table[i]
while not L.IsEmpty()
if L.Head().Key() = k
error
L ← L.Tail()
table[i] ← new ConsList(new Keyed(x, k), table[i])
size ← size + 1
if size/SizeOf(table) > λ
hash ← new HashFunction(2 · SizeOf(table))
t ← new Array[0..hash.Size() − 1]
for j ← 0 to SizeOf(t) − 1
t[j] ← new ConsList()
for j ← 0 to SizeOf(table) − 1
L ← table[j]
while not L.IsEmpty()
y ← L.Head(); i ← hash.Index(y.Key())
t[i] ← new ConsList(y, t[i]); L ← L.Tail()
table ← t

construct a deterministic hash function for which such cases are very unlikely
to occur in practice.
We will assume that our keys are represented as natural numbers. This
assumption does not result in any loss of generality, because all data types
can be viewed as sequences of bytes, or more generally, as w-bit compo-
nents. We can view each component as a natural number less than 2w . The
sequence hk1 , . . . , kl i then represents the natural number
l
X
ki 2w(l−i) ;
i=1

thus, we view the key as encoding a natural number in radix 2w . We must

realize, however, that the keys may be very large, so that they do not fit in
a single machine word.
The basic idea of the division method is simple. For a natural number
CHAPTER 7. STORAGE/RETRIEVAL II: UNORDERED KEYS 256

Figure 7.7 The Index operation for the DivisionMethod implementation

of HashFunction

DivisionMethod.Index(k)
components[1..l] ← ToArray(k, w); h ← 0
for i ← 1 to l
h ← (h · 2w + components[i]) mod size
return h

k, we define
h(k) = k mod m,
where m is the number of array locations in the hash table. Thus, 0 ≤
k mod m < m. The table shown in Figure 7.3 uses the division method.
It is not hard to show that

(xy + z) mod m = (x(y mod m) + z) mod m

(see Exercise 7.5). This relationship gives us a top-down solution to the

problem of computing h(k) for large k. If z is the last w-bit component of
k, we can write k = 2w y + z, where y is the value obtained by removing the
component z from k. We then have

h(k) = (2w y + z) mod m

= (2w (y mod m) + z) mod m
= (2w h(y) + z) mod m.

We can therefore compute h(k) bottom-up by starting with the first com-
ponent of k and repeatedly multiplying by 2w , adding the next component,
and taking the result mod m.
The division method is illustrated in Figure 7.7, where an implementa-
tion of HashFunction is presented. The representation of HashFunction
is a Nat size, and the structural invariant is size > 0. We assume the exis-
tence of a function ToArray(x, w), which returns an array of Nats, each
strictly less than 2w , and which together give a representation of x. It is
easily seen that Index runs in time linear in the length of the key.
One advantage of the division method is that it can be applied quickly.
Because the multiplication is by a power of 2, it can be implemented by
CHAPTER 7. STORAGE/RETRIEVAL II: UNORDERED KEYS 257

shifting to the left by w bits. The addition then adds a w-bit number to
a number whose binary representation ends in w zeros; hence, the addition
can be accomplished via a bitwise or. Otherwise, there is only an application
of the mod operator for each word of the key.
The effectiveness of the division method as a hash function depends on
the value chosen for the table size. Knowing that we cannot prevent bad
cases from ever occurring, the best we can do is to try to avoid bad behavior
on cases which may be likely to occur. If data were random, our job would
be much simpler, because we could take advantage of this randomness to
generate a random distribution in the table. Real data sets, however, tend to
contain patterns. We need our hash function to perform well in the presence
of these patterns.
Suppose, for example, that the table size m = 255, and that each byte of
the key is a character encoded in ASCII. From the binomial theorem ((6.15)
on page 241), we can write the key as
l
X l
X
256l−i ki = (255 + 1)l−i ki
i=1 i=1
l−i
l X
X l−i
= 255j ki .
j
i=1 j=0

Each term of the inner sum such that j > 0 is divisible by 255; hence,
computing the key mod 255 yields:
l
! l
X X
i−1
256 ki mod 255 = ki mod 255.
i=1 i=1

Thus, applying this hash function to the key is equivalent to applying it

to the sum of the bytes in the key. Because addition is commutative, we can
see that the hash function would yield the same value for any permutation
of the bytes in the key; hence every permutation of the same bytes would
hash to the same location. Such behavior is undesirable. Similar behavior
can occur for other values very near powers of 2.
As another example, suppose that all of the keys are even numbers. If
m is even, then k mod m will always be even. As a result, only half the
locations in the table are used, so we could expect the lists to be at least
twice as long. More generally, if even keys are more likely than odd keys,
even m will cause the division method to perform poorly. This tells us that
m should not be even.
CHAPTER 7. STORAGE/RETRIEVAL II: UNORDERED KEYS 258

We can generalize the above arguments to conclude that m should ideally

be a prime number — a number with no factors other than itself and 1 —
and not too close to a power of two. It turns out that these restrictions are
nearly always sufficient to yield good performance from the division method.
The constructor DivisionMethod(n) therefore needs to select a table
size m that is a prime number in the range n ≤ m < 3n, such that m is not
too close to a power of 2. Searching for such a number can be somewhat
expensive.
Fortunately, we can simplify the search in practice. Consider the follow-
ing sequence of prime numbers:
2 5 11 23 47 97
197 397 797 1597 3203 6421
12853 25717 51437 102877 205759 411527
823117 1646237 3292489 6584983 13169977 26339969
52679969 105359939 210719881 421439783 842879579 1685759167
Suppose we were to initialize an array with these values, beginning with
index 1. Then for 2 ≤ n ≤ 230 = 1,073,741,824, the value at location ⌈lg n⌉
is at least n and strictly less than 3n. Thus, by adding an extra 2 at location
0, we can easily find a prime table size in the correct range for all n ≤ 230 ,
which is sufficiently large for most applications. Furthermore, except for
the first three or four of these values, none are close to any power of 2. We
can avoid using the first three or four sizes by setting the initial size to be
sufficiently large, but even if we were to use them, rehashing guarantees that
they will only be used for very small data sets.
One drawback to the Put operation as shown in Figure 7.6, is that
when rehashing is performed, all of the hash values must be recomputed
from scratch. When the division method is used on long keys, this can lead
to a significant amount of computation.
An improvement is to use a combination of two hash functions. The
first hash function produces an index in a range that may be too large
to be used as a table size, but which is small enough to fit into a single
machine word. This first hash function is called a compression map. The
second hash function is then applied to the result of the compression map
to produce an index into the actual table. Because the compression map
generates an index that will fit into a single machine word, the computation
of the hash table index from the compression map index can be done quickly;
for example the division method would consist of a single mod operation.
Thus, if we save the result of the compression map with the element we are
storing, we can perform rehashing by applying a new hash function to the
CHAPTER 7. STORAGE/RETRIEVAL II: UNORDERED KEYS 259

stored compression map index. We leave the implementation details of such

a scheme as an exercise.
The division method could be used for both of the hash functions in such
a scheme. However, if the modulus is near the maximum value that can be
stored in a single machine word, double-word arithmetic is required. An al-
ternative that avoids double-word arithmetic for computing the compression
map is polynomial hashing.
In order to motivate polynomial hashing, let’s consider what happens
when we pack four bytes, k1 , k2 , k3 , and k4 , into a 4-byte word. If we wish
to retain all of the information, we might produce the value

k1 2563 + k2 2562 + k3 256 + k4 .

Polynomial hashing generalizes this technique by producing, for a given key

hk1 , . . . , kl i, !
Xl
ki rl−i mod 2w ,
i=1
where w is the number of bits in a machine word, excluding any sign bit. The
final “mod 2w ” describes the effect of overflow in a w-bit unsigned integer.
Thus, if an unsigned integer is used, this operation need not be explicitly
performed.
One reason this technique is popular is that it can be computed quickly.
Note that
l
X l−1
X
ki rl−i = kl + ki rl−i
i=1 i=1
Xl−1
= kl + r ki rl−1−i .
i=1

This gives us a top-down solution that can be applied bottom-up in the same
way as we applied the division method directly to large keys. Specifically,
we start with k1 and repeatedly multiply by r and add the next ki . This
procedure requires one multiplication and one addition for each component
of the key. Furthermore, all computation can be done with single-word
arithmetic.
In order for this method to work well, r must be chosen properly. We first
note that 256 is a poor choice, because 256i mod 2w = 0 for all i ≥ w/8;
thus only the first w/8 components of the key are used in computing the hash
value. More generally, r should never be even, because (c2j )i mod 2w = 0
CHAPTER 7. STORAGE/RETRIEVAL II: UNORDERED KEYS 260

for j > 0 and i ≥ w/j. Furthermore, not all odd values work well. For
example, r = 1 yields ri = 1 for all i, so that the result is simply the
sum of the components, mod 2w . This has the disadvantage of causing all
permutations of a key to collide.
More generally, if r is odd, ri mod 2w will repeat its values in a cyclic
fashion. In other words, for every odd r there is a natural number n such
that rn+i mod 2w = ri for all i ∈ N. Fortunately, there are only a few values
of r (like 1) that have short cycles. In order to avoid these short cycles, we
would like to choose r so that this cycle length is as large as possible. It
is beyond the scope of this book to explain why, but it turns out that this
cycle length is maximized whenever r mod 8 is either 3 or 5.
We can run into other problems if r is small and the component size
is smaller than w. Suppose, for example that r = 3, w = 32, and each
component is one byte. For any key containing fewer than 15 components,
the polynomial-hash value will be less than 231 . We have therefore reduced
the range of possible results by more than half — much more for shorter
keys. As a result, more collisions than necessary are introduced. A similar
phenomenon occurs if r is very close to 2w .
If we avoid these problems, polynomial hashing usually works very well
as a compression map. To summarize, we should choose r so that r mod 8
is either 3 or 5, and not too close to either 0 or 2w . This last condition can
typically be satisfied if we choose an r with 5-9 bits (i.e., between 16 and
512). The division method can then be used to obtain an index into the
table. Because it will be applied to a single word, its computation consists
of a single mod operation.

7.4 Universal Hashing

Though the techniques discussed so far are widely used and work well in
practice, we cannot prove much of anything useful about their performance.
In this section, we consider how to use randomization to yield a hashing
strategy with provably good expected behavior.
We cannot simply store data items in random array locations because
we would then be unable to find them quickly. We can, however, randomly
select a hash function from a set of alternatives. If the set of potential hash
functions is chosen well, we can prove that the resulting hash table will be
expected to have few collisions.
Let U be our universe of keys, and let H be some countable set of hash
functions of the form h : U → M , where M is the set of natural numbers less
CHAPTER 7. STORAGE/RETRIEVAL II: UNORDERED KEYS 261

than m. Let us also suppose that each element of H has some probability, so
that H is a discrete probability space. Two distinct keys k1 and k2 collide for
h ∈ H iff h(k1 ) = h(k2 ). Taking h(k1 ) and h(k2 ) as random variables over
H, we see that the probability that these keys collide is P (h(k1 ) = h(k2 )). If
two values from M are chosen independently with uniform probability, then
the probability that they are the same is 1/m. We therefore say that a H is
a universal family of hash functions if for any two keys in U , the probability
that they collide is no more than 1/m. As we showed in Section 7.2, this
probability bound implies that for any hash table access, the expected length
of the list searched is in Θ(1).
Several universal families of hash functions have been defined, but most
of them require some number theory in order to prove that they are universal
families. In what follows, we present a universal family that is easier to
understand at the cost of requiring a bit more computational overhead.
Then in the next section, we will show how number theory can be utilized
to define universal families whose hash functions can be computed more
efficiently.
Suppose each key k ∈ U is encoded by l bits. Ideally, we would like to
generate each function mapping U into M with equal probability. However,
doing so is too expensive. There are 2l keys in U , and m possible values to
which each could be mapped. The total number of possible hash functions is
l
therefore m2 . Uniquely identifying one of these functions therefore requires
l
at least lg m2 = 2l lg m bits. If, for example, each key is 32 bits and our
hash table size is 256, four gigabytes of storage would be needed just to
identify the hash function.
Instead, we will randomly generate a table location for each of the l bit
positions. Let these locations be t1 , . . . , tl . We will assume that m is a power
of 2 so that each of these locations is encoded using lg m bits. A given key
k will select the subsequence of ht1 , . . . , tl i such that ti is included iff the ith
bit of k is a 1. Thus, each key selects a unique subsequence of locations.
The hash table location of k is then given by the bitwise exclusive-or of the
locations in the subsequence; in other words, the binary encoding of the hash
location has a 1 in position j iff the number of selected locations having a 1
in position j is odd.

Example 7.2 Suppose our keys contain 4 bits, and we want to use a hash
table with 8 locations. We then randomly generate 4 table locations, one
for each of the 4 bit positions in the keys:

• t1 = 3, or 011 in binary;
CHAPTER 7. STORAGE/RETRIEVAL II: UNORDERED KEYS 262

• t2 = 6, or 110 in binary;

• t3 = 0, or 000 in binary;

• t4 = 3, or 011 in binary.

Note that these locations don’t need to be distinct.

Now let us compute the hash value for the keys 5 and 11, whose binary
encodings are 0101 and 1011. The key 5 selects the locations t2 and t4 , whose
binary encodings are 110 and 011, respectively. The bitwise exclusive-or of
these two values is 101 because the first and third bit positions each have an
odd number of 1, but the second has an even number. The key 5 therefore
is placed in location 5. Likewise, the key 11 selects the locations t1 , t3 ,
and t4 , whose binary encodings are 011, 000, and 011, respectively. The
bitwise exclusive-or of these three values is 000 because each of the three bit
positions contains an even number of 1s. The hash value for 11 is therefore
0.
To see why we use the exclusive-or operation, suppose we have a bit
value x that is 1 with probability p, 0 ≤ p ≤ 1. Suppose we then assign a
value to bit y by flipping a fair coin; i.e., y has a value of 1 with probability
1/2, independent of the value of x. The exclusive-or of x and y is 1 iff the
values of x and y are different. The probability of this event is therefore
p 1−p 1
+ = .
2 2 2
Thus, the probability distribution of the exclusive-or of two independent
random bits is uniform if at least one of the two has uniform probability
distribution. We can easily conclude that the method outlined above for
selecting a hash function results in each key mapping to any given table
location with probability 1/m.
However, knowing that each key maps to any given table location with
probability 1/m is not sufficient to conclude that any two keys collide with
probability at most 1/m. Suppose, for example, that we were to select a
hash function by randomly generating a natural number i < m with uniform
probability. The hash function then maps all keys to i. For any given key k,
this strategy maps k to any given location with probability 1/m. However,
because all keys map to the same location, the probability that two given
keys collide is 1 for each pair of keys.
Before we try to prove that this family of hash functions is universal, we
will define it more formally. In order to accommodate a formal definition,
CHAPTER 7. STORAGE/RETRIEVAL II: UNORDERED KEYS 263

we must first define a discrete probability space that will represent the set
of hash functions. Let Sl,m be the set of all l-tuples of bit strings of length
lg m, where l is a positive integer and m is a power of 2. Each of these
l-tuples will represent a hash function. Note that Sl,m has ml elements. We
therefore assign each element of Sl,m a probability of m−l ; hence, Sl,m is
a discrete probability space in which each elementary event has the same
probability.
We can now formally define a hash function corresponding to each ele-
ment in Sl,m . Let select be the function that takes a sequence s = ht1 , . . . , tn i
of bit strings all having the same length, together with a bit string k1 · · · kn ,
and returns the subsequence of s such that ti is included iff ki = 1. Fur-
thermore, let X be the function that takes a sequence of bit strings each
having the same length and returns their bitwise exclusive-or. Given s =
ht1 , . . . , tl i ∈ Sl,m , let hs : U → M such that

hs (k) = X(select(s, k)).

We now define
1
Hl,m = {hs | s ∈ Sl,m }.
1
Each element h ∈ Hl,m corresponds to the event consisting of all sequences
s ∈ Sl,m such that h = hs . We leave it as an exercise to show that for
1 , there is exactly one such s; hence, there is a one-to-one
each h ∈ Hl,m
correspondence between elementary events in Sl,m and hash functions in
1 . We will now show that for every distinct k, k ′ ∈ U , P (h(k) = h(k ′ )) =
Hl,m
1
1/m, so that Hl,m is a universal family of hash functions. In the proof and
the implementation that follows, we use ⊗ to denote bitwise exclusive-or.

Theorem 7.3 Let l be a positive integer and m be a power of 2. Then

1
Hl,m is a universal family of hash functions.

Proof: Suppose k and k ′ are two keys differing in bit position i. Without
loss of generality, suppose the ith bit of k is 0 and the ith bit of k ′ is 1.
Let k ′′ be the key obtained from k ′ by changing the ith bit to 0. Let tj
be the discrete random variable giving the value of the jth component of s
for s ∈ Sl,m , and let h(x) be the random variable giving the hash value of
x ∈ U . Then h(k ′ ) = h(k ′′ ) ⊗ ti . Thus, h(k) = h(k ′ ) iff h(k) = h(k ′′ ) ⊗ ti .
Because the ith bits of both k and k ′′ are 0, we can evaluate h(k) and
h(k ′′ ) knowing only t1 , . . . , ti−1 , ti+1 , . . . tl . For each choice of these values,
there is exactly one value of ti for which h(k) = h(k ′′ ) ⊗ ti , namely ti =
CHAPTER 7. STORAGE/RETRIEVAL II: UNORDERED KEYS 264

Figure 7.8 UniversalHash1 implementation of HashFunction

Structural Invariant: size = 2i for some Nat i, and for 1 ≤ j ≤ l,

indices[j] < size.

UniversalHash1(n)
size ← 2⌈lg n⌉ ; indices ← new Array[1..l]
for i ← 1 to l
indices[i] ← Random(size)

UniversalHash1.Index(k)
bits[1..l] ← ToArray(k, 1); h ← 0
for i ← 1 to l
if bits[i] = 1
h ← h ⊗ indices[i]
return h

h(k) ⊗ h(k ′′ ). There are then ml−1 hash functions for which k and k ′ collide.
Because each hash function occurs with probability m−l ,

P (h(k) = h(k ′ )) = ml−1 m−l

= 1/m.

To represent an instance of this family, we use a readable Nat size and

an array indices[1..l] of Nats; we assume for now that l, the number of bits
in a key, is fixed. Our structural invariant is that size = 2i for some natural
number i, and that for 1 ≤ j ≤ l, indices[j] < size. The implementation
is shown in Figure 7.8. It uses a function Random, which takes a positive
integer n as input and returns, with uniform probability, any natural number
strictly less than n. It is easily seen that both the constructor and the Index
operation run in Θ(l) time, assuming Random runs in Θ(1) time.
If we use this implementation of HashFunction with the HashTable
implementation shown in Figures 7.5 and 7.6, the expected search time is
in Θ(1). Furthermore, it is not hard to show that the expected amortized
cost of rehashing is in Θ(l).
CHAPTER 7. STORAGE/RETRIEVAL II: UNORDERED KEYS 265

In many applications, the key lengths may vary, and we may not know
the maximum length in advance. Such situations can be handled easily,
provided we may pad keys with zeros without producing other valid keys.
This padding may be done safely if the length of the key is encoded within
the key, or if each key is terminated by some specific value. We can therefore
consider each key as having infinite length, but containing only finitely many
1s. We can ensure that we have bit strings for indices[1..i] for some i. If we
encounter a key with a 1 in bit position j > i, we can generate bit strings for
positions i + 1 through j at that time. Note that neither of these strategies
add any significant overhead — they simply delay the generation of the bit
strings. We leave the implementation details as an exercise.

7.5 Number Theoretic Universal Hash Families

In this section, we will use some elementary number theory to obtain uni-
versal families whose hash functions resemble those of Section 7.3. The
1 .
resulting functions may require less overhead than do the functions in Hl,m
We first need the following fact from number theory:

Theorem 7.4 Let a, b, and m be natural numbers such that 0 < a < m
and b < m. Then the equation

ai mod m = b

has a unique solution in the range 0 ≤ i < m iff a and m are relatively prime
(i.e., 1 is the greatest common divisor of a and m).

Proof: Because we will only need to use this theorem in one direction, we
will only prove one implication and leave the other as an exercise.

⇐: Suppose a and m are relatively prime. We will show that if ai mod m =

aj mod m, where 0 ≤ i < m and 0 ≤ j < m, then i = j. Thus, each of the
m possible values of i will result in a distinct value of ai mod m. Because
only m distinct values of ai mod m are possible, it will follow that one of
the values of i must yield ai mod m = b.
Suppose ai mod m = aj mod m. Then there exist natural numbers q1
and q2 such that

ai − q1 m = aj − q2 m
a(i − j) = (q1 − q2 )m,
CHAPTER 7. STORAGE/RETRIEVAL II: UNORDERED KEYS 266

so that a(i − j) is divisible by m. Because a and m are relatively prime, it

must be the case that (i − j) is divisible by m. Given the ranges for i and j,
it must be the case that |i − j| < m. The only multiple of m with absolute
value strictly less than m is 0; hence, i = j.

For our next universal family, we will interpret the keys as natural num-
bers and assume that there is some maximum value for a key. Let p be a
prime number strictly larger than this maximum key value. Our hash func-
tions will consist of two steps. The first step will map each key to a unique
natural number less than p. We will design this part so that, depending
on which hash function is used, a distinct pair of keys will be mapped with
uniform probability to any of the pairs of distinct natural numbers less than
p. The second step will apply the division method to scale the value to an
appropriate range.
For the first step, let

hp,a,b (k) = (ak + b) mod p (7.1)

for a and b strictly less than p. Consider distinct keys k and k ′ . We then
have

(hp,a,b (k) − hp,a,b (k ′ )) mod p = ((ak + b) mod p − (ak ′ + b) mod p) mod p

= a(k − k ′ ) mod p

from Exercise 7.5. Because k − k ′ 6= 0 and p is prime, Theorem 7.4 tells us

that for each natural number j < p, there is a unique a, 0 ≤ a < p, such
that

j = a(k − k ′ ) mod p
= (hp,a,b (k) − hp,a,b (k ′ )) mod p,

independent of the value of b. Because hp,0,b (k)−hp,0,b (k ′ ) = 0, each positive

a < p yields a distinct positive value of (hp,a,b (k) − hp,a,b (k ′ )) mod p. Fur-
thermore, for a given positive a, each choice of b clearly results in a distinct
value for hp,a,b (k).
Each choice of a and b, where 0 < a < p and 0 ≤ b < p, therefore
results in a unique pair of distinct values hp,a,b (k) and hp,a,b (k ′ ). Because
the number of choices of a and b is exactly the same as the number of pairs
of distinct values hp,a,b (k) and hp,a,b (k ′ ), each of these pairs can be produced
by exactly one choice of a and b. We therefore have the following lemma.
CHAPTER 7. STORAGE/RETRIEVAL II: UNORDERED KEYS 267

Lemma 7.5 Let p be a prime number, and let k and k ′ be distinct natural
numbers strictly less than p. If a and b are chosen independently and uni-
formly such that 1 ≤ a < p and 0 ≤ b < p, then hp,a,b (k) and hp,a,b (k ′ ) are
any pair of distinct natural numbers less than p with uniform probability.
To apply the second step of the hash function, let

fm (i) = i mod m,

where m is a positive integer and i is a natural number. We then define

2
Hp,m = {fm ◦ hp,a,b | 0 < a < p, 0 ≤ b < p},

where ◦ denotes function composition (i.e., fm ◦ hp,a,b (k) = fm (hp,a,b (k))).

We define the probability of each element of Hp,m2 by selecting a and b
independently with uniform probability. We can then show the following
theorem.
2
Theorem 7.6 For any prime number p and positive integer m, Hp,m is a
universal family of hash functions.

Proof: Let k and k ′ be two distinct keys. As we argued above, hp,a,b (k)
and hp,a,b (k ′ ) are distinct natural numbers less than p, and each possible
pair of distinct values can be obtained by exactly one pair of values for a
and b. fm (hp,a,b (k)) = fm (hp,a,b (k ′ )) iff hp,a,b (k) mod m = hp,a,b (k ′ ) mod m
iff hp,a,b (k)−hp,a,b (k ′ ) is divisible by m. For any natural number i < p, there
are strictly fewer than p/m natural numbers j < p (other than i) such that
i − j is divisible by m. Because the number of these values of j is an integer,
it is at most (p − 1)/m. Because there are p possible values of hp,a,b (k) and
p(p − 1) possible pairs of values for hp,a,b (k) and hp,a,b (k ′ ), each of which is
equally likely, the probability that fm (hp,a,b (k)) = fm (hp,a,b (k ′ )) is at most

p p−1
m 1
= .
p(p − 1) m

Note that by the above theorem, Hp,m 2 is universal for any positive m.
As a result, the size of the hash table does not need to be a particular kind of
number, such as a prime number or a power of 2, in order for this strategy
to yield good expected performance. However, the restriction that p is a
CHAPTER 7. STORAGE/RETRIEVAL II: UNORDERED KEYS 268

prime number larger than the value of the largest possible key places some
limitations on the effectiveness of this approach. Specifically, if there is no
upper bound on the length of a key, we cannot choose a p that is guaranteed
to work. Furthermore, even if an upper bound is known, unless it is rather
small, the sizes of p, a, and b would make the cost of computing the hash
function too expensive.
Let us therefore treat keys as sequences of natural numbers strictly
smaller than some value p, which we presume to be not too large (e.g.,
small enough to fit in a single machine word). Furthermore, let us choose
p to be a prime number. Let hk1 , . . . , kl i be a key, and let s = ha1 , . . . , al i
be a sequence of natural numbers, each of which is strictly less than p. We
then define !
l
X
hp,s (hk1 , . . . , kl i) = ai ki mod p.
i=1

We first observe that we cannot guarantee that hp,s (k) 6= hp,s (k ′ ) for
each distinct pair of keys k and k ′ . The reason for this is that there are
potentially more keys than there are values of hp,s . However, suppose k and
k ′ are distinct keys, and let ki 6= ki′ , where 1 ≤ i ≤ l. Let us arbitrarily fix
the values of all aj such that j 6= i, and let
 
i−1
X l
X i−1
X l
X
c= aj kj′ + aj kj′ − aj kj − aj kj  mod p.
j=1 j=i+1 j=1 j=i+1

Then
 
l
X l
X
(hp,s (k) − hp,s (k ′ )) mod p =  aj kj − aj kj′  mod p
j=1 j=1

= (ai (ki − ki′ ) − c) mod p.

Because 0 ≤ c < p, the above value is 0 iff

ai (ki − ki′ ) mod p = c.

Because ki 6= ki′ and p is prime, ki − ki′ and p are relatively prime.

From Theorem 7.4, the above equation has a unique solution for ai such
that 0 ≤ ai < p. Thus, for each choice of a1 , . . . , ai−1 , ai+1 , . . . , al , there
is exactly one choice of ai such that (hp,s (k) − hp,s (k ′ ) mod p) = 0. Note
that from the range of hp,s , the only way (hp,s (k) − hp,s (k ′ )) mod p = 0 is if
hp,s (k) = hp,s (k ′ ). We therefore have the following lemma.
CHAPTER 7. STORAGE/RETRIEVAL II: UNORDERED KEYS 269

Lemma 7.7 Let p be a prime number and l a positive integer. In addition,

let s = ha1 , . . . , al i, where each aj is chosen independently and uniformly
such that 0 ≤ aj < p. Then the probability that hp,s (k) = hp,s (k ′ ) for
distinct keys k and k ′ is 1/p.
We now define
3
Hp,l = {hp,s | s = ha1 , . . . , al i, 0 ≤ ai < p for 1 ≤ i ≤ l}. (7.2)

We define the probability of each element of Hp,l3 by selecting each a in-

i
dependently with uniform probability. Note that the range of each hash
3 is the set of natural numbers strictly less than p. Therefore,
function in Hp,l
based on the above discussion, we have the following theorem.
3 is a
Theorem 7.8 For any prime number p and positive integer l, Hp,l
universal family of hash functions.
If we know in advance the approximate size of our data set and the
maximum key length, we can select an appropriate prime value for p and
randomly select the appropriate hash function from Hp,l 3 . Because we can

apply the mod operation after each addition, we are always working with
values having no more than roughly twice the number of bits as p; hence,
we can compute this hash function reasonably quickly for each key. Fur-
thermore, even if we don’t know the maximum key length, we can generate
the multipliers ai as we need them.
However, if we don’t know in advance the approximate size of the data
set, we may need to use rehashing. For the sake of efficiency, we would like to
avoid the need to apply a new hash function to the entire key. Furthermore,
as we will see in the next section, it would be useful to have a universal family
that is appropriate for large keys and for which the table size is unrestricted.
A straightforward attempt to achieve these goals is to combine Hp,l 3 with
2 . Specifically, we define
Hp,m
4 2 3
Hp,l,m = {h1 ◦ h2 | h1 ∈ Hp,m , h2 ∈ Hp,l }.

Hash functions in this family are of the form,

h(k) = hp,a,b (hp,s (k)) mod m

l
!
X
= a ai ki + b mod p mod m (7.3)
i=1
CHAPTER 7. STORAGE/RETRIEVAL II: UNORDERED KEYS 270

where a, b, and each ai are natural numbers, and a 6= 0. We define the prob-
4
ability of each element of Hp,l,m by selecting a, b, and each ai independently
with uniform probability. (We leave it as an exercise to show that the same
4
probability distribution for Hp,l,m can be achieved by setting a = 1 and
selecting b and each ai independently with uniform probability.)
Because Hp,m2 is a universal family, it causes any pair of distinct keys to
collide with probability at most 1/m. However, Hp,l 3 also causes distinct keys

to collide with probability 1/p. When the function from Hp,m 2 is applied to
equal values, it yields equal values. We must therefore be careful in analyzing
4
the probability of collisions for Hp,l,m .
Let us first consider the case in which two distinct keys k and k ′ are
mapped to distinct values by hp,s ∈ Hp,l 3 . From Lemma 7.7, the probability

that this occurs is

1 p−1
1− = .
p p
Furthermore, from Lemma 7.5, hp,a,b (hp,s (k)) and hp,a,b (hp,s (k ′ )) are with
uniform probability any pair of distinct natural numbers less than p, pro-
vided a and b are chosen independently with uniform probability such that
1 ≤ a < p and 0 ≤ b < p. Because there are p(p − 1) pairs of distinct natural
numbers less than p, this probability is
1
.
p(p − 1)
Therefore, given any two distinct keys k and k ′ , and any two distinct natural
numbers i and j strictly less than p, the probability that hp,a,b (hp,s (k)) = i
and hp,a,b (hp,s (k ′ )) = j is

p−1 1 1
= 2.
p p(p − 1) p

Now consider the case in which hp,s (k) = hp,s (k ′ ). From Lemma 7.7, this
case occurs with probability 1/p. For any value of a, 1 ≤ a < p, and any
value of i, 0 ≤ i < p, there is exactly one value of b such that 0 ≤ b < p and

(ahp,s (k) + b) mod p = i.

Thus, each value of i is reached with probability 1/p. Therefore, for each
natural number i < p, the probability that hp,a,b (hp,s (k)) = hp,a,b (hp,s (k ′ )) =
i is 1/p2 .
Thus, for a hash function h chosen from Hp,l,m4 , h(k) = i mod m and
′
h(k ) = j mod m, where i and j are natural numbers less than p chosen
CHAPTER 7. STORAGE/RETRIEVAL II: UNORDERED KEYS 271

Figure 7.9 Pairs resulting in collisions when applying mod

p − (p mod m)

p
m collisions p − (p mod m) p

independently with uniform probability. Furthermore, i mod m = j mod m

iff i − j is divisible by m. Because p − (p mod m) is divisible by m, for any
i, exactly 1 of every m values j such that 0 ≤ j < p − (p mod m) is such
that i − j is divisible by m. Likewise, for any j, exactly 1 of every m values
i such that 0 ≤ i < p − (p mod m) is such that i − j is divisible by m (see
Figure 7.9). Thus, of the p2 − (p mod m)2 pairs in which at least one value
is less than p − (p mod m), exactly

p2 − (p mod m)2
m
pairs result in collisions. Of the remaining (p mod m)2 pairs, only those
in which i = j result in collisions. There are exactly p mod m such pairs.
CHAPTER 7. STORAGE/RETRIEVAL II: UNORDERED KEYS 272

Thus, the probability of a collision is exactly

p2 − (p mod m)2 p mod m p2 + m(p mod m) − (p mod m)2
+ =
mp2 p2 mp2
1 (m − (p mod m))(p mod m)
= + . (7.4)
m mp2
Clearly, m − (p mod m) is always positive and p mod m is always non-
negative. Furthermore, because p is prime, the only way p mod m can be
0 is if m = 1 or m = p. A hash table of size 1 is simply a ConsList,
and selecting m = p would defeat the purpose of combining Hp,l 3 with H2 .
p,m
Thus, for all reasonable values of m, the second fraction on the right-hand
4
side of (7.4) is strictly positive; therefore, Hp,l,m is not a universal family.
However, recall from Section 7.2 that Θ(1) amortized expected perfor-
mance can be achieved using rehashing if the probability of collisions is
bounded by c/m for some positive real number c. We therefore define a
family of hash functions to be c-universal if for each pair of distinct keys,
the probability of a collision is at most c/m. In what follows, we will derive
4
a c such that Hp,l,m is c-universal whenever 1 < m < p.
Specifically, we need to find a real number c such that whenever p is
prime and 1 < m < p,
c 1 (m − (p mod m))(p mod m)
≥ +
m m mp2
(m − (p mod m))(p mod m)
c≥1+ . (7.5)
p2
Let us fix p at an arbitrary prime value and try to maximize the right-hand
side of (7.5). Suppose m < p/2. Then p − (p mod m) > m, and

p mod (p − p mod m) = p mod m.

Thus, by replacing m with p − p mod m in (7.5), we increase the value of

the expression. In order to maximize its value, we therefore need m ≥ p/2.
Because p is prime, we cannot have m = p/2, so we can assume m > p/2.
If m > p/2, then p mod m = p − m. It therefore suffices to maximize

(m − (p − m))(p − m) = (2m − p)(p − m)

= −(2m2 − 3mp + p2 ),

or equivalently, to minimize

f (m) = 2m2 − 3mp + p2 .

CHAPTER 7. STORAGE/RETRIEVAL II: UNORDERED KEYS 273

There are several ways to find the minimum value of a quadratic, but
one way that does not involve calculus is by the technique of completing
the square. A quadratic of the form (ax − b)2 is clearly nonnegative for all
values of a, x, and b. Furthermore, it reaches a value of 0 (its minimum) at
x = b/a. We can therefore minimize f (m) by finding a value d such that
f (m) − d is of the form

(am − b)2 = a2 m2 − 2abm + b2 .

Because f (m) − d reaches a minimum value of 0 at m = b/a, f (m) reaches

a minimum value of d at the same point. √
In order to make the coefficient of m2 have a value of 2, a must be 2.
To find the coefficient b, we must solve
√
3mp = 2 2bm
3p
b= √ .
2 2
To find d, we must then solve
2
2 3p
p −d= √
2 2
9p2
=
8
p2
d=− .
8
Thus, −f (m) — and hence the numerator of the second term in the right-
hand side of (7.5) — is never more than p2 /8. Furthermore, this value is
achieved (assuming for the moment that m varies over the real numbers)
when
3p
√
2 2
m= √
2
3p
= .
4
We conclude that the right-hand side of (7.5) is bounded above by

p2 /8
1+ = 9/8.
p2
We therefore have the following theorem.
CHAPTER 7. STORAGE/RETRIEVAL II: UNORDERED KEYS 274

Theorem 7.9 For any prime number p and positive integers l and m such
4
that 1 < m < p, Hp,l,m is 9/8-universal.
The upper bound of 9/8 can be reached when m = 3p/4; however, in
order for this equality to be satisfied, p must be a multiple of 4, and hence
cannot be prime. We can, however, come arbitrarily close to this bound by
using a sufficiently large prime number p and setting m to either ⌊3p/4⌋ or
⌈3p/4⌉. Practically speaking, though, such values for m are much too large.
In practice, m would be much smaller than p, and as a result, the actual
probability of a collision would be much closer to 1/m.
By choosing p to be of an appropriate size, we can choose a single h of
the form !
X l
h(k) = a ai ki + b mod p,
i=1

and change m as we need to rehash. For example, if the maximum array

size on a given platform is 231 − 1, which by happy coincidence is prime,
we can set p to this value. We can then break the keys into 2- or 3-byte
components and randomly select a, b, and a1 , . . . , al . We can select any value
of m < 231 − 1 as our table size, but a power of 2 works particularly well,
because h(k) mod m is just the low-order lg m bits of h(k). As we compute
the hash value h(k) mod m for each key k, we save the value h(k). Note that
this value can be computed using 64-bit arithmetic and stored as a 32-bit
(signed or unsigned) integer. If we need to rehash, we double the size of the
table. We can then compute the new hash values for each k by looking up
h(k) and computing h(k) mod 2m.
If p = 231 −1 and m is a power of 2, then p mod m = m−1. Substituting
this value into (7.4), we see that the probability of a collision is
1 m−1 1 1
+ < +
m mp2 m (231 − 1)2
1
< + 2−61 .
m

7.6 Perfect Hashing

In this section, we consider a restricted form of Dictionary for which
Put and Remove are not allowed; i.e., updates will never be made to the
structure after it is created. In order for such a structure to be useful, we
need to modify the constructor to receive as input the elements to be stored.
CHAPTER 7. STORAGE/RETRIEVAL II: UNORDERED KEYS 275

Figure 7.10 The ImmutableDictionary ADT

Precondition: elements[0..n − 1] is an array of non-nil items, keys[0..n − 1]

is an array of distinct Keys, n ∈ N.
Postcondition: Constructs a ImmutableDictionary containing all of
the items in elements[0..n − 1] using keys[i] as the key for elements[i] for
0 ≤ i < n.
ImmutableDictionary(elements[0..n − 1], keys[0..n − 1])
Precondition: k is a key.
Postcondition: Returns the item with key k, or nil if no item with key k
is contained in the set.
ImmutableDictionary.Get(k)
Precondition: true.
Postcondition: Returns the number of items in the set.
ImmutableDictionary.Size()

The formal specification of the ImmutableDictionary ADT is shown in

Figure 7.10.
If we expect to make a large number of accesses to an ImmutableDic-
tionary, it might make sense to invest more time in constructing it if we
can then guarantee that accesses will be fast. To achieve this goal, we use a
technique called perfect hashing.
One of the drawbacks to hashing is that we can’t guarantee that there will
be no collisions. In fact, we can’t even guarantee that all of keys don’t hash
to the same location. Universal hashing gives us an expectation that the
resulting hash table will not have too many collisions. Thus, even though we
might be unlucky and choose a hash function that yields poor performance
on our data set, if we randomly select several different hash functions, we
can expect to find one that yields a small number of collisions.
With perfect hashing, our goal is to produce a hash table with no colli-
sions. Unfortunately, as we saw in Section 7.2, unless the size of the hash
table is much larger than the number of keys, we can expect to have at least
one collision. With a reasonable table size, we would probably need to try
many different hash functions before we found one that yielded no collisions.
We can avoid this difficulty, however, by employing a two-level approach
CHAPTER 7. STORAGE/RETRIEVAL II: UNORDERED KEYS 276

Figure 7.11 The structure of a perfect hash table

First level Second level

0 12
0 57
1
1
2
0 2
3 51
1 15 3 64
4
2 27 4
5 24
3 5 36
6
6
7 16
7
8
8
9 83

(see Figure 7.11). Instead of using a ConsList to store all of the elements
that hash to a certain location, we use a secondary hash table with its own
hash function. The secondary hash tables that store more than one element
are much larger than the number of elements they store. As a result, we
will be able to find a hash function for each secondary hash table such that
no collisions occur. Furthermore, we will see that the sizes of the secondary
hash tables can be chosen so that the total number of locations in all of the
hash tables combined is linear in the number of elements stored.
Let us first determine an appropriate size m for a secondary hash table in
which we need to store n distinct keys. We saw in Section 7.2 that in order
for the expected number of collisions to be less than 1, if the probability
that two keys collide is 1/m, then m must be nearly n2 . We will therefore
assume that m ≥ n2 .
Let Hm be a c-universal family of hash functions. We wish to determine
an upper bound on the number of hash functions we would need to select
from Hm before we can expect to find one that produces no collisions among
CHAPTER 7. STORAGE/RETRIEVAL II: UNORDERED KEYS 277

the given keys. Let coll be the discrete random variable giving the total
number of collisions, as defined in Section 7.2, produced by a hash function
h ∈ Hm on distinct keys k1 , . . . , kn . As we showed in Section 7.2,
n−1
X n
X
E[coll] = P (h(ki ) = h(kj )).
i=1 j=i+1

Because the probability that any two distinct keys collide is no more than
c/m ≤ c/n2 , we have
n−1 n
X X c
E[coll] ≤
n2
i=1 j=i+1
n−1
c X
= (n − i)
n2
i=1
n−1
c X
= i (reversing the sum)
n2
i=1
cn(n − 1)
= (by (2.1))
2n2
< c/2.

From Markov’s Inequality (5.3) on page 194, the probability that there is at
least one collision is therefore less than c/2.
Suppose, for example, that c = 1, as for a universal hash family. Then
the probability that a randomly chosen hash function results in no collisions
4
is greater than 1/2. If c = 9/8, as for Hp,l,m , then the probability is greater
than 7/16. Suppose we repeatedly select hash functions and try storing
the keys in the table. Because the probability that there are no collisions
is positive whenever c < 2, we will eventually find a hash function that
produces no collisions.
Let us now determine how many hash functions we would expect to
try before finding one that results in no collisions. Let reps be the discrete
random variable giving this number. For a given positive integer i, P (reps ≥
i) is the probability that i − 1 successive hash functions fail; i.e.,

P (reps ≥ i) < (c/2)i−1

= (2/c)1−i .
CHAPTER 7. STORAGE/RETRIEVAL II: UNORDERED KEYS 278

From Theorem 5.5,

∞
X
E[reps] = P (reps ≥ i)
i=1
X∞
< (2/c)1−i .
i=1

Suppose c < 2. Then we can re-index the sum to begin at 0 and apply
Theorem 6.7, yielding
∞
X
E[reps] < (2/c)−i
i=0
2/c
=
(2/c) − 1
2
= .
2−c
Note that the above value is a fixed constant for fixed c < 2. Thus,
the expected number of attempts at finding an appropriate secondary hash
function is bounded by a fixed constant. For example, with c = 1, the value
of this constant is less than 2, or with c = 9/8, the value is less than 16/7.
As a result, we would expect that the number of times a secondary hash
function is applied to any key during the process placing keys in secondary
hash tables is bounded by a constant.
We must now ensure that the total space used by the primary and sec-
ondary hash tables (and hence the time needed to initialize them) is linear
in n, the total number of keys. Suppose the primary hash table has m loca-
tions. Further suppose that ni keys are mapped to index i in the primary
hash table. We will then construct a HashFunction by passing n2i to the
constructor of an implementation providing a c-universal hash family. Due
to the specification of the HashFunction constructor, the actual Hash-
Function constructed may contain up to 3n2i − 1 locations when ni > 0.
The size of the table constructed is therefore linear in n2i . The actual space
used by all of the secondary hash tables is therefore linear in
m−1
X
n2i .
i=0

Let sumsq be a discrete random variable denoting the above sum. The
expected space usage of the secondary hash tables is then linear in E[sumsq].
CHAPTER 7. STORAGE/RETRIEVAL II: UNORDERED KEYS 279

In order to analyze E[sumsq], we first observe that n2i is closely related to

the number of collisions at index i. The number of collisions at index i is
ni (ni − 1)/2, so that
"m−1 #
X ni (ni − 1)
E[coll] = E
2
i=0
"m−1 # "m−1 #!
1 X X
= E n2i − E ni
2
i=0 i=0
= (E[sumsq] − E[n])/2
= (E[sumsq] − n)/2.

Rearranging terms, we have

E[sumsq] = 2E[coll] + n.

By reasoning as in Section 7.2, it is easily seen that if the probability

that two keys collide is at most c/m, then

cn(n − 1)
E[coll] ≤ .
2m
Hence,

E[sumsq] = 2E[coll] + n
cn(n − 1)
≤ + n. (7.6)
m
Thus, if m ∈ Θ(n), the expected number of locations in the primary
hash table and all of the secondary hash tables is in Θ(n). In particular, if
m ≥ n, then E[sumsq] ≤ (c + 1)n. It turns out that the value of m that
minimizes
cn(n − 1)
+n+m
m
is roughly n (see Exercise 7.17); hence, we will construct our primary hash
function by passing n to constructor for an appropriate implementation of
HashFunction.
Of course, we could be unlucky in selecting a primary hash function, so
that the number of secondary locations is much larger than what we expect.
For example, if it happens that all keys hash to the same location, then a
single secondary hash table with at least n2 locations will be used. In order
CHAPTER 7. STORAGE/RETRIEVAL II: UNORDERED KEYS 280

to guarantee linear space usage in the worst case, we therefore need to select
primary hash functions repeatedly until we get one that yields a reasonable
total space usage. Because the space usage is linear in sumsq, we don’t need
to construct the actual secondary hash tables in order to determine whether
the space usage is reasonable — we can instead simply compute sumsq. We
should therefore determine some maximum acceptable value for sumsq.
In order to ensure a reasonable probability of success, we don’t want
this maximum value to be too small. From Markov’s Inequality (5.3), the
probability that a discrete random variable is at least twice its expected
value is at most 1/2, provided its expected value is strictly positive. Based
on (7.6) above, because we will be using a primary table size of at least
n, it makes sense to use 2(c + 1)n as the maximum allowable value for
sumsq. Furthermore, our derivations have assumed that c < 2; hence, we
can simplify the maximum allowable value to 6n. By using this maximum,
we would expect to select no more than 2 primary hash functions, on average,
and still guarantee linear space usage.
We represent an ImmutableDictionary with the following variables:

• table[0..m−1]: an array of (possibly nil) arrays of (possibly nil) Keyed

elements;

• hash: a HashFunction;

• functions[0..m − 1]: an array of (possibly nil) HashFunctions; and

• size: a readable Nat.

Our structural invariant is that:

• the size of hash is the number of locations m in table;

• for 0 ≤ i < m, table[i] is nil iff functions[i] is nil;

• if table[i] 6= nil, then the array stored there is indexed 0..s − 1, where
s is the size of functions[i];

• if an element with key k is stored at table[i][j], then hash.Index(k) = i

and functions[i].Index(k) = j; and

• size = n, the total number of keys stored.

We interpret the Keyed items stored as the elements of the Immutable-

Dictionary, together with their associated keys.
CHAPTER 7. STORAGE/RETRIEVAL II: UNORDERED KEYS 281

The implementation of the constructor is shown in Figure 7.12. Based

on the above discussion, the first repeat loop is expected to iterate no more
than twice. Furthermore, it is easily seen that each iteration of this loop runs
in Θ(nf (l) + g(n)) time in the worst case, where g(n) is the time required
by the HashFunction constructor, and f (l) is the time required by the
Index operation on keys of length l. In order to simplify the discussion that
follows, we will assume that the HashFunction constructor runs in O(n)
time; consequently, the expected running time of this loop is in Θ(nf (l)).
The effect of this loop is to create a HashFunction hash and an array
t[0..hash.Size() − 1] that is essentially an ordinary hash table for hash.
The analysis of the remainder of the algorithm is somewhat more in-
volved, but again relies heavily on the above discussion. The outer loop
constructs a secondary hash table for each t[i] and places it in table[i]. Each
iteration of the repeat loop generates a hash function and attempts to con-
struct a secondary hash table for the elements at t[i]. It iterates until it
has a secondary hash table with no collisions. By the above discussion, the
expected number of iterations is in Θ(1). Let ni be the number of elements
at t[i]. Then the for loop iterates n2i times, and the while loop iterates ni
times in the worst case. Therefore, the repeat loop requires Θ(n2i + ni f (l))
expected time.
The expected running time of the entire for loop is then in
m−1
X m−1
X m−1
X
Θ(n2i + ni f (l)) = Θ(n2i ) + Θ(ni f (l))
i=0 i=0 i=0
m−1
! m−1
!
X X
=Θ n2i + Θ f (l) ni
i=0 i=0
= Θ((c + 1)n) + Θ(nf (l))
= Θ(nf (l)).
The total expected running time of the constructor is therefore in Θ(nf (l)).
Thus, for Hp,m 2 , the constructor runs in Θ(n) expected time, and for
4
Hp,l,m , the constructor runs in Θ(nl) expected time. It is not hard to show
1 as well; the details
that the constructor runs in Θ(nl) expected time for Hl,m
are left as an exercise.
It is easily seen that the space required by the ImmutableDictionary
is in Θ(n) in the worst case. However, the hidden constants can be rather
large. Specifically, the primary hash table can contain nearly 3n locations,
and the secondary hash tables can contain a total of nearly 18n locations.
As a result, nearly 21n array locations can be used to store n data items.
CHAPTER 7. STORAGE/RETRIEVAL II: UNORDERED KEYS 282

Figure 7.12 Constructor for PerfectHash implementation of Im-

mutableDictionary

PerfectHash(elements[0..n − 1], keys[0..n − 1])

size ← n
repeat
hash ← new HashFunction(n); m ← hash.Size()
t ← new Array[0..m − 1]; count ← new Array[0..m − 1]
for i ← 0 to m − 1
t[i] ← new ConsList(); count[i] ← 0
for i ← 0 to n − 1
h ← hash.Index(keys[i])
t[h] ← new ConsList(new Keyed(elements[i], keys[i]), t[h])
count[h] ← count[h] + 1
sum ← 0
for i ← 0 to m − 1
sum ← sum + count[i]2
until sum ≤ 6n
table ← new Array[0..m − 1]; functions ← new Array[0..m − 1]
for i ← 0 to m − 1
if t[i].IsEmpty()
table[i] = nil; functions[i] = nil
else
repeat
functions[i] = new HashFunction(count[i]2 )
table[i] = new Array[0..functions[i].Size() − 1]
for j ← 0 to SizeOf(table[i]) − 1
table[i][j] ← nil
L ← t[i]; coll = false
while not L.IsEmpty() and not coll
h ← functions[i].Index(L.Head().Key())
if table[i][h] = nil
table[i][h] ← L.Head(); L ← L.Tail()
else
if table[i][h].Key() = L.Head().Key()
error
else
coll ← true
until L.IsEmpty()
CHAPTER 7. STORAGE/RETRIEVAL II: UNORDERED KEYS 283

Figure 7.13 The PerfectHash.Get operation

PerfectHash.Get(k)
h ← hash.Index(k)
if table[h] = nil
return nil
else
return table[h][functions[h].Index(k)]

1 will return a hash

However, observe that the constructor for the family Hl,m
2
function with size less than 2n, and the constructors for the families Hp,m
4
and Hp,l,m both return hash functions with size n. Furthermore, if we were
to fix a specific c-universal family of hash functions, we could reduce the
bound on the first repeat loop to 2(c + 1)n.
Combining the above results, we see that the worst-case total number of
array locations can be reduced to:
1 ;
• 10n for Hl,m
2 ; or
• 5n for Hp,m
4
• 21n/4 for Hp,l,m .

Finally, we observe that because E[sumsq] < (c + 1)n, the expected total
number of array locations is no more than
1 ;
• 6n for Hl,m
2 ; or
• 3n for Hp,m
4
• 25n/8 for Hp,l,m .

These last bounds hold regardless of whether we change the bound on the
first repeat loop.
The Get operation is shown in Figure 7.13. It clearly runs in Θ(f (l))
time, where f (l) is the time needed to compute the hash function on a key
of length l.
CHAPTER 7. STORAGE/RETRIEVAL II: UNORDERED KEYS 284

7.7 Summary
If keys are natural numbers, we can implement Dictionary using a VAr-
ray and thus achieve constant-time accesses in the worst case. However,
the space usage of a VArray makes it impractical. For this reason, hash
tables are the preferred implementation in practice. Furthermore, hashing
can be done for arbitrary types of keys.
Deterministic hashing yields data accesses that, in practice, run in amor-
tized time proportional to the length of the key, independent of the number
of data items in the set. This compares very well to the structures pre-
sented in Chapter 6, which give Θ(lg n) access times, where n is the number
of data items in the set. In our analyses in Chapter 6, we did not consider
the key length. Our analyses thus implicitly assumed that keys could be
compared in constant time. Each of the structures in Chapter 6 require
Θ(lg n) comparisons in either the worst, amortized, or expected case, de-
pending on the structure. In the worst case, each of these comparisons
requires a time proportional to the length of the key. As a result, the per-
formance of deterministic hashing is usually significantly better in practice
than those structures given in Chapter 6. The trade-off is that hash tables
do not permit fast access to all of the keys in a predetermined order.
The division method, which computes the value of the key mod the ta-
ble size, is the most common type of hash function. In order for this method
to work well, the table size should be a prime number that is not too close to
a power of 2. The division method is often combined with polynomial hash-
ing in order to produce a single-word index, which can then be converted
to locations in tables of different sizes. Polynomial hashing involves multi-
plying each component of the key by a radix raised to successively higher
powers, retaining only those bits that will fit in a single machine word. The
radix r should use 5 to 9 bits, and should be such that r mod 8 is either 3
or 5.
Though it works very well in practice, in the worst case, deterministic
hashing results in accesses having a running time in Θ(n). We can achieve
better theoretical results using universal hashing, in which a hash function
is selected at random from a universal family of hash functions. When
universal hashing is used, data accesses have an expected amortized running
time proportional to the key length.
Perfect hashing is an application of universal hashing which produces
an ImmutableDictionary. Using the inherent randomization in universal
hashing, we can construct an ImmutableDictionary in expected time
linear in the sum of the key lengths. Retrievals can then be performed by
CHAPTER 7. STORAGE/RETRIEVAL II: UNORDERED KEYS 285

computing two hash functions — no searching is required. However, if the

keys are long, the cost of computing a second hash function may exceed the
cost of searching for the key in an ordinary hash table.

7.8 Exercises
Exercise 7.1 Prove that VArray, shown in Figure 7.2, meets its specifi-
cation.

Exercise 7.2 Give an algorithm that takes as input an array A[1..n] of

natural numbers and returns an array B[1..n] such that for 1 ≤ i ≤ n, B[i]
gives the last location in A that contains A[i]. Your algorithm must run in
O(n) time in the worst case, and you may make no assumptions about how
large the elements in A are. Prove the correctness and time complexity of
your algorithm. [Hint: Use a VArray.]

Exercise 7.3 Complete the implementation of HashTable shown in Fig-

ures 7.5 (p. 252) and 7.6 (p. 255) by adding a Remove operation as specified
in Figure 6.2 (p. 196).

Exercise 7.4 Prove that if the cost of rehashing, as implemented in Figure

7.6 (p. 255), is amortized over all Put and Remove operations, the amor-
tized cost of rehashing is proportional to the cost of computing the index
for a single key.

Exercise 7.5 Prove the following for all integers x and y and all positive
integers m:
a. (x + (y mod m)) mod m = (x + y) mod m.
b. (x(y mod m)) mod m = (xy) mod m.
c. (−(x mod m)) mod m = (−x) mod m.

Exercise 7.6 Show the hash table that results from inserting the following
keys in the order listed, assuming the division method is used with a table
of size 13:
27, 36, 14, 40, 42, 15, 25, 2.
You may assume that no rehashing is done. How does the number of colli-
sions, as defined by the random variable coll in Section 7.2, compare with
the expected number, assuming that distinct keys collide with probability
1/13?
CHAPTER 7. STORAGE/RETRIEVAL II: UNORDERED KEYS 286

Exercise 7.7 Give a modified version of HashTable.Put(x, k) (see Fig-

ure 7.6) which uses a compression map, as described in Section 7.3. Use
the HashFunction ADT to represent both hash functions (i.e., both the
compression map and the function to compute the index in the hash table).
You may need to define an additional data structure in order to save results
of the compression map for use in rehashing (you need to be able to retrieve
these values quickly).

Exercise 7.8 Give an implementation of HashFunction that uses poly-

nomial hashing, as described in Section 7.3. You may assume that the
variable r contains an appropriate value to use as the radix, and that the
variable w contains the number of bits in a machine word. The size should
be the smallest power of 2 that is no smaller than the parameter given to the
constructor. For clarity, your implementation should include the mod oper-
ation rather than relying on overflow (we don’t assume any explicit bounds
on integer variables in our algorithms).

* Exercise 7.9 Prove that for each h ∈ Hl,m 1 there is exactly one s ∈ Sl,m
such that hs = h. [Hint: Prove that if s 6= s′ , then hs 6= hs′ . In order to do
this, it is sufficient to find a k such that hs (k) 6= hs′ (k).]

* Exercise 7.10 Modify UniversalHash1 (Figure 7.8) to handle varying-

length keys of unbounded length. Use the expandable-array design pattern
to store the randomly-generated indices. Show that when this implementa-
tion is used with HashTable (shown in Figures 7.5 and 7.6), the amortized
expected running time of the Dictionary operations is in O(l), where l is
the number of bits in the longest key in the table. You may assume that
the actual running time of Remove(k) is proportional to the running time
of hash.Index(k) plus the length of the ConsList at the resulting index.

* Exercise 7.11 Complete the proof of Theorem 7.4.

Exercise 7.12 Implement HashFunction to provide Hp,m 2 . You may as-

sume the variable p contains a prime number larger than any key. You may
also assume that all values will fit into integer variables.

Exercise 7.13 Implement HashFunction to provide Hp,l 3 . You may as-

sume the variable p contains a prime number larger than w bits, where w
is another variable. You may also assume that if a, b, and c are all natural
numbers less than p, then ab + c will fit in an integer variable; however, you
CHAPTER 7. STORAGE/RETRIEVAL II: UNORDERED KEYS 287

may not assume that arbitrarily many of these values added together will
fit.

* Exercise 7.14 Suppose we were to modify the definition of Hp,l 3 (7.2) so

that for each ai , 1 ≤ ai < p. Show that for every l ≥ 2 and prime number
p, the resulting family of hash functions is not universal. Specifically, show
that there are two distinct keys that collide with probability strictly greater
than 1/p. [Hint: First consider l = 2, then generalize.]
4
Exercise 7.15 Implement HashFunction to provide Hp,l,m using the same
assumptions as for Exercise 7.13.

Exercise 7.16 Let p be a prime number and l and m be positive integers

such that m < p.
* a. Prove that for every h ∈ Hp,l,m 4 and every positive integer a < p, there
is exactly one choice of natural numbers b, a1 , . . . , al less than p such
that !
Xl
h(hk1 , . . . , kl i) = a ai ki + b mod p mod m
i=1
for every l-tuple hk1 , . . . , kl i of natural numbers less than p.
b. Consider the following two methods of randomly selecting a hash func-
tion h of the form given by equation (7.3):
i. Select a with uniform probability from the positive integers less
than p, and select b, a1 , . . . , al independently with uniform prob-
ability from the natural numbers less than p.
ii. Set a to 1, and select b, a1 , . . . , al independently with uniform
probability from the natural numbers less than p.
4
Prove that for any h ∈ Hp,l,m , h is chosen with the same probability
by both methods.

* Exercise 7.17 In terms of c and n, find the value of m ∈ R≥0 that

minimizes
cn(n − 1)
+ n + m,
m
assuming c ∈ R>0 , n ∈ N, and n > 1.

Exercise 7.18 Prove that the constructor for PerfectHash runs in Θ(nl)
1 is used as the universal hash family, where m is a power
expected time if Hl,m
of 2.
CHAPTER 7. STORAGE/RETRIEVAL II: UNORDERED KEYS 288

7.9 Chapter Notes

Virtual initialization was suggested by Aho, Hopcroft, and Ullman [2, Ex-
ercise 2.12].
The first description of hashing in the literature was by Dumey [31],
who also introduced the division method. However, the concept appears
to have been discovered a few years earlier at IBM by H. P. Luhn and
independently by Gene M. Amdahl, Elaine M. Boehme, N. Rochester, and
Arthur L. Samuel. Knuth [80] gives a detailed treatment of deterministic
hashing.
Universal hashing was introduced by Carter and Wegman [20]. They
1
presented the universal families Hl,m 2 . The notion of a c-universal
and Hp,m
family is closely related to the notion of an ǫ-universal family defined by
Cormen, et al. [25].
The perfect hashing strategy given in Section 7.6 is due to Fredman,
Komlós, and Szermerédi [43].
Chapter 8

Disjoint Sets

In order to motivate the topic of this chapter, let us consider the following
problem. We want to design an algorithm to schedule a set of jobs on a
single server. Each job requires one unit of execution time and has its own
deadline. We must assign a job with deadline d to some time slot t, where
1 ≤ t ≤ d. Furthermore, no two jobs can be assigned to the same time slot.
If we can’t find a time slot for some jobs, we simply won’t schedule them.
One way to construct such a schedule is to assign each job in turn to the
latest available time slot prior to its deadline, provided there is such a time
slot. The challenge here is to find an efficient way of locating the latest
available time slot prior to the deadline.
One way to think about this problem is to partition the time slots into
disjoint sets — i.e., a collection of sets such that no two sets have any
element in common. In this case, each set will contain a nonempty range of
time slots such that the first has not been assigned to a job, but all the rest
have been assigned to jobs. In order to be able to handle the case in which
time slot 1 has been assigned a job, we will also include a time slot 0, which
we will consider to be always available.
Suppose, for example, that we have scheduled jobs in time slots 1, 2,
5, 7, and 8. Each set must have a single available time slot, which must
be the smallest time slot in that set; thus, the elements 0, 3, 4, 6, and
all elements greater than 8 must be in different sets and must each be the
smallest element of its set. If 10 is the latest deadline, our disjoint sets will
therefore be {0, 1, 2}, {3}, {4, 5}, {6, 7, 8}, {9}, and {10}. If we then wish
to schedule a job with deadline 8, we need to find the latest available time
slot prior to 8. This is simply the first time slot in the set containing 8
— namely, 6. Thus, in order to find this time slot, we need to be able to

289
CHAPTER 8. DISJOINT SETS 290

determine which set contains the deadline 8, and what is the first time slot
in that set. When we then schedule the job at time slot 6, the set {6, 7, 8}
no longer contains an available time slot. We therefore need to merge the
set {6, 7, 8} with the set containing 5, namely, {4, 5}.
The operations of finding the set containing a given element and merging
two sets are typical of many algorithms that manipulate disjoint sets. The
operation of finding the smallest element of a given set is not as commonly
needed, so we will ignore this operation for now; however, as we will see
shortly, it is not hard to use an array to keep track of this information.
Furthermore, we often need to manipulate objects other than Nats; how-
ever, we can always store these objects in an array and use their indices as
the elements of the disjoint sets. For this reason, we will simplify matters
by assuming that the elements of the disjoint sets are the Nats 0..n − 1.
In general, the individual sets will be allowed to contain non-consecutive
integers.
The DisjointSets ADT, shown in Figure 8.1, specifies the data struc-
ture we need. Each of the sets contains an element that is distinguished
as its representative. The Find operation simply returns that represen-
tative. Thus, if two calls to Find return the same result, we know that
both elements belong to the same set. The Merge operation takes two
representatives, combines the sets identified by these elements, and returns
the resulting set’s representative. In this chapter, we will consider how the
DisjointSets ADT can be implemented efficiently. Before we do this, how-
ever, let us take a closer look at how the DisjointSets ADT can be used
to implement the scheduling algorithm outlined above.

8.1 Using DisjointSets in Scheduling

We will use an instance of the DisjointSets ADT to maintain the disjoint
sets in the scheduling algorithm outlined above. In order to find the time slot
in which to schedule a job, we first need to find the time interval containing
the last time slot prior to the job’s deadline. We can use the Find operation
for this purpose. Assuming that we can obtain the available time slot in a
given partition, and assuming this time slot is not 0, we have the time slot
i in which to schedule the job. We then need to combine this partition with
the one immediately preceding it. We can find the preceding partition with
Find(i − 1). We can then combine the two partitions using Merge.
We need one additional data structure in order to be able to find the
available slot in an interval, given the representative of that interval. For this
CHAPTER 8. DISJOINT SETS 291

Figure 8.1 The DisjointSets ADT

Precondition: n is a positive Nat.

Postcondition: Constructs a DisjointSets object in which each element
in 0..n − 1 forms a singleton set and is the representative of that set.
DisjointSets(n)
Precondition: k is a Nat less than the size of the universe of elements.
Postcondition: Returns the representative of the partition containing k.
DisjointSets.Find(k)
Precondition: i and j are representatives of two different partitions.
Postcondition: Merges the partitions containing i and j into a new parti-
tion and returns the representative of the new partition.
DisjointSets.Merge(i, j)

purpose, we can use an array avail[0..n] such that if j is the representative of

an interval, then avail[j] is the available time slot in that interval. Initially,
avail[i] = i for all i. Suppose we schedule a job at time i. We then merge the
interval containing i with the preceding interval. Let j be the representative
of the preceding interval prior to the merger. Then avail[j] is the available
element in the resulting interval. If k is the value returned by the call to
Merge, then we can update avail by assigning to avail[k] the value avail[j].
Because no other representatives change, no other updates are needed. The
entire algorithm is shown in Figure 8.2.

8.2 A Tree-Based Implementation

In this section, we will consider a tree-based implementation of the Dis-
jointSets ADT, as illustrated in Figure 8.3. Each partition will be repre-
sented by a tree. The nodes of the tree will be the elements of the partition.
The element at the root of the tree will be the representative of that parti-
tion. Because we will need to find the root from an arbitrary node in the
tree, the children will maintain references to their parents, rather than vice
versa. Note that because parents do not need to reference children, a node
can have arbitrarily many children.
Given a value k, we need to be able to find the parent of the node
CHAPTER 8. DISJOINT SETS 292

Figure 8.2 Scheduling algorithm using DisjointSets

Precondition: deadlines[1..m] contains positive Nats no larger than n.

Postcondition: Returns an array sched[1..n] of Nats no larger than m
such that if sched[i] > 0, then i ≤ deadlines[sched[i]].
Schedule(deadlines[1..m], n)
sched ← new Array[1..n]; avail ← new Array[0..n]
intervals ← new DisjointSets(n + 1); avail[0] ← 0
// Invariant: For 1 ≤ j < i, sched[j] = 0 and avail[j] = j.
for i ← 1 to n
sched[i] ← 0; avail[i] ← i
// Invariant: For 1 ≤ s ≤ n, sched[s] ≤ m and if sched[s] > 0, then
// s ≤ deadlines[sched[s]]. Each partition in intervals contains a range
// of natural numbers u, u + 1, . . . u + d such that sched[u] = 0,
// sched[u + v] > 0 for 1 ≤ v ≤ d, and for the representative w of this
// partition, avail[w] = u.
for i ← 1 to m
k ← intervals.Find(deadlines[i]); t ← avail[k]
if t 6= 0
sched[t] ← i; j ← intervals.Find(t − 1)
k ← intervals.Merge(j, k); avail[k] ← avail[j]
return sched

representing k. In order to accommodate this functionality, we will use an

array parent[0..n − 1] to represent the trees. Specifically, parent[k] will give
the parent of k in its tree, or if k is the root, parent[k] will be k. Thus,
parent will be the only representation variable. Our structural invariant
will be that for 0 ≤ i < n:

• 0 ≤ parent[i] < n; and

• there is a finite sequence parent[i], parent[parent[i]], ..., k such that

parent[k] = k.

Note that this invariant implies that the values in the universe are grouped
into trees, where the root k of a tree is denoted by parent[k] = k.
The Merge and Find operations are now straightforward. Merge sim-
ply makes one tree a child of the root of the other, and Find follows the
CHAPTER 8. DISJOINT SETS 293

Figure 8.3 A tree-based implementation of DisjointSets.

2 3 6

0 1 4 5

7 8

parent references until the root is reached. The full implementation is shown
in Figure 8.4.
Clearly, the constructor operates in Θ(n) time, and Merge operates in
Θ(1) time. The number of iterations of the while loop in Find is the depth
of k, which in the worst case is the height of the tree. Clearly, the height is
at most n − 1. Unfortunately, this height can be achieved by the sequence

Merge(0, 1), Merge(1, 2), . . . , Merge(n − 2, n − 1).

Thus, the worst-case running time for Find is in Θ(n).

Because the constructor will only be executed once for each structure,
Θ(n) is not a bad running time. However, we will probably need to execute
Find repeatedly. We would therefore like to improve its performance. In
the next section we will examine a simple way to do this.

8.3 A Short Tree Implementation

Because the worst-case running time for Find is proportional to the height of
the tree, we can improve the worst-case performance by controlling heights
of the trees. In order to accomplish this, when we merge two trees, we will
always make the tree with smaller height the child of the root of the other
tree. If both trees have the same height, we arbitrarily choose one as the
child. Note that by using this technique, the only way we can increase the
height of a tree is to merge it with another tree of the same height. We will
show that as a result, the heights of the trees are always at most logarithmic
in their number of nodes.
CHAPTER 8. DISJOINT SETS 294

Figure 8.4 TreeDisjointSets implementation of DisjointSets

Structural Invariant: For 0 ≤ i < n, 0 ≤ parent[i] < n and there is a

finite sequence parent[i], parent[parent[i]], ..., k such that parent[k] = k.

TreeDisjointSets(n)
parent ← new Array[0..n − 1]
// Invariant: For 0 ≤ j < i, parent[j] = j.
for i ← 0 to n − 1
parent[i] ← i

TreeDisjointSets.Find(k)
i←k
// Invariant: i is an ancestor of k.
while parent[i] 6= i
i ← parent[i]
return i

TreeDisjointSets.Merge(i, j)
if i = j or i 6= parent[i] or j 6= parent[j]
error
else
parent[i] ← j
return j

As we did in analyzing the heights of AVL trees (Section 6.2), let us

compute the minimum number of nodes required to achieve a tree of height
h, assuming we always make the tree with smaller height the child. Let f (h)
give this number. Then f (0) = 1. In order to build a tree with height h > 0
using the fewest nodes, we must merge two trees of height h − 1, each having
the fewest nodes possible. Thus, each of the two merged trees must have
f (h − 1) nodes. The total number of nodes is given by the recurrence

f (h) = 2f (h − 1).

It is easily seen that f (h) = 2h , so that h = lg f (h). Thus, if k is the

CHAPTER 8. DISJOINT SETS 295

number of nodes in a tree of height h, we have

h = lg f (h)
≤ lg k.

We conclude that if the universe contains n elements, then no tree has a

height greater than lg n.
In order to be able to merge trees in this way, we need to keep track
of the height of each tree. For this purpose, we include an additional
representation variable height[0..n − 1]. Our structural invariant will then
be that for 0 ≤ i < n, height[i] is the maximum of 0 and height[j] + 1 for all
j 6= i such that parent[j] = i, and that 0 ≤ parent[i] < n. Note that because
height[parent[i]] > height[i] whenever parent[i] 6= i, each value i must have
an ancestor j such that parent[j] = j; thus, the elements of the universe
must be grouped into trees rooted at nodes with parent[i] = i.
The constructor and Merge operation for this implementation are shown
in Figure 8.5. The Find operation is implemented exactly as TreeDis-
jointSets.Find in Figure 8.4.
As in the previous implementation, the constructor runs in Θ(n) time,
and Merge runs in Θ(1) time. Based on the above discussion, Find runs
in Θ(lg n) time.

8.4 * Path Compression

In order to improve the performance of a Find, we would like to decrease
the distance from a node to the root as much as possible. An effective way
of accomplishing this is to modify the Find so that it changes the parent of
every node it encounters to be the root of its tree. This technique is called
path compression. We implement this technique using a recursive internal
function, Compress. If the given node k is not the root, Compress first An internal
function is used
performs a recursive Compress on k’s parent. If we treat the distance in order to avoid
between a node and the root as the size of a call to Find, we see that making a
such a recursive call is valid. Furthermore, it compresses the path from k’s callback.

parent to the root and returns the root. It can therefore complete its task
by making k a child of the root and returning the root. The resulting Find
algorithm is shown in Figure 8.6.
The rest of the implementation is the same as for ShortDisjointSets;
however, because path compression can decrease the height of a node i
without updating height[i], this value is no longer guaranteed to give the
height of node i. For this reason, we will change the name of this array
CHAPTER 8. DISJOINT SETS 296

Figure 8.5 ShortDisjointSets implementation of DisjointSets

Structural Invariant: For 0 ≤ i < n, height[i] is the maximum of 0 and

height[j] + 1 for all j 6= i such that parent[j] = i, and 0 ≤ parent[i] < n.

ShortDisjointSets(n)
parent ← new Array[0..n − 1]; height ← new Array[0..n − 1]
for i ← 0 to n − 1
parent[i] ← i; height[i] ← 0

ShortDisjointSets.Merge(i, j)
if i = j or i 6= parent[i] or j 6= parent[j]
error
else if height[i] ≤ height[j]
parent[i] ← j
if height[i] = height[j]
height[j] ← height[j] + 1
return j
else
parent[j] ← i
return i

to rank and weaken the structural invariant so that rank[i] is at least the
height of i. The precise meaning of rank[i] is now rather elusive, other than
the fact that it gives an upper bound on the height of i.
Clearly, the running time of CompressedDisjointSets.Find is in O(h),
where h is the height of the tree. Furthermore, the height of a tree in
this implementation can certainly be no larger than it would have been
if no path compression had been done; hence, the upper bound of O(lg n)
shown for ShortDisjointSets.Find also holds for CompressedDisjoint-
Sets.Find. Because we can still construct a tree with height in Θ(lg n) with
a sequence of Merges, we conclude that CompressedDisjointSets.Find
runs in Θ(lg n) time in the worst case. Clearly, CompressedDisjoint-
Sets.Merge runs in Θ(1) time, so that in the worst case, the asymptotic
performance of CompressedDisjointSets is identical to that of Short-
DisjointSets.
On the other hand, because each path compression has the tendency
CHAPTER 8. DISJOINT SETS 297

Figure 8.6 The Find algorithm for the CompressedDisjointSets imple-

mentation of DisjointSets

Structural Invariant: For 0 ≤ i < n, 0 ≤ parent[i] < n and there is a

finite sequence parent[i], parent[parent[i]], ..., k such that parent[k] = k.

CompressedDisjointSets.Find(k)
return Compress(k)

// Internal function
Precondition: k is a Nat less than the size of the universe of elements,
and the structural invariant holds.
Postcondition: Returns the representative of the partition containing k,
and makes each other element that was an ancestor of k a child of the
representative. The structural invariant holds.
CompressedDisjointSets.Compress(k)
if parent[k] 6= k
parent[k] ← Compress(parent[k])
return parent[k]

to improve the performance of subsequent Finds, we might suspect that

the amortized performance of this structure is improved. Indeed, we will
show that the amortized performance of the Merge and Find operations
for CompressedDisjointSets is “almost” constant.
In order to perform an amortized analysis, we need to assign actual
costs to the operations. The running time of Find(i) is proportional to the
number of nodes on the path from i to the root of its tree. Another way to
say this is that the running time is proportional to the number of locations
of the parent array that are accessed. Let us therefore define the actual cost
of an operation to be the number of locations accessed in the parent array.
The actual cost of Merge is therefore 2.
The key to a good amortized analysis is finding a good potential function.
Let S be the set of all possible states of some CompressedDisjointSets
with size n. We need a function Φ : S → R≥0 such that the initial state
is mapped to 0. To arrive at this function, we will define, for a given state
s ∈ S, a potential for each node; i.e., we will define a function φs : U → N,
CHAPTER 8. DISJOINT SETS 298

where U = {i ∈ N | i < n}. We will then define the potential function to be

the sum of the potentials of all of the nodes:
n−1
X
Φ(s) = φs (i).
i=0

Let rank s and parents denote the values of the rank and parent arrays
in state s. We will define our potential function based on these values. Let
ǫ denote the initial state. Thus, for 0 ≤ i < n, rank ǫ [i] = 0. In order for
Φ to be a valid potential function, we need Φ(ǫ) = 0. To accomplish this,
we let φs (i) = 0 if rank s [i] = 0 for 0 ≤ i < n and any state s. Note that a
node can only obtain a nonzero rank when a Merge makes it the parent of
another node; thus, rank s [i] = 0 iff i is a leaf.
We have two operations we need to consider as we define φs (i) for non-
leaf i. Merge is a cheap operation, having an actual cost of 2, whereas
Find is more expensive in the worst case. We therefore need to amortize
the cost of an expensive Find over preceding Merges. This means that
we need the potential function to increase by some amount — say α(n),
where α is some appropriate function — for at least some of the Merges.
The Merge operation only operates on roots, so let us focus our attention
there. When a Merge is performed, the rank of one node — a root — may
increase by 1, but otherwise, no ranks increase (see Figure 8.5, and recall
that the rank array replaces the height array in this algorithm). Therefore,
let us define φs (i) to be α(n)rank s [i] if i is a root (i.e., if parents [i] = i).
Note that the above definitions are consistent with each other: if i is both
a leaf and a root, its rank is 0, and hence its potential is 0 by either of the
above definitions. Furthermore, if we can ensure that a Merge causes the
potential of no node other than the root of the resulting tree to increase, we
will have a bound of α(n) + 2 on the amortized cost of Merge with respect
to Φ. We still need to define the potentials for nodes that are neither leaves
nor roots in such a way that an expensive Find causes Φ to decrease enough
to offset much of the actual cost.
Consider the effect of Find(j) on a node i that is neither a leaf nor a root.
i’s rank doesn’t change, but its parent may. In particular, it may receive a
parent with different rank than its original parent. It is not hard to show
as a structural invariant that if parent[i] 6= i, then rank[parent[i]] > rank[i].
We would therefore like the potential of i to decrease, generally speaking,
as rank[parent[i]] increases. Furthermore, in order that its potential does
not increase when a Merge changes it from a root to a non-root, we should
have φs (i) ≤ α(n)rank s [i] for all states s and nodes i.
CHAPTER 8. DISJOINT SETS 299

As a first approximation to a definition of φs (i), where i is neither a leaf

nor a root, suppose we let φs (i) = (α(n) − f (s, i))rank s [i], where f is some
function that depends on the ranks of i and its parent in s and obeys the
constraint
0 ≤ f (s, i) < α(n).
If f does not decrease when i receives a parent of higher rank, φs would
satisfy the constraints outlined above. However, it would suffer the disad-
vantage that its value is always a multiple of rank s [i]. In order to give us
more control over how much the potential function changes, we would like
for this function to have a larger range when rank s [i] is fixed, as it is when
i is not a root. Therefore, we wish to find, for each state s, a function

φs (i) = (α(n) − f (s, i))rank s [i] − g(s, i),

where f is as above and g is some function that depends on the ranks of i

and its parent in s and obeys the constraint

0 < g(s, i) ≤ rank s [i].

Let us consider the nodes examined during the operation Find(j) on

some state s. These nodes are the ancestors of j. Let us restrict our attention
to those ancestors that have nonzero rank and are not the root. (Note that
as a result, we ignore at most two ancestors.) Let Ak be a function describing
the relationship between the ranks of i and its parent for any non-root i,
where k = f (s, i); i.e., rank s [parents [i]] ≥ Af (s,i) (rank s [i]). When the path
compression is done, the parent of i may have the same rank (if i had already
been a child of the root), or it may be very little more than the rank of i’s
original parent. However, suppose that in the original state s, i has a proper
ancestor i′ other than the root such that f (s, i′ ) = f (s, i) = k. Because i′
is not a root, the rank of i’s new parent is at least the rank of the original
parent of i′ . Therefore, if Ak is nondecreasing, we have

rank s′ [parents′ [i]] ≥ rank s [parents [i′ ]]

≥ Ak (rank s [i′ ])
≥ Ak (rank s [parents [i]]).

Thus, each time a path compression moves a non-leaf i having a proper

ancestor i′ (other than the root) with f (s, i) = f (s, i′ ) = k, the rank of i’s
parent increases by at least the application of the function Ak . Furthermore,
because 0 ≤ f (s, i) < α(n), there can be at most α(n) nodes i on the path
CHAPTER 8. DISJOINT SETS 300

from j to the root, other than leaves or the root, that do not have a proper
ancestor i′ with f (s, i′ ) = f (s, i). Thus, if we can decrease the potential of
each i having a proper ancestor i′ with f (s, i′ ) = f (s, i), without increasing
any potentials, then we will have a bound of α(n) + 2 on the amortized cost
of a Find with respect to Φ.
The behavior described above gives us some insight into how we might
define f , g, and each Ak . First, we would like g to give the maximum
number of times Af (s,i) can be applied to the rank of i without exceeding
the rank of i’s parent. Thus, if i has a proper ancestor i′ with f (s, i′ ) =
f (s, i), and if f (s′ , i) = f (s, i), then g(s′ , i) > g(s, i). As a result, the
potential of i decreases. In order to keep g(s, i) within the proper range, we
should define f (s, i) and Ak so that if we apply Af (s,i) more than rank s (i)
times to rank s (i), we must attain a value of at least Af (s,i)+1 (rank s [i]).
Then if we define f (s, i) to be the maximum k such that rank s [parents [i]] ≥
Ak (rank s [i]), we can never have rank s [parents [i]] ≥ Af (s,i)+1 (rank s [i]).
We still need to define the functions Ak . In order to facilitate this def-
inition, we first define the iteration operator for functions. Let F : N → N.
We then define

F (0) (n) = n
F (k) (n) = F (F (k−1) (n)) for k > 0.

For example, if F (n) = 2n, then F (2) (n) = 4n and F (3) (n) = 8n; more
generally, F (k) (n) = 2k n.
We now define:
(
n+1 if k = 0
Ak (n) = (n+1)
Ak−1 (n) if k ≥ 1.

We can then define, for each node i that is neither a leaf nor a root,

f (s, i) = max{k | rank s [parents [i]] ≥ Ak (rank s [i])}

and
(k)
g(s, i) = max{k | rank s [parents [i]] ≥ Af (s,i) (rank s [i])}.
Finally, we need f (s, i) < α(n) whenever i is neither a leaf nor a root.
Thus, we need to ensure that whenever i is neither a leaf nor a root, we have

Aα(n) (rank s [i]) > rank s [parents [i]].

CHAPTER 8. DISJOINT SETS 301

We have shown that without path compression, the height of a tree never
exceeds lg n; hence, with path compression, the rank of a node never exceeds
lg n. It therefore suffices to define
α(n) = min{k | Ak (1) > lg n}.
As the subscript k increases, Ak (1) increases very rapidly. We leave it
as an exercise to show that
2
··
2·
A4 (1) ≥ 2 ,
where there are 2051 2s on the right-hand side. It is hard to comprehend
just how large this value is, for if the right-hand side contained only six 2s,
the number of bits required to store it would be 265536 + 1. By contrast, the
number of elementary particles in the universe is currently estimated to be
no more than about 2300 . Hence, there is not nearly enough matter in the
universe to store A4 (1) in binary. Because α(n) ≤ 4 for all n < 2A4 (1) , we
can see that α grows very slowly.
To summarize, we define our potential function Φ so that
n−1
X
Φ(s) = φs (i),
i=0

where

0
 if rank s [i] = 0
φs (i) = α(n)rank s [i] if parents [i] = i

(α(n) − f (s, i))rank s [i] − g(s, i) otherwise,


for α, f , and g as defined above. Before we can complete the amortized

analysis, we need to show that both f and g satisfy the properties outlined
in the discussion above.

Lemma 8.1 Let s be a state of a CompressedDisjointSets of size n,

and let 0 ≤ i < n such that parents [i] 6= i and rank s [i] > 0. Then
0 ≤ f (s, i) < α(n).

Proof: First, because the rank of the parent of i is strictly larger than that
of i, we have
rank s [parents [i]] ≥ rank s [i] + 1
= A0 (rank s [i]),
CHAPTER 8. DISJOINT SETS 302

from the definition of A0 . Thus, from the definition of f , f (s, i) ≥ 0.

It is not hard to show that each Ak is nondecreasing — we leave the
details as an exercise. Using this fact, along with the definition of α and the
fact that no rank exceeds lg n, we have

Aα(n) (rank s [i]) ≥ Aα(n) (1)

> lg n
≥ rank s [parents [i]].

Thus, from the definition of f , f (s, i) < α(n).

Lemma 8.2 Let s be a state of a CompressedDisjointSets of size n,

and let 0 ≤ i < n such that parents [i] 6= i and rank s [i] > 0. Then

0 < g(s, i) ≤ rank s [i].

Proof: First, from the definition of f , we have

rank s [parents [i]] ≥ Af (s,i) (rank s [i])

(1)
= Af (s,i) (rank s [i]).

Thus, from the definition of g, g(s, i) > 0.

Now from the definitions of A and f , we have
(rank [i]+1)
Af (s,i)s (rank s [i]) = Af (s,i)+1 (rank s [i])
> rank s [parents [i]].

Thus, g(s, i) ≤ rank s [i].

We are now ready to show that the amortized costs of Merge and Find
are in O(α(n)).

Theorem 8.3 With respect to Φ, the amortized cost of Merge on a Com-

pressedDisjointSets of size n is in O(α(n)).

Proof: Suppose we do Merge(i, j) in state s, yielding state s′ . Without

loss of generality, assume j is made the parent of i. Then i is the only
node whose parent changes, and j is the only node whose rank may change;
CHAPTER 8. DISJOINT SETS 303

hence, the potentials for all other nodes remain unchanged. The change in
potential for node i is given by

φs′ (i) − φs (i) = (α(n) − f (s′ , i))rank s′ [i] − g(s′ , i) − α(n)rank s [i]
< α(n)(rank s′ [i] − rank s [i])
= 0.

The change in potential for node j is given by

φs′ (j) − φs (j) = α(n)rank s′ [j] − α(n)rank s [j]

≤ α(n),

because the rank of j can increase by at most 1. The change in Φ is therefore

less than α(n). Because the actual cost is 2, the amortized cost is less than
α(n) + 2 ∈ O(α(n)).

Theorem 8.4 With respect to Φ, the amortized cost of Find on a Com-

pressedDisjointSets of size n is in O(α(n)).

Proof: Suppose we perform Find(j) on state s. Let s′ be the resulting

state. Suppose there are d nodes on the path from j to the root in s. Then
the actual cost of the operation is d. We will show that as a result of this
operation, at least d − α(n) − 2 nodes decrease in potential, and no nodes
increase in potential. As a result, we will have shown the amortized cost to
be at most α(n) + 2 ∈ O(α(n)).
First, we will show that no potentials increase as a result of Find(j).
Because the Find operation does not change any ranks and does not change
which nodes are roots, no leaves or roots can change potential. The potential
for any other node i can change only due to changes in f and g. Because i
cannot receive a parent with a smaller rank as a result of path compression,
f (s′ , i) ≥ f (s, i). If f (s′ , i) = f (s, i), then clearly g(s′ , i) ≥ g(s, i). In
this case, the potential does not increase. If, on the other hand, f (s′ , i) >
f (s, i), from Lemma 8.2 and the fact that path compression leaves all ranks
unchanged, g(s, i) − g(s′ , i) < rank s [i]. Then

φs′ (i) − φs (i) = (α(n) − f (s′ , i))rank s′ [i] − g(s′ , i) −

(α(n) − f (s, i))rank s [i] + g(s, i)
< (f (s, i) − f (s′ , i))rank s [i] + rank s [i]
≤ 0.
CHAPTER 8. DISJOINT SETS 304

In this case, the potential of i decreases.

The only nodes whose parents change are ancestors of j, and no ranks
change. Hence, the only nodes whose potentials change are ancestors of
j. The ancestors of j include a root and at most one leaf. For the other
ancestors i, from Lemma 8.1, there can be at most α(n) distinct values for
f (s, i). For a given value k, each node i with f (s, i) = k except the one
nearest the root has a proper ancestor i′ with f (s, i′ ) = k. We will show
that all of these nodes — i.e., at least d − α(n) − 2 of the d ancestors of j
— decrease in potential as a result of the Find.
Let i be an ancestor of j in s such that i is neither a leaf nor a root
and such that for some proper ancestor i′ of i other than the root, f (s, i) =
f (s, i′ ) = k. We have already shown that if f (s′ , i) > f (s, i), the potential
of i decreases. Therefore, suppose f (s′ , i) = f (s, i) = k. Then
rank s′ [parents′ [i]] ≥ rank s [parents [i′ ]]
≥ Ak (rank s [i′ ]) (definition of f )
≥ Ak (rank s [parents [i]])
(g(s,i))
≥ Ak (Ak (rank s [i])) (definition of g)
(g(s,i)+1)
= Ak (rank s [i])
(g(s,i)+1)
= Ak (rank s′ [i]).
Because f (s′ , i) = k, g(s′ , i) > g(s, i), so that φs′ (i) < φs (i).

The above theorems show that the amortized running times of Merge
and Find are in O(α(n)). However, α appears to be a somewhat contrived
function. We have argued intuitively that α increases very slowly, but we
have not formally compared it with any better-known slow-growing function
like lg or lg lg. We address this issue more formally in the Exercises. For
now, we will simply state that the collection of functions Ak form a variation
of Ackermann’s function, and that α is one way of defining its inverse. There
have actually been several different 2- or 3-variable functions that have been
called Ackermann’s function, and all grow at roughly the same rapid rate.

8.5 Summary
Tree-based implementations of disjoint sets provide very efficient Merge
and Find operations, particularly when path compression is used. The
worst-case running times for these operations are in Θ(1) and Θ(lg n), re-
spectively, for both ShortDisjointSets and CompressedDisjointSets.
CHAPTER 8. DISJOINT SETS 305

Figure 8.7 Comparison of running times of the DisjointSets operations

for various implementations

Find
TreeDisjointSets Θ(n)
ShortDisjointSets Θ(lg n)
CompressedDisjointSets O(α(n))
amortized

Notes:

• n is the number of elements in the universe of the sets.

• The constructor runs in Θ(n) worst-case time for each implementation.

• The Merge operation runs in Θ(1) worst-case time for each operation.

• Unless otherwise noted, all running times are worst-case.

The latter implementation yields nearly constant amortized running time.

A summary of the running times of the operations for the different imple-
mentations is shown in Figure 8.7. As we will see in later chapters, these
structures are very useful in the design of efficient algorithms.

8.6 Exercises
Exercise 8.1 Draw the trees that result from the following sequence of
operations: Corrected
4/12/12.

t ← new TreeDisjointSets(8)
t.Merge(0, 1)
t.Merge(t.Find(1), 2)
t.Merge(3, 4)
t.Merge(5, 6)
t.Merge(t.Find(3), t.Find(6))
t.Merge(t.Find(3), t.Find(0))

Exercise 8.2 Repeat Exercise 8.1 using a ShortDisjointSets implemen-

CHAPTER 8. DISJOINT SETS 306

tation.

Exercise 8.3 Repeat Exercise 8.1 using a CompressedDisjointSets im-

plementation.

Exercise 8.4 Prove that Schedule, shown in Figure 8.2, meets its speci-
fication.

Exercise 8.5 Prove that an algorithm that returns an array sched[1..n]

containing all 0s meets the specification of Schedule (Figure 8.2).

Exercise 8.6 Analyze the worst-case running time of Schedule (Figure

8.2) assuming the TreeDisjointSets implementation of DisjointSets.
Your analysis should be in terms of n, and you may assume that m ≤ n.
Express your result as simply as possible using Θ-notation.

Exercise 8.7 Repeat Exercise 8.6 assuming the ShortDisjointSets im-

plementation of DisjointSets.

Exercise 8.8 Repeat Exercise 8.7 assuming the CompressedDisjoint-

Sets implementation of DisjointSets. Express the best result you can as
simply as possible using big-O notation.

Exercise 8.9 Prove that TreeDisjointSets, shown in Figure 8.4, meets

the DisjointSets specification, given in Figure 8.1.

Exercise 8.10 Prove that ShortDisjointSets, shown in Figure 8.5, meets

the DisjointSets specification, given in Figure 8.1.

Exercise 8.11 Prove that CompressedDisjointSets, described in Sec-

tion 8.4, meets the DisjointSets specification, given in Figure 8.1. Use as
the structural invariant that for 0 ≤ i < n,

• if rank[i] = 0 then there is no j, such that 0 ≤ j < n, j 6= i, and

parent[j] = i; and

• if rank[i] > 0, then

– there is some j such that 0 ≤ j < n, j 6= i, and parent[j] = i; and

– rank[i] > max{rank[j] | 0 ≤ j < n, j 6= i, parent[j] = i}.
CHAPTER 8. DISJOINT SETS 307

Exercise 8.12 Suppose that we modify ShortDisjointSets so that in

the Merge operation we make the tree with fewer nodes a child of the root
of the other tree (choosing arbitrarily if both trees have the same number
of nodes). Prove by induction on k that any tree with k > 0 nodes formed
in this way will have height at most lg k.

* Exercise 8.13 Suppose that we modify TreeDisjointSets so that in

the Merge operation we flip a fair coin to determine which node will be
the new root. Analyze the worst-case expected running time of Find for
such an implementation. Express your answer as simply as possible using Θ-
notation. In showing the lower bound, describe a sequence of operations for
an arbitrarily large universe of elements such that the last Find is expected
to require the stated running time.
(i)
Exercise 8.14 Prove by induction on i that A0 (n) = n+i, so that A1 (n) =
2n + 1.

Exercise 8.15 Using the result of Exercise 8.14, prove by induction on i

(i)
that A1 (n) = 2i (n + 1) − 1, so that A2 (n) = 2n+1 (n + 1) − 1.

Exercise 8.16 Prove by induction on k that each Ak is nondecreasing.

Exercise 8.17 Using the results of Exercises 8.15 and 8.16, prove by in-
duction on i that n
·2
(i) ··
A2 (n) ≥ 22 ,
where the right-hand side has i 2s.

Exercise 8.18 Using the result of Exercise 8.15, evaluate A3 (1).

Exercise 8.19 Using the result of Exercises 8.17 and 8.18, show that
2
··
2·
A4 (1) ≥ 2 ,

where the right-hand side has 2051 2s.

Exercise 8.20 For the following, you may use the results of Exercises 8.15
and 8.16.

a. Prove by induction on i that for each i ∈ N,

(i)
lg(i) (n) ≥ min{k | A2 (k) ≥ n}.
CHAPTER 8. DISJOINT SETS 308

(k)
b. Prove by induction on k that for k ≥ 4, Ak (1) ≥ A2 (k).

c. Using the results of parts a and b, prove that for each i ∈ N, there is
an ni ∈ N such that whenever n ≥ ni , α(n) ≤ lg(i) (n).

d. Using the result of part c, prove that for each i ∈ N, α(i) ∈ o(lg(i) n).

* Exercise 8.21 Let

lg∗ n = min{k | lg(k) n ≤ 1}.

Prove that α(n) ∈ o(lg∗ n).

8.7 Chapter Notes

The TreeDisjointSets implementation of DisjointSets is due to Galler
and Fisher [45]. The improvement of Section 8.3 is presented by Hopcroft
and Ullman [63], who credit McIlroy and Morris with having implemented
it.
The improvement using path compression is credited to Tritter by Knuth
[76]. The amortized analysis of this structure yielding results similar to those
presented here was done by Tarjan [103, 104]. The analysis given here is
based on the presentation by Cormen, et al. [25], which is based on a proof
due to Kozen [82].
Exercise 8.12 is from Brassard and Bratley [18].
Chapter 9

Graphs

Often we need to model relationships that can be expressed using a set

of pairs. Examples include distances between points on a map, links in a
communications network, precedence constraints between tasks, and com-
patibility of items or people. In some cases, the relationship is symmetric;
e.g., if A is compatible with B, then B is compatible with A. In other cases,
the relationship is asymmetric; e.g., the requirement that A precedes B is
not the same as the requirement that B precedes A. All of these relation-
ships can be modeled using graphs. Having modeled the relationship, we can
then apply graph algorithms for extracting such information as a shortest
path between two points or a valid ordering of tasks.
There are two kinds of graphs, depending on whether the relationship
to be modeled is symmetric or asymmetric. For symmetric relationships,
we define an undirected graph to be a pair (V, E), where V is a finite set of
vertices (or nodes) and E is a set of 2-element subsets of V . We refer to the
elements of E as edges. We can represent undirected graphs pictorially as
in Figure 9.1, where vertices are denoted by circles and edges are denoted
by line segments or curves connecting their constituent vertices. We often
say that an edge {u, v} is incident on vertices u and v, and that u and v are
therefore adjacent.
For modeling asymmetric relationships, we define a directed graph to be
a pair (V, E), where again V is a finite set of vertices or nodes, but E is a set
of ordered pairs of distinct elements of V . Again, we refer to the elements of In some cases,
we drop the
E as edges. In order to differentiate the edges of an undirected graph from requirement that
the edges of a directed graph, we sometimes refer to the former as undirected the elements be
edges and to the latter as directed edges. We can represent directed graphs distinct.

in a manner similar to our depiction of undirected graphs, using arrows to

309
CHAPTER 9. GRAPHS 310

Figure 9.1 An undirected graph

Figure 9.2 A directed graph

indicate the directions of the edges. Conventionally, we draw the edge (u, v)
as an arrow from u to v (see Figure 9.2). For a directed edge (u, v) we say
that v is adjacent to u, but not vice versa (unless (v, u) is also an edge in
the graph).
We usually want to associate some additional information with the ver-
tices and/or the edges. For example, if the graph is used to represent dis-
tances between points on a map, we would want to associate a distance
with each edge. In addition, we might want to associate the name of a city
with each vertex. In order to simplify our presentation, we will focus our
attention on the edges of a graph and any information associated with them.
Specifically, as we did for disjoint sets in the previous chapter, we will adopt
the convention that the vertices of a graph will be designated by natural
CHAPTER 9. GRAPHS 311

numbers 0, . . . , n − 1. If additional information needs to be associated with

vertices, it can be stored in an array indexed by the numbers designating
the vertices. While some applications might require more flexibility, this
scheme is sufficient for our purposes.
Although in practice it may be beneficial to define separate ADTs for
directed and undirected graphs, respectively, it will simplify our presentation
if we specify a single Graph ADT, as shown in Figure 9.3. This actually
specifies a directed graph, but we can use it to represent an undirected graph
if we make sure that whenever (i, j) is an edge, then (j, i) is an edge with
the same associated information. We can therefore use the same ADT for
both types of graph, though the specification itself will never guarantee that
the graph is undirected.
This specification uses the data type Edge, which is implemented by
three readable representation variables, source, dest, and data. We assume
that it contains a constructor Edge(i, j, x), which sets source to i, dest to j,
and data to x. It contains no other operations other than the tree accessor
operations, so it is an immutable structure. Although we place no restric-
tions on the values stored in its representation variables, the specification
of Graph.AllFrom ensures that any Edge in the ConsList returned by
this operation will have natural numbers for source and dest and a non-nil
value for data.
In the next two sections, we will consider two applications of the Graph
ADT. Each of these applications will result in a graph algorithm. We will
then examine two implementations of Graph and analyze the running times
of their operations. We will also analyze the space usage of each implemen-
tation. Using these analyses, we will analyze the running times of the two
algorithms with each of these implementations.

9.1 Universal Sink Detection

Our first example is somewhat contrived, but it serves as a useful introduc-
tion to graph algorithms. To begin, we define a sink in a directed graph
G = (V, E) to be a vertex v with no outgoing edges. A universal sink is a
sink v such that for every vertex u 6= v, (u, v) ∈ E. In this section, we will
examine the problem of finding a universal sink in a directed graph, if one
exists.
We first observe that a directed graph can have at most one universal
sink. Let us therefore consider the related problem of returning a universal
sink in a nonempty graph if one exists, or returning an arbitrary vertex
CHAPTER 9. GRAPHS 312

Figure 9.3 The Graph ADT

Precondition: n is a Nat.
Postcondition: Constructs a Graph with vertices 0, . . . , n − 1 and no
edges.
Graph(n)
Precondition: true.
Postcondition: Returns the number of vertices in the graph.
Graph.Size()
Precondition: i and j are Nats less than the number of vertices, i 6= j,
and x refers to a non-nil item.
Postcondition: Associates x with the edge (i, j), adding this edge if nec-
essary.
Graph.Put(i, j, x)
Precondition: i and j are Nats less than the number of vertices.
Postcondition: Returns the data item associated with edge (i, j), or nil if
(i, j) is not in the graph.
Graph.Get(i, j)
Precondition: i is a Nat less than the number of vertices.
Postcondition: Returns a ConsList of Edges representing the edges
proceeding from vertex i, where an edge (i, j) with data x is represented by
an Edge with source = i, dest = j, and data = x.
Graph.AllFrom(i)
CHAPTER 9. GRAPHS 313

otherwise. We will then reduce the universal sink detection problem to this
variant. Suppose we consider any two distinct vertices, u and v (if there is
only one vertex, clearly it is a universal sink). If (u, v) ∈ E, then u cannot
be a sink. Otherwise, v cannot be a universal sink. Let G′ be the graph
obtained by removing from G one vertex x that is not a universal sink,
along with all edges incident on x. If G has a universal sink w, then w must
also be a universal sink in G′ . We have therefore transformed this problem
to a smaller instance. Because this reduction is a transformation, we can
implement it using a loop.
In order to implement this algorithm using the Graph ADT, we need to
generalize the problem to a subgraph of G comprised of the vertices i, . . . , j
and all edges between them. If j > i, we can then eliminate either i or j
from the range of vertices, depending on whether (i, j) is an edge. Note
that by generalizing the problem in this way, we do not need to modify
the graph when we eliminate vertices — we simply keep track of i and j,
the endpoints of the range of vertices we are considering. If there is an edge
(i, j), we eliminate vertex i by incrementing i; otherwise, we eliminate vertex
j by decrementing j. When all vertices but one have been eliminated (i.e.,
when i = j), the remaining vertex must be the universal sink if there is one.
We can therefore solve the original universal sink detection problem for a
nonempty graph by first finding a candidate vertex i as described above. We
know that if there is a universal sink, it must be i. We then check whether
i is a universal sink by verifying that for every j 6= i, (j, i) is an edge but
(i, j) is not. The resulting algorithm is shown in Figure 9.4.

9.2 Topological Sort

A cycle in a directed graph G = (V, E) is a finite sequence of vertices
v0 , . . . , vk such that (vk , v0 ) ∈ E, and for 0 ≤ i < k, (vi , vi+1 ) ∈ E. A
directed graph with no cycles is said to be acyclic. These two terms apply
analogously to undirected graphs as well, except that in this case all the
edges in the sequence must be distinct. In either type of graph, a cycle in
which all the vertices in the sequence are distinct is said to be simple.
Directed acyclic graphs are often used to model precedence relationships
between objects or activities. Suppose, for example, that we have four jobs,
A, B, C, and D. We must schedule these jobs sequentially such that A pre-
cedes C, B precedes A, and B precedes D. These precedence relationships
can be modeled by the directed acyclic graph shown in Figure 9.5. We need
to find an ordering of the vertices such that for every edge (u, v), u precedes
CHAPTER 9. GRAPHS 314

Figure 9.4 An algorithm to find a universal sink in a directed graph

Precondition: G refers to a Graph.

Postcondition: Returns the universal sink in G, or −1 if G has no universal
sink.
UniversalSink(G)
if G.Size() = 0
return −1
else
i ← 0; j ← G.Size() − 1
// Invariant: If G has a universal sink k, then i ≤ k ≤ j.
while i < j
if G.Get(i, j) = nil
j ←j−1
else
i←i+1
// Invariant: For 0 ≤ k < j, if k 6= i, then (k, i) is an edge,
// but (i, k) is not.
for j ← 0 to G.Size() − 1
if j 6= i and (G.Get(i, j) 6= nil or G.Get(j, i) = nil)
return −1
return i

v in the ordering. Such an ordering is called a topological sort of the graph.

Examples of topological sorts of the graph in Figure 9.5 are hB, A, C, Di
and hB, D, A, Ci. In this section, we will present an algorithm for finding a
topological sort of a given directed acyclic graph. First, we will show that
every directed acyclic graph has a topological sort.

Lemma 9.1 Every nonempty directed acyclic graph has at least one vertex
with no incoming edges.

Proof: By contradiction. Suppose every vertex in some nonempty directed

acyclic graph G has incoming edges. Then starting from any vertex, we may
always traverse an incoming edge backwards to its source. Because G has
finitely many vertices, if we trace a path in this fashion, we must eventually
repeat a vertex. We will have then found a cycle — a contradiction.
CHAPTER 9. GRAPHS 315

Figure 9.5 A directed acyclic graph modeling precedence relationships

A B

C D

Theorem 9.2 Every directed acyclic graph G = (V, E) has a topological

sort.

Proof: By induction on the size of V .

Base: V = ∅. Then the empty ordering is a topological sort.

Induction Hypothesis: Suppose that for some n > 0, every directed

acyclic graph with fewer than n vertices has a topological sort.

Induction Step: Let G = (V, E) be a directed acyclic graph with n ver-

tices. Because G is nonempty, it must have at least one vertex v0 with no
incoming edges. Let G′ be the graph obtained from G by removing v0 and
all of its outgoing edges. By the induction hypothesis, G′ has a topological
sort v1 , . . . , vn−1 . Because v0 has no incoming edges in G, v0 , . . . , vn−1 must
therefore be a topological sort for G.

The proof of Theorem 9.2 is constructive; i.e., it gives an algorithm for

finding a topological sort. First, we find a vertex v0 with no incoming edges.
v0 will come first in the topological sort. The remainder of the topological
sort is obtained by removing v0 and finding a topological sort of the resulting
graph. We have therefore transformed the problem to a smaller instance.
The above sketch is missing a few details. For example, we need to
know how to find a vertex with no incoming edges. Also, our Graph ADT
provides no mechanism for removing vertices. In order to overcome these
problems, we will maintain an array incount[0..n − 1] so that incount[i] gives
the number of edges to i from vertices not yet in the topological sort. When
we add a vertex i to the topological sort, we do not need to remove it from
CHAPTER 9. GRAPHS 316

the graph; instead, we can simply decrement incount[j] for all j adjacent to i.
To initialize incount, we can simply examine each edge (i, j) and increment
incount[j].
In order to speed up finding the next vertex in the topological sort, let
us keep track of all vertices i for which incount[i] = 0. We can use a Stack
for this purpose. After we initialize incount, we can traverse it once and
push each i such that incount[i] = 0 onto the stack. Thereafter, when we
decrement an entry incount[i], we need to see if it reaches 0, and if so, push
it onto the stack. The algorithm is shown in Figure 9.6.

9.3 Adjacency Matrix Implementation

Our first implementation of Graph will have a single representation vari-
able:

• edges[0..n − 1, 0..n − 1].

Our structural invariant will be that edges[i, i] = nil for 0 ≤ i < n. We

interpret edges[i, j] as giving the information associated with edge (i, j),
provided this value is non-nil. We interpret a nil value for edges[i, j] as
indicating the absence of an edge (i, j). The full implementation is shown
in Figure 9.7.
Each of the operations Size, Put, and Get runs in Θ(1) time. The
constructor is easily seen to run in Θ(n2 ) time. The AllFrom operation
clearly runs in Θ(n) time, where n is the number of vertices in the graph.
The space usage is clearly in Θ(n2 ).
We can now analyze the running times of the algorithms given in the
previous two sections. We will first consider UniversalSink from Figure
9.4. Let n be the number of vertices in G. The while loop begins with
j − i = n − 1 and decreases j − i each iteration. Because it terminates when
j − i = 0 it iterates n − 1 times. Because MatrixGraph.Get runs in Θ(1)
time, the entire loop runs in Θ(n) time. The for loop iterates at most n
times, and each iteration runs in Θ(1) time. The entire algorithm therefore
operates in Θ(n) time.
Let us now consider TopSort from Figure 9.6. We will need to break
the algorithm into the four for loops.
In the body of the second for loop, MatrixGraph.AllFrom runs in
Θ(n) time. The body of the while loop runs in Θ(1) time. Each iteration
decreases the number of elements in L by 1 until L is empty. Because there
can be at most n − 1 edges from any vertex, the while loop iterates at most
CHAPTER 9. GRAPHS 317

Figure 9.6 Topological sort algorithm

Precondition: G is a directed acyclic graph.

Postcondition: Returns an array listing the vertices of G in topological
order.
TopSort(G)
n ← G.Size(); s ← new Array[0..n − 1]
incount ← new Array[0..n − 1]; avail ← new Stack()
for i ← 0 to n − 1
incount[i] ← 0
// Invariant: For 0 ≤ j < n, incount[j] gives the number of edges to j
// from vertices 0, . . . , i − 1.
for i ← 0 to n − 1
L ← G.AllFrom(i)
while not L.IsEmpty()
k ← L.Head().Dest(); incount[k] ← incount[k] + 1; L ← L.Tail()
for i ← 0 to n − 1
if incount[i] = 0
avail.Push(i)
// Invariant: For 0 ≤ j < n, incount[j] is the number of edges to j from
// vertices not in s[0..i − 1], and if incount[j] = 0, then j is either in
// s[0..i − 1] or avail, but not both.
for i ← 0 to n − 1
s[i] ← avail.Pop(); L ← G.AllFrom(s[i])
while not L.IsEmpty()
k ← L.Head().Dest(); incount[k] ← incount[k] − 1; L ← L.Tail()
if incount[k] = 0
avail.Push(k)
return s
CHAPTER 9. GRAPHS 318

Figure 9.7 MatrixGraph implementation of Graph

Structural Invariant: edges[i, i] = nil for 0 ≤ i < n.

MatrixGraph(n)
edges ← new Array[0..n − 1, 0..n − 1]
for i ← 0 to n − 1
for j ← 0 to n − 1
edges[i, j] ← nil

MatrixGraph.Size()
return SizeOf(edges[0]) // Number of columns

MatrixGraph.Put(i, j, x)
if i = j
error
else
edges[i, j] ← x

MatrixGraph.Get(i, j)
return edges[i, j]

MatrixGraph.AllFrom(i)
L ← new ConsList()
for j ← 0 to n − 1
if edges[i, j] 6= nil
L ← new ConsList(new Edge(i, j, edges[i, j]), L)
return L
CHAPTER 9. GRAPHS 319

n − 1 times. Its running time is therefore in O(n). The body of the second
for loop therefore runs in Θ(n) time. Because it iterates n times, its running
time is in Θ(n2 ).
The first and third for loops clearly run in Θ(n) time. Furthermore, the
analysis of the fourth for loop is similar to that of the second. Therefore,
the entire algorithm runs in Θ(n2 ) time.
Note that the second and fourth for loops in TopSort each contain a
nested while loop. Each iteration of this while loop processes one of the
edges. Furthermore, each edge is processed at most once by each while
loop. The total number of iterations of each of the while loops is therefore
the number of edges in the graph. While this number can be as large as
n(n − 1) ∈ Θ(n2 ), it can also be much smaller.
The number of edges does not affect the asymptotic running time, how-
ever, because MatrixGraph.AllFrom runs in Θ(n) time, regardless of
how many edges it retrieves. If we can make this operation more efficient,
we might be able to improve the running time for TopSort on graphs with
few edges. In the next section, we will examine an alternative implementa-
tion that accomplishes this.

9.4 Adjacency List Implementation

In this section, we consider an implementation designed to improve the ef-
ficiency of the AllFrom operation. The two-dimensional array used in the
adjacency matrix implementation can be thought of as an array of arrays
each containing the adjacency information for a single vertex. In the ad-
jacency list implementation, we still use an array indexed by vertices to
store the adjacency information for each vertex; however, we maintain this
adjacency information in a ConsList instead of an array. In such a rep-
resentation, the ConsList for vertex i is exactly the ConsList that needs
to be returned by AllFrom(i). Furthermore, because a ConsList is im-
mutable, we can return it without violating security.
We again use a single representation variable:

• elements[0..n − 1]: an array of ConsLists.

Our structural invariant will be that for 0 ≤ i < n, elements[i] refers to a

ConsList containing Edges representing at most one (i, j) for each j such
that 0 ≤ j < n and j 6= i, where each Edge has a non-nil data item (see the Corrected
4/29/10.
specification for Graph.AllFrom in Figure 9.3 for an explanation of this
representation). We interpret the Edges in this structure as representing
CHAPTER 9. GRAPHS 320

the edges in the graph, along with the information associated with each
edge.
A partial implementation of ListGraph is shown in Figure 9.8. In Figure 9.8
corrected
addition, the Size operation returns the size of elements, and AllFrom(i) 4/29/10.
returns elements[i].
It is easily seen that the Size and AllFrom operations run in Θ(1)
time, and that the constructor runs in Θ(n) time. Each iteration of the
while loop in the Get operation reduces the size of L by 1. The length of
L is initially the number of vertices adjacent to i. Because each iteration
runs in Θ(1) time, the entire operation runs in Θ(m) time in the worst case,
where m is the number of vertices adjacent to i. The worst case for Get
occurs when vertex j is not adjacent to i. Similarly, it can be seen that the
Put operation runs in Θ(m) time. Note that Θ(m) ⊆ O(n). The space
usage of ListGraph is easily seen to be in Θ(n + a), where a is the number
of edges in the graph.
Let us now revisit the analysis of the running time of TopSort (Figure
9.6), this time assuming that G is a ListGraph. Consider the second for
loop. Note that running time of the nested while loop does not depend on
the implementation of G; hence, we can still conclude that it runs in O(n)
time. We can therefore conclude that the running time of the second for
loop is in O(n2 ). However, because we have reduced the running time of
AllFrom from Θ(n) to Θ(1), it is no longer clear that the running time of
this loop is in Ω(n2 ). Indeed, if there are no edges in the graph, then the
nested while loop will not iterate. In this case, the running time is in Θ(n).
We therefore need to analyze the running time of the nested while loop
more carefully. Notice that over the course of the for loop, each edge is
processed by the inner while loop exactly once. Therefore, the body of the
inner loop is executed exactly a times over the course of the entire outer
loop, where a is the number of edges in G. Because the remainder of the
outer loop is executed exactly n times, the running time of the outer loop
is in Θ(n + a).
We now observe that the fourth loop can be analyzed in exactly the same
way as the second loop; hence, the fourth loop also runs in Θ(n + a) time.
In fact, because the structure of these two loops is quite common for graph
algorithms, this method of calculating the running time is often needed for
analyzing algorithms that operate on ListGraphs.
To complete the analysis of TopSort, we observe that the first and
third loops do not depend on how G is implemented; hence, they both run
in Θ(n) time. The total running time of TopSort is therefore in Θ(n + a).
For graphs in which a ∈ o(n2 ), this is an improvement over the Θ(n2 )
CHAPTER 9. GRAPHS 321

Figure 9.8 ListGraph implementation (partial) of Graph

Structural Invariant: For 0 ≤ i < n, elements[i] refers to a ConsList

containing Edges representing at most one (i, j) for each j such that
0 ≤ j < n and j 6= i, where each Edge has a non-nil data item.

ListGraph(n)
elements ← new Array[0..n − 1]
for i ← 0 to n − 1
elements[i] ← new ConsList()

ListGraph.Put(i, j, x)
if j ≥ SizeOf(elements) or j < 0 or j = i or x = nil
error
else
elements[i] ← AddEdge(new Edge(i, j, x), elements[i])

ListGraph.Get(i, j)
L ← elements[i]
while not L.IsEmpty()
if L.Head().Dest() = j
return L.Head().Data()
else
L ← L.Tail()
return nil

— Internal Functions Follow —

Precondition: e refers to an Edge and L refers to a ConsList of Edges
each having the same source as e.
Postcondition: Returns a ConsList with the contents of L, plus the
edge e. If the edge represented by e is already in L, it is replaced by e.
ListGraph.AddEdge(e, L)
if L.IsEmpty()
return new ConsList(e, L)
else if L.Head().Dest() = e.Dest()
return new ConsList(e, L.Tail())
else
return new ConsList(L.Head(), AddEdge(e, L.Tail()))
CHAPTER 9. GRAPHS 322

running time when G is implemented as a MatrixGraph.

Let us now consider the impact of the ListGraph implementation on
the analysis of UniversalSink (Figure 9.4). Due to the increased running
time of Get, the body of the while loop runs in Θ(m) time, where m is the
number of vertices adjacent to i. This number cannot be more than n − 1,
nor can it be more than a. Because this loop iterates Θ(n) times, we obtain
an upper bound of O(n min(n, a)). Likewise, it is easily seen that the for
loop runs in O(n min(n, a)) time.
To see that this bound is tight for the while loop, let us first consider
the case in which a ≤ n(n − 1) − ⌊n/2⌋. Suppose that from vertex 0 there
is an edge to each of the vertices 1, . . . , min(a, ⌊(n − 1)/2⌋), but no edge
to any other vertex. From vertices other than 0 we may have edges to
any of the other vertices. Note that with these constraints, we can have
up to n(n − 1) − ⌊n/2⌋ edges. For such a graph, the first ⌊n/2⌋ iterations
of the while loop will have i = 0, while j ranges from n − 1 down to
⌊(n − 1)/2⌋ + 1. For each of these iterations, Get(i, j) runs in Θ(min(a, n))
time, because there are Θ(min(a, n)) vertices adjacent to 0, but j is not
adjacent to 0. Because the number of these iterations is in Θ(n), the total
time is in Θ(n min(n, a)).
Now let us consider the case in which a > n(n − 1) − ⌊n/2⌋. In this case,
we make sure that from each of the vertices 0, . . . , ⌊n/2⌋ − 1, there is an
edge to every other vertex. Furthermore, we make sure that in each of the
ConsLists of edges from these first ⌊n/2⌋ vertices, the edge to vertex n − 1
occurs last. From the remaining vertices we may have any edges listed in any
order. For such a graph, the first ⌊n/2⌋ iterations of the while loop will have
j = n − 1, while i ranges from 0 to ⌊n/2⌋ − 1. For each of these iterations,
Get(i, j) runs in Θ(n) time, because there are Θ(n) vertices adjacent to
i, and n − 1 is the last of these. Because the total number of iterations is
in Θ(n), the total time is in Θ(n2 ). Because a ≥ n, this is the same as
Θ(n min(a, n)).
Based on the analyses of the two algorithms, we can see that neither
implementation is necessarily better than the other. If an algorithm relies
more heavily on Get than on AllFrom, it is better to use MatrixGraph.
If an algorithm relies more heavily on AllFrom, it is probably better to
use ListGraph, particularly if there is a reasonable expectation that the
graph will be sparse — i.e., that it will have relatively few edges. Note also
that for sparse graphs, a ListGraph will use considerably less space.
CHAPTER 9. GRAPHS 323

9.5 Multigraphs
Let us briefly consider the building of a ListGraph. We must first con-
struct a graph with no edges, then add edges one by one using the Put
operation. The constructor runs in Θ(n) time. The Put operation runs
in Θ(m) time, where m is the number of vertices adjacent to the source
of the edge. It is easily seen that the time required to build the graph is
in O(n + a min(n, a)), where a is the number of edges. It is not hard to Corrected
3/28/12.
match this upper bound using graphs in which the number of vertices with
outgoing edges is minimized for the given number of edges. An example of
a sparse graph (specifically, with a ≤ n) that gives this behavior is a graph
whose edge set is
{(0, j) | 1 ≤ j < a}.
A dense graph giving this behavior is a complete graph — a graph in which
every possible edge is present. Note that in terms of the number of vertices,
the running time for building a complete graph is in Θ(n3 ).
Building a ListGraph is expensive because the Put operation must
check each edge to see if it is already in the graph. We could speed this
activity considerably if we could avoid this check. However, the check is
necessary not only to satisfy the operation’s specification, but also to main-
tain the structural invariant. If we can modify the specification of Put and
weaken the structural invariant so that parallel edges (i.e., multiple edges
from a vertex i to a vertex j) are not prohibited, then we can build the
graph more quickly.
We therefore extend the definitions of undirected and directed graphs to
allow parallel edges from one vertex to another. We call such a structure a
multigraph. We can then define the Multigraph ADT by modifying the
following postconditions in the specification of Graph:
• Put(i, j, x): Adds the edge (i, j) and associates x with it.

• Get(i, j): Returns the data item associated with an edge (i, j), or nil
if (i, j) is not in the graph.
We could also specify additional operations for retrieving all edges (i, j) or
modifying the data associated with an edge, but this specification is sufficient
for our purposes.
We will represent a Multigraph using adjacency lists in the same way
as in the ListGraph implementation. The structural invariant will be mod-
ified to allow parallel edges (i, j) for the same i and j. The implementation
of the operations remains the same except for Put, whose implementation is
CHAPTER 9. GRAPHS 324

Figure 9.9 The Put operation in the ListMultigraph implementation

of Multigraph

ListMultigraph.Put(i, j, x)
if j ≥ SizeOf(elements) or j < 0 or x = nil
error
else
elements[i] ← new ConsList(new Edge(i, j, x), elements[i])

shown in Figure 9.9. It is easily seen that a ListMultigraph can be built Figure 9.9
corrected
in Θ(n + a) time, where n is the number of vertices and a is the number of 5/7/12.
edges.
Because it is more efficient to build a ListMultigraph than to build
a ListGraph, it may be advantageous to represent a graph using a List-
Multigraph. If we are careful never to add parallel edges, we can maintain
an invariant that the ListMultigraph represents a graph. This transfers
the burden of maintaining a valid graph structure from the Put operation
to the code that invokes the Put operation.
Although we can use a ListMultigraph to represent a graph, an inter-
esting problem is how to construct a ListGraph from a given ListMulti-
graph. Specifically, suppose we wish to define a ListGraph constructor
that takes a ListMultigraph as input and produces a ListGraph with
the same vertices and edges, assuming the ListMultigraph has no par-
allel edges. We may be able to do this more efficiently than simply calling
ListGraph.Put repeatedly.
One approach would be to convert the ListMultigraph to a Matrix-
Graph, then convert the MatrixGraph to a ListGraph. In fact, we
do not really need to build a MatrixGraph — we could simply use a
two-dimensional array as a temporary representation of the graph. As we
examine each edge of the ListMultigraph, we can check to see if it has
been added to the array, and if not, add it to the appropriate adjacency
list. If we ever find parallel edges, we can immediately terminate with an
error. Each edge can therefore be processed in Θ(1) time, for a total of Θ(a)
time to process the edges. Unfortunately, Θ(n2 ) time is required to initialize
the array, and the space usage of Θ(n2 ) is rather high, especially for sparse
graphs. However, the resulting time of Θ(n2 ) is still an improvement (in
CHAPTER 9. GRAPHS 325

most cases) over the Θ(a min(n, a)) worst-case time for repeatedly calling
ListGraph.Put.
In the above solution, the most natural way of processing the edges of
the ListMultigraph is to consider each vertex i in turn, and for each
i, to process all edges proceeding from i. As we are processing the edges
from vertex i, we only need row i of the array. We could therefore save a
significant amount of space by replacing the two-dimensional array with a
singly-dimensioned array A[0..n − 1]. Before we consider any vertex i, we
initialize A. For each edge (i, j), we check to see if it has been recorded in
A[j]. If so, we have found a parallel edge; otherwise, we record this edge in
A[j]. Thus, we have reduced our space usage to Θ(n). However, because A
must be initialized each time we consider a new vertex, the overall running
time is still in Θ(n2 ).
Note that if we ignore the time for initializing A, this last solution runs
in Θ(n + a) time in the worst case. We can therefore use the virtual ini-
tialization technique of Section 7.1 to reduce the overall running time to
Θ(n + a). However, the technique of virtual initialization was inspired by
the need to avoid initializing large arrays, not to avoid initializing small ar-
rays many times. If we are careful about the way we use the array, a single
initialization should be significant. Thus, if we can find a way to avoid re-
peated initializations of A, we will be able to achieve Θ(n + a) running time
without using virtual initialization.
Consider what happens if we simply omit every initialization of A except
the first. If, when processing edge (i, j), we find edge (i′ , j), for some i′ <
i, recorded in A[j], then we know that no other edge (i, j) has yet been
processed. We can then simply record (i, j) in A[j], as if no edge had been
recorded there. If, on the other hand, we find that (i, j) has already been
recorded in A[j], we know that we have found a parallel edge.
As a final simplification to this algorithm, we note that there is really
no reason to store Edges in the array. Specifically, the only information we
need to record in A[j] is the most recent (i.e., the largest) vertex i for which
an edge (i, j) has been found. Thus, A can be an array of integers. We
can initialize A to contain only negative values, such as −1, to indicate that
no such edge has yet been found. The resulting ListGraph constructor is
shown in Figure 9.10. It is easily seen to run in Θ(n + a) time and use Θ(n)
temporary space in the worst case, where n and a are the number of vertices
and unique edges, respectively, in the given ListMultigraph.
CHAPTER 9. GRAPHS 326

Figure 9.10 Constructor for building a ListGraph from a ListMulti-

graph

Precondition: G refers to a ListMultigraph with no parallel edges.

Postcondition: Constructs a ListGraph with the same vertices and edges
as G.
ListGraph(G)
size ← G.Size(); elements ← new Array[0..size − 1]
A ← new Array[0..size − 1]
// Invariant: For 0 ≤ k < i, elements[k] = G.AllFrom(k) and
// A[k] = −1.
for i ← 0 to size − 1
elements[i] ← G.AllFrom(i); A[i] ← −1
// Invariant: For 0 ≤ j < size, A[j] < i. For 0 ≤ k < i and 0 ≤ j < size,
// elements[k] has at most one edge to j.
for i ← 0 to size − 1
L ← elements[i]
// Invariant: For 0 ≤ j < size, A[j] ≤ i. If A[j] = i, then elements[i]
// contains one more edge to j than does L; otherwise, they contain
// the same number of edges to j.
while not L.IsEmpty()
j ← L.Head().Dest()
if A[j] < i
A[j] ← i; L ← L.Tail()
else
error
CHAPTER 9. GRAPHS 327

9.6 Summary
Graphs are useful for representing relationships between data items. Various
algorithms can then be designed for manipulating graphs. As a result, we
can often use the same algorithm in a variety of different applications.
Graphs may be either directed or undirected, but we can treat undirected
graphs as directed graphs in which for every edge (u, v), there is a reverse
edge (v, u). We then have two implementations of graphs. The adjacency
matrix implementation has Get and Put operations that run in Θ(1) time,
but its AllFrom operation runs in Θ(n) time, where n is the number of
vertices in the graph. Its space usage is in Θ(n2 ). On the other hand, the
adjacency list implementation has an AllFrom operation that runs in Θ(1)
time, but its Get and Put operations run in Θ(m) time in the worst case,
where m is the number of vertices adjacent to the given source vertex. Its
space usage is in Θ(n + a) where n is the number of vertices and a is the
number of edges.
In order to improve the running time of the Put operation — and hence
of building a graph — when using an adjacency list, we can relax our defi-
nition to allow parallel edges. The resulting structure is known as a multi-
graph. We can always use a multigraph whenever a graph is required, though
it might be useful to maintain an invariant that no parallel edges exist. Fur-
thermore, we can construct a ListGraph from a ListMultigraph with
no parallel edges in Θ(n + a) time and Θ(n) space, where n is the number
of vertices and a is the number of edges. Figure 9.11 shows a summary of
the running times of these operations for each of the implementations of
Graph, as well as for ListMultigraph.

9.7 Exercises
Exercise 9.1 Prove that UniversalSink, shown in Figure 9.4, meets its
specification.

Exercise 9.2 Prove that TopSort, shown in Figure 9.6, meets its specifi-
cation.

Exercise 9.3 Give an algorithm that takes as input a directed graph G =

(V, E) and returns a directed graph G′ = (V, E ′ ), where

E ′ = {(u, v) | (v, u) ∈ E}.

CHAPTER 9. GRAPHS 328

Figure 9.11 Comparison of running times for two implementations of

Graph, along with ListMultigraph

constructor Get Put AllFrom

MatrixGraph Θ(n2 ) Θ(1) Θ(1) Θ(n)
ListGraph Θ(n) Θ(m) Θ(m) Θ(1)
ListMultigraph Θ(n) Θ(m) Θ(1) Θ(1)

Notes:

• n is the number of vertices in the graph, and m is the number of

vertices adjacent to the given source vertex.

• The constructor referenced above is the one that constructs a graph

with no edges — the ListGraph constructor that takes a ListMulti-
graph as input runs in Θ(n+a) worst-case time, where n is the number
of vertices and a is the number of edges in the graph, assuming there
are no parallel edges.

• The Size operation runs in Θ(1) worst-case time for all implementa-
tions.

• All running times are worst-case.

Thus, G′ contains the same edges as does G, except that they are reversed
in G′ . Express the running time of your algorithm as simply as possible
using Θ-notation in terms of the number of vertices n and the number of
edges a, assuming the graphs are implemented using
a. MatrixGraph
b. ListGraph
c. ListMultigraph.

Exercise 9.4 Give an algorithm to compute the number of edges in a given

graph. Express the running time of your algorithm as simply as possible
using Θ-notation in terms of the number of vertices n and the number of
edges a, assuming the graph is implemented using
a. MatrixGraph
CHAPTER 9. GRAPHS 329

b. ListGraph

Exercise 9.5 A directed graph is said to be transitively closed if whenever

(u, v) and (v, w) are edges, then (u, w) is also an edge. Give an O(n3 )
algorithm to determine whether a given directed graph is transitively closed.
You may assume the graph is implemented as a MatrixGraph.

Exercise 9.6 An undirected graph is said to be connected if for every pair

of vertices u and v, there is a path of edges leading from u to v. A tree
is a connected undirected acyclic graph. Prove each of the following for an
undirected graph G with n vertices:

a. If G is a tree, then G has exactly n − 1 edges.

b. If G is connected and has exactly n − 1 edges, then G is a tree.

c. If G is acyclic and has exactly n − 1 edges, then G is a tree.

Exercise 9.7 Prove that MatrixGraph, shown in Figure 9.7, meets its
specification.

Exercise 9.8 Prove that the ListGraph constructor shown in Figure 9.10
meets its specification.

Exercise 9.9 Give an algorithm that takes as input a graph G and returns
true iff G is undirected; i.e., if for every edge (u, v), (v, u) is also an edge.
Give the best upper bound you can on the running time, expressed as simply
as possible using big-O notation in terms of the number of vertices n and
the number of edges a, assuming the graph is implemented using Matrix-
Graph.

* Exercise 9.10 An Euler path in a connected undirected graph G is a

path that contains every edge in G exactly once. Give an efficient algorithm
to find an Euler path in a connected undirected graph, provided one exists.
Your algorithm should return some representation of the Euler path, or nil if
no Euler path exists. Your algorithm should run in O(a) time, where a is the
number of edges in G, assuming that G is implemented as a ListGraph.
CHAPTER 9. GRAPHS 330

9.8 Chapter Notes

The study of graph theory began in 1736 with Leonhard Euler’s famous
study of the Königsberg Bridge Problem [36], which is simply the problem
of finding an Euler path in a connected undirected graph (see Exercise 9.10).
Good references on graph theory and graph algorithms include Even [37]
and Tarjan [104]. In the early days of electronic computing, graphs were
typically implemented using adjacency matrices. Hopcroft and Tarjan [62]
first proposed using adjacency lists for sparse graphs. The topological sort
algorithm of Section 9.2 is due to Knuth [76].
Part III

Algorithm Design Techniques

331
Chapter 10

Divide and Conquer

In Part I of this text, we introduced several techniques for applying the top-
down approach to algorithm design. We will now take a closer look at some
of these techniques. In this chapter, we will look at the divide-and-conquer
technique.
As we stated in Chapter 3, the divide-and-conquer technique involves
reducing a large instance of a problem to one or more instances having
a fixed fraction of the size of the original instance. For example, recall
that the algorithm MaxSumDC, shown in Figure 3.3 on page 77, reduces
large instances of the maximum subsequence sum problem to two smaller
instances of roughly half the size.
Though we can sometimes convert divide-and-conquer algorithms to it-
erative algorithms, it is usually better to implement them using recursion.
One reason is that typical divide-and-conquer algorithms implemented using
recursion require very little stack space to support the recursion. If we di-
vide an instance of size n into instances of size n/b whenever n is divisible by
b, we can express the total stack usage due to recursion with the recurrence
f (n) ∈ f (n/b) + Θ(1).
Applying Theorem 3.32 to this recurrence, we see that f (n) ∈ Θ(lg n). The
other reason for retaining the recursion is that when a large instance is
reduced to more than one smaller instance, removing the recursion can be
difficult and usually requires the use of a stack to simulate at least one
recursive call.
Because divide-and-conquer algorithms are typically expressed using re-
cursion, the analysis of their running times usually involves the asymptotic
solution of a recurrence. Theorem 3.32 almost always applies to this recur-
rence. Not only does this give us a tool for analyzing running times, it also

332
CHAPTER 10. DIVIDE AND CONQUER 333

can give us some insight into what must be done to make an algorithm more
efficient. We will explain this concept further as we illustrate the technique
by applying it to several problems.

10.1 Polynomial Multiplication

Suppose we are given two polynomials:
n−1
X
p(x) = ai xi
i=0
n−1
X
q(x) = bi xi .
i=0

We wish to compute the product polynomial:

 
2n−2
X X i
pq(x) =  aj bi−j  xi ,
i=0 j=0

where we define aj = bj = 0 for n ≤ j < 2n. We can clearly compute each

of the 2n − 1 coefficients in pq in Θ(n) time; hence, by Theorem 3.28, we
can compute the entire product in Θ(n2 ) time.
We wish to apply the divide-and-conquer technique to obtain a more
efficient solution. Observe that if n > 1, then we can divide each polynomial
into two smaller polynomials:
p(x) = p0 (x) + xm p1 (x)
q(x) = q0 (x) + xm q1 (x),
where
m−1
X
p0 (x) = ai xi
i=0
n−m−1
X
p1 (x) = am+i xi
i=0
m−1
X
q0 (x) = bi xi
i=0
n−m−1
X
q1 (x) = bm+i xi .
i=0
CHAPTER 10. DIVIDE AND CONQUER 334

If we set m = ⌊n/2⌋, then each of the smaller polynomials has roughly n/2
terms.
The product polynomial is now

pq(x) = p0 (x)q0 (x) + xm (p0 (x)q1 (x) + p1 (x)q0 (x)) + x2m p1 (x)q1 (x). (10.1)

To obtain the coefficients of pq, we can first compute the four products of
the smaller polynomials. We can then obtain any given coefficient of pq
by performing at most two additions. We can therefore obtain all 2n −
1 coefficients in Θ(n) time after the four smaller products are computed.
Setting m = ⌊n/2⌋, we can describe the running time of this divide-and-
conquer algorithm with the following recurrence:

f (n) ∈ 4f (n/2) + Θ(n) (10.2)

when n > 1 is a power of 2. Unfortunately, applying Theorem 3.32 yields

f (n) ∈ Θ(n2 ), which is the same running time as the brute-force calculation.
This exercise illustrates an important point about the divide-and-conquer
technique, namely, that the technique by itself does not guarantee improved
running times. In order for the technique to be effective, it must save some
work. Sometimes, as with MaxSumDC, the savings in work comes about
naturally. In such cases, it may be rather hard to see how work was saved.
In other cases, we must be more clever in order to save work.
As we suggested earlier, Theorem 3.32 can give insight into how a divide-
and-conquer solution might be designed or improved. For example, consider
recurrence (10.2) above. Because the third case of Theorem 3.32 applies,
we cannot obtain a more efficient solution by reducing the Θ(n) overhead
outside of the recursive calls. In order to improve the performance, we need
either to reduce the number of recursive calls or decrease the size of the
smaller instances. We will focus on reducing the number of recursive calls.
In the exercises, we explore alternative solutions involving decreasing the
size of the smaller instances.
The following observation gives us the insight we need in order to reduce
the number of recursive calls:

(p0 (x) + p1 (x))(q0 (x) + q1 (x)) =

p0 (x)q0 (x) + p0 (x)q1 (x) + p1 (x)q0 (x) + p1 (x)q1 (x).

Note that all four of the terms in the right-hand-side above appear in the
product pq (see (10.1) above). In order to make this fact useful, however, we
need to be able to separate out the first and last terms. We can do this by
CHAPTER 10. DIVIDE AND CONQUER 335

Figure 10.1 Divide-and-conquer polynomial multiplication algorithm

Precondition: p and q are arrays of Numbers and n is a positive Nat.

Postcondition: Returns an array P [0..2n − 2] such that P [i] is the coef-
ficient of xi in the product pq, where p[i] denotes the coefficient in p of xi
and q[i] denotes the coefficient in q of xi .
PolyMult(p[0..n − 1], q[0..n − 1])
P ← new Array[0..2(n − 1)]
if n = 1
P [0] = p[0]q[0]
else
m ← ⌈n/2⌉; s ← new Array[0..m − 1]; t ← new Array[0..m − 1]
Copy(p[0..m − 1], s[0..m − 1]); Copy(q[0..m − 1], t[0..m − 1])
for i ← m to n − 1
s[i − m] ← s[i − m] + p[i]; t[i − m] ← t[i − m] + q[i]
P1 ← PolyMult(p[0..m − 1], q[0..m − 1])
P2 ← PolyMult(s[0..m − 1], t[0..m − 1])
P3 ← PolyMult(p[m..n − 1], q[m..n − 1])
Copy(P1 [0..2(m − 1)], P [0..2(m − 1)]); P [2m − 1] ← 0
Copy(P3 [0..2(n − m − 1)], P [2m..2(n − 1)])
for i ← 0 to 2(m − 1)
P [m + i] ← P [m + i] + P2 [i] − P1 [i]
for i ← 0 to 2(n − m − 1)
P [m + i] ← P [m + i] − P3 [i]
return P

computing the products p0 (x)q0 (x) and p1 (x)q1 (x), then subtracting. Thus,
we can compute the product pq using the following three products:

P1 (x) = p0 (x)q0 (x)

P2 (x) = (p0 (x) + p1 (x))(q0 (x) + q1 (x))
P3 (x) = p1 (x)q1 (x)

We can then compute any given coefficient of pq with at most two subtrac-
tions and one addition.
The algorithm is shown in Figure 10.1. This implementation uses the Figure 10.1
corrected
Copy function specified in Figure 1.18 on page 22. Note that when we 5/7/12.
CHAPTER 10. DIVIDE AND CONQUER 336

divide the polynomials, the low-order parts will be of degree m − 1, and

the high-order parts will be of degree n − m − 1. We cannot select m so
that these two degrees are the same if n is odd. Therefore, we need to be
careful to note the degrees of each polynomial we construct. By choosing
m = ⌈n/2⌉, we ensure that m ≥ n − m. Thus, we can add the two halves
of a polynomial by first recording the low-order half, then adding in the
high-order half, yielding a polynomial of degree m − 1. After the recursive
multiplications, P1 and P2 will both have degree 2(m − 1), but P3 will have
degree 2(n − m − 1). To construct P , we can first copy P1 and P3 to the
proper locations, and fill in 0 for the coefficient of x2m−1 . We can then add
P2 [i] − P1 [i] − P3 [i] to the coefficient of xm+i ; however, because P3 has a
different degree than P1 and P2 , we use a separate loop to subtract this
polynomial.
From Figure 10.1 and Exercise 3.23 (page 102), it is evident that a total
of Θ(n) time is needed apart from the recursive calls. Thus, we can describe
the running time with the following recurrence:

f (n) ∈ 3f (n/2) + Θ(n)

whenever n > 1 is a power of 2. From Theorem 3.32, f (n) ∈ Θ(nlg 3 ).

Because lg 3 ≈ 1.59, this algorithm is an improvement over the brute-force
calculation.

10.2 Merge Sort

In Section 5.6, we saw that we can sort using a worst-case running time
in Θ(n lg n) using heap sort (Figure 5.20). However, as was suggested by
Exercise 5.24, heap sort is not stable when sorting Keyed items; i.e., items
with equal keys can be reordered by heap sort. In this section, we will apply
divide-and-conquer to obtain a stable sorting algorithm that also runs in
Θ(n lg n) time in the worst case.
We can apply divide-and-conquer to sorting by first dividing in half any
array with more than one element and sorting the two halves. We then
need to combine the two sorted halves into a single sorted array. We have
therefore reduced the sorting problem to the problem of merging two sorted
arrays into a single sorted array. Furthermore, in order to ensure stability,
we require the following behavior for a merging algorithm:

• If two elements in the same input array have equal keys, they remain
in the same order in the output array.
CHAPTER 10. DIVIDE AND CONQUER 337

• If an element x from the first input array has a key equal to some
element y in the second input array, then x must precede y in the
output array.

Suppose we are given two sorted arrays. If either is empty, we can simply
use the other. Otherwise, the element with minimum key in the two arrays
needs to be first in the sorted result. The element with minimum key in each
array is the first element in the array. We can therefore determine the overall
minimum by comparing the keys of the first elements of the two arrays. If
the keys are equal, in order to ensure stability, we must take the element
from the first array. To obtain the remainder of the result, we merge the
remainder of the two input arrays. We have therefore transformed a large
instance of merging to a smaller instance.
Putting it all together, we have the MergeSort algorithm shown in
Figure 10.2. Note that in the Merge function, when the loop terminates,
either i > m or j > n. Hence, either A[i..m] or B[j..n] is empty. As a
result, only one of the two calls to Copy (see Figure 1.18 on page 22 for its
specification) will have any effect.
It is easily seen that Merge runs in Θ(m + n) time. Therefore, the time
required for MergeSort excluding the recursive calls is in Θ(n). For n > 1
a power of 2, the following recurrence gives the worst-case running time of
MergeSort:
f (n) ∈ 2f (n/2) + Θ(n).
From Theorem 3.32, f (n) ∈ Θ(n lg n).

10.3 Quick Sort

Though merge sort and heap sort both run in Θ(n lg n) time in the worst
case, another divide-and-conquer algorithm is more commonly used when
stability is not required. As was suggested in Exercise 2.17, sorting can be
reduced to the Dutch national flag problem, which was introduced in Section
2.4. We first select from the array to be sorted a pivot element p, which we
use to determine the colors of the elements, as follows:

• If x.Key() < p.Key(), then x is red.

• If x.Key() = p.Key(), then x is white.

• If x.Key() > p.Key(), then x is blue.

CHAPTER 10. DIVIDE AND CONQUER 338

Figure 10.2 The MergeSort algorithm

Precondition: A[1..n] is an array of Keyed items.

Postcondition: A[1..n] is a permutation of its original values sorted in
nondecreasing order. If initially, x = A[i] and y = A[j], where x.Key() =
y.Key() and i < j, then x precedes y in the result.
MergeSort(A[1..n])
if n > 1
m ← ⌊n/2⌋; MergeSort(A[1..m]); MergeSort(A[m + 1..n])
B[1..n] ← Merge(A[1..m], A[m + 1..n])
Copy(B[1..n], A[1..n])

Precondition: A[1..m] and B[1..n] are arrays of Keyed items sorted in

nondecreasing order.
Postcondition: Returns an array C[1..m + n] containing the elements of
A and B in nondecreasing order, maintaining the same order as in the input
arrays. If A[i].Key() = B[j].Key(), then A[i] precedes B[j] in C.
Merge(A[1..m], B[1..n])
C ← new Array[1..m + n]; i ← 1; j ← 1; k ← 1
// Invariant: C[1..k − 1] contains A[1..i − 1] and B[1..j − 1] in correct
// order.
while i ≤ m and j ≤ n
if A[i].Key() ≤ B[j].Key()
C[k] ← A[i]; i ← i + 1
else
C[k] ← B[j]; j ← j + 1
k ←k+1
Copy(A[i..m], C[k..k + m − i]); Copy(B[j..n], C[k..k + n − j])
return C
CHAPTER 10. DIVIDE AND CONQUER 339

By solving the resulting Dutch national flag problem, we will have parti-
tioned the array into three sections:
• The first section consists of all elements with keys less than p.Key().

• The second section consists of all elements with keys equal to p.Key().

• The third section consists of all elements with keys greater than p.Key().
By sorting the first and third sections, we will have sorted the array. This
general strategy is known as quick sort.
Following the divide-and-conquer paradigm, we would like to select the
pivot so that after the array has been partitioned, the first and third sections
have roughly the same number of elements. Thus, the median element would
be a good choice for p. As we will show in the next section, it is possible to
find the median in Θ(n) time in the worst case. We saw in Section 3.6 that
the Dutch national flag problem can be solved in Θ(n) time. Because each
of the two subproblems is at most half the size of the original problem, we
can bound the running time of this sorting algorithm with the recurrence

f (n) ∈ 2f (n/2) + Θ(n).

From Theorem 3.32, f (n) ∈ Θ(n lg n).

However, it turns out that the overhead of choosing the median as the
pivot is too expensive in practice, so that heap sort, for example, outperforms
this algorithm. On the other hand, choosing an arbitrary element, such as
the first, degrades the worst case performance. For example, suppose that
the input array is already sorted and that all keys are distinct. If we always
choose the first element as the pivot, then we always choose the smallest
element. As a result, one of the two subproblems is empty, and the other
contains all but one of the original elements. Because the empty subproblem
can be sorted in Θ(1) time, we can describe the running time for such a case
with the following recurrence:

f (n) ∈ f (n − 1) + Θ(n).

From Theorem 3.31, f (n) ∈ Θ(n2 ), so that the running time for this algo-
rithm is in Ω(n2 ) in the worst case. Observing that each element is chosen
as a pivot at most once, we can easily see that O(n2 ) is an upper bound
on the running time, so that the algorithm runs in Θ(n2 ) time in the worst
case.
It turns out that the versions of quick sort used most frequently do, in
fact, run in Θ(n2 ) time in the worst case. However, choosing the first element
CHAPTER 10. DIVIDE AND CONQUER 340

(or the last element) as the pivot is a bad idea, because an already-sorted
array yields the worst-case performance. Furthermore, the performance is
nearly as bad on a nearly-sorted array. To make matters worse, it is not
hard to see that when the running time is in Θ(n2 ), the stack usage is in
Θ(n). Because we often need to sort a nearly-sorted array, we don’t want
an algorithm that performs badly in such cases.
The above analyses illustrate that it is better for the pivot element to be
chosen to be near the median than to be near the smallest (or equivalently,
the largest) element. More generally, it illustrates why divide-and-conquer
is often an effective algorithm design strategy: when a problem is reduced
to multiple subproblems, it is best if these subproblems are the same size.
For quick sort, we need a way to choose the pivot element quickly in such a
way that it tends to be near the median.
One way to accomplish this is to choose the pivot element randomly.
This algorithm is shown in Figure 10.3. In order to make the presentation
easier to follow, we have specified the algorithm so that the array is indexed
with arbitrary endpoints.
Let us now analyze the expected running time of QuickSort on an
array of size n. We first observe that for any call in which lo < hi, the loop
will execute at least once. Furthermore, by an easy induction on n, we can
show that at most n + 1 calls have lo ≥ hi. Because each of these calls
requires Θ(1) time, a total of at most O(n) time is used in processing the
base cases. Otherwise, the running time is proportional to the number of
times the loop executes over the course of the algorithm.
Each iteration of the loop involves comparing one pair of elements. For a
given call to QuickSort, the pivot is compared to all elements currently in
the array, then is excluded from the subsequent recursive calls. Thus, once a
pair of elements is compared, they are never compared again on subsequent
loop iterations (though they may be compared twice in the same iteration —
once in each if statement). The total running time is therefore proportional
to the number of pairs of elements that are compared. We will only concern
ourselves with pairs of distinct elements, as this will only exclude O(n) pairs.
Let F [1..n] be the final sorted array, and let comp be a discrete random
variable giving the number of pairs (i, j) such that 1 ≤ i < j ≤ N and F [i]
is compared with F [j]. We wish to compute E[comp]. Let cij denote the
CHAPTER 10. DIVIDE AND CONQUER 341

Figure 10.3 The randomized QuickSort algorithm

Precondition: A[lo..hi] is an array of Keyed items, and lo and hi are

Ints.
Postcondition: A[lo..hi] is a permutation of its original values in nonde-
creasing order.
QuickSort(A[lo..hi])
if lo < hi
p ← A[RandomInteger(lo, hi)].Key()
r ← 0; w ← 0; b ← 0
// Invariant: r, w, b ∈ N, r + w + b ≤ hi − lo + 1, A[i].Key() < p
// for lo ≤ i < lo + r, A[i].Key() = p for hi − b − w < i ≤ hi − b, and
// A[i].Key() > p for hi − b < i ≤ hi.
while r + w + b < hi − lo + 1
j ← hi − b − w
if A[j].Key() < p
A[j] ↔ A[lo + r]; r ← r + 1
else if A[j].Key() = p
w ←w+1
else
A[j] ↔ A[hi − b]; b = b + 1
QuickSort(A[lo..lo + r − 1])
QuickSort(A[hi − b + 1..hi])

Precondition: i and j are integers, i ≤ j.

Postcondition: Returns an integer k such that i ≤ k ≤ j. Each value in
the range has an equal probability, independent of previous calls.
RandomInteger(i, j)
CHAPTER 10. DIVIDE AND CONQUER 342

event that F [i] is compared with F [j]. Then

 
X n X n
E[comp] = E  I(cij )
i=1 j=i+1
n
X n
X
= E[I(cij )]
i=1 j=i+1
Xn X n
= P (cij ).
i=1 j=i+1

We observe that F [i] is compared with F [j] iff one of them is in the
subarray being sorted when the other is chosen as the pivot. Furthermore,
two elements F [i] and F [j] are in the same subarray as long as no element
k such that
F [i].Key() ≤ F [k].Key() ≤ F [j].Key()
is chosen as the pivot. Thus, the probability that F [i] and F [j] are compared
is the probability that one of them is chosen as pivot before any other F [k]
satisfying the above inequality. Because there are at least j − i + 1 elements
F [k] satisfying the above inequality when j > i,
2
P (cij ) ≤ .
j−i+1
We therefore have
n X
X n
E[comp] = P (cij )
i=1 j=i+1
n X n
X 2
≤
j−i+1
i=1 j=i+1
n n−i+1
X X 1
=2 . (10.3)
j
i=1 j=2

The inner sum above is closely related to the harmonic series:

n
X 1
Hn = .
i
i=1

Tight bounds for Hn are given by the following theorem, whose proof is left
as an exercise.
CHAPTER 10. DIVIDE AND CONQUER 343

Theorem 10.1 For all n ≥ 1:

ln(n + 1) ≤ Hn ≤ 1 + ln n.

Applying Theorem 10.1 to inequality (10.3), we have

n n−i+1
X X 1
E[comp] ≤ 2
j
i=1 j=2
Xn
=2 (Hn−i+1 − 1)
i=1
Xn
≤2 ln(n − i + 1)
i=1
Xn
=2 ln i
i=1
∈ O(n lg n)

from Theorem 3.28. For an array of distinct elements, a similar analy-

sis shows that E[comp] ∈ Ω(n lg n); hence, the expected running time of
QuickSort on any array of n elements is in Θ(n lg n).
The expected-case analysis of QuickSort suggests that it would work
well in practice, and indeed, there are versions that outperform both heap
sort and merge sort. The most widely-used versions, however, are not ran-
domized. Instead of choosing the pivot element randomly, they use heuris-
tics, such as choosing the median of the first, middle, and last elements.
Such heuristics tend to choose pivot elements nearer to the median than
those chosen randomly. Furthermore, they typically involve less overhead
than generating a random (or pseudorandom) number. Though these varia-
tions have worst-case running times in Θ(n2 ), the inputs that result in poor
performance are so pathological that they rarely occur in practice.

10.4 Selection
In Section 1.1, we introduced the selection problem. Recall that this problem
is to find the kth smallest element of an array of n elements. We showed
that it can be reduced to sorting. Using either heap sort or merge sort,
we therefore have an algorithm for this problem with a running time in
Θ(n lg n). In this section, we will improve upon this running time.
CHAPTER 10. DIVIDE AND CONQUER 344

Section 2.4 shows that the selection problem can be reduced to the Dutch
National Flag problem and a smaller instance of itself. This reduction is
very similar to the reduction upon which quick sort is based. Specifically,
we choose a pivot element p and solve the resulting Dutch national flag
problem as we did for the quick sort reduction. Let r and w denote the
numbers of red items and white items, respectively. We then have three
cases:

• If r ≥ k, we return the kth smallest red item.

• If r < k and r + w ≥ k, we return p.

• If r + w < k, we return the (k − r − w)th smallest blue element.

Due to the similarity of this algorithm to quick sort, some of the same
problems arise in choosing the pivot element appropriately. For example, if
we always use the first element as the pivot, then selecting the nth smallest
element in a sorted array of n distinct elements always results in a recursive
call with all but one of the original elements. As we saw in Section 10.3, this
yields a running time in Θ(n2 ). On the other hand, it is possible to show
that selecting the pivot at random yields an expected running time in Θ(n)
— the details are left as an exercise.
Our goal is to construct a deterministic algorithm with worst-case run-
ning time in O(n). As we saw in Section 10.3, quick sort achieves a better
asymptotic running time if the median is chosen as the pivot. It stands to
reason that such a choice might be best for the selection algorithm. We
mentioned in Section 10.3 that it is possible to find the median in O(n)
time. The way to do this is to use our linear-time selection algorithm to find
the ⌈n/2⌉nd smallest element. However, this doesn’t help us in designing
the linear-time selection algorithm because the reduction is not to a smaller
instance.
Instead, we need a way to approximate the median well enough so that
the resulting algorithm runs in O(n) time. Consider the following strategy
for approximating the median. First, we arrange the n elements into an
M × ⌊n/M ⌋ array, where M is some fixed odd number. If n is not evenly
divisible by M , we will have up to M − 1 elements that will not fit in the
array — we will ignore these elements for now. Suppose we sort each column
in nondecreasing order. Further suppose that we order the columns (keeping
each column intact) so that the middle row is in nondecreasing order. We
then select an element in the center of the array as the pivot p (see Figure
10.4).
CHAPTER 10. DIVIDE AND CONQUER 345

Figure 10.4 Finding an approximate median

≤p

≥p

By choosing p in this fashion, we ensure that all elements above and to

the left of p are no greater than p, and that all elements below and to the
right of p are no less than p. In other words, no more than about 3/4 of the
elements can be greater than p, and no more than about 3/4 of the elements
can be less than p. Thus, if we can find this p in O(n) time, we have the
following recurrence giving the running time of the algorithm:
f (n) ∈ f (⌊3n/4⌋) + Θ(n).
Unfortunately, Theorem 3.32 does not apply to this recurrence, as we
would need b = 4/3, which is not a natural number. However, given Theorem
3.32, we might suspect that f (n) ∈ Θ(n), as this is the solution yielded by
the theorem if we could use b = 4/3. We will soon show that this is, in fact,
the case.
Let us now consider the time needed to find p. Sorting M elements using
either heap sort or merge sort uses Θ(M lg M ) time. Because M is a fixed
constant, however, M lg M is also a fixed constant, so that the time is in
Θ(1). This must be done for each of the ⌊n/M ⌋ columns, so that the time
to sort all of the columns is in O(n). However, the time to sort the ⌊n/M ⌋
elements in the middle row is in Θ(n lg n). On the other hand, we really
don’t need to sort the middle row in order to find p — we only need to
find the median of this row. If M > 1, this row has strictly fewer than n
elements, so that finding its median is a smaller instance of the selection
problem.
If we use this technique for finding the median, we need two recursive
calls – one to find the median of ⌊n/M ⌋ elements, and one to solve the
CHAPTER 10. DIVIDE AND CONQUER 346

smaller selection problem after the Dutch national flag algorithm has been
applied. This latter recursive call has no more than about 3n/4 elements.
The recurrence describing the running time is therefore of the form

f (n) ∈ f (⌊3n/4⌋) + f (⌊n/M ⌋) + Θ(n), (10.4)

for some odd number M .

We don’t have a theorem that applies to the above recurrence. We can
gain some intuition by comparing this recurrence with recurrences of the
form
g(n) ∈ ag(⌊n/b⌋) + Θ(n),
where b is an integer greater than 1. From Theorem 3.32, g(n) ∈ Θ(n) iff
a < b, or equivalently, iff a/b < 1. If a is a positive integer, this condition is
equivalent to
a
X 1
< 1.
b
i=1
The following theorem generalizes this condition to a sum of several different
positive real numbers.

Theorem 10.2 Let n0 ≥ 1 and m ≥ 1 be integers, and let c1 , . . . , cm be

positive real numbers such that
m
X
ci < 1.
i=1

Let f : N → R≥0 be an eventually nondecreasing function satisfying

m
X
f (n) ∈ f (⌊ci n⌋) + X(n)
i=1

whenever n ≥ n0 , where X is either O, Ω, or Θ. Then f (n) ∈ X(n).

Proof: If X is Ω, clearly we have f (n) ∈ Ω(n). In what follows, we will

show that if X is O, then f (n) ∈ O(n). It will then follow that if X is Θ,
then f (n) ∈ Θ(n).
Suppose X is O. Then for some natural number n1 and some positive
real number a, we have
m
X
f (n) ≤ f (⌊ci n⌋) + an
i=1
CHAPTER 10. DIVIDE AND CONQUER 347

whenever n ≥ n0 and n ≥ n1 . By Exercise 3.8, it is sufficient to show that

for some positive real number b, f (n) ≤ bn whenever n > 0.
The technique we will use to prove this fact is called constructive induc-
tion. With this technique, we use the constant b as if it had already been
fixed. As the proof progresses, we will need to assume certain constraints
on b. Once the proof is finished, if the assumed constraints are consistent,
we can then fix a value for b that satisfied all of the constraints.

Induction Hypothesis: Assume that for some n, if 0 < k < n, then

f (k) ≤ bk.

Induction Step: First, we will assume that n is large enough to apply

the recurrence, then to apply the induction hypothesis to each term in the
resulting summation. If n ≥ max(n0 , n1 ),
m
X
f (n) ≤ f (⌊ci n⌋) + an.
i=1

In order to be able to apply the induction hypothesis to each f (⌊ci n⌋), we

need ⌊ci n⌋ > 0. Therefore we assume that n ≥ n2 , where
n2 ≥ max{n0 , n1 , 1/ci | 1 ≤ i ≤ m}.
Let
m
X
c= ci .
i=1
By the Induction Hypothesis,
m
X
f (n) ≤ b⌊ci n⌋ + an
i=1
m
X
≤ bn ci + an
i=1
= bcn + an
= (bc + a)n.
Thus, f (n) ≤ bn if
bc + a ≤ b
a ≤ b(1 − c)
a
≤ b,
1−c
CHAPTER 10. DIVIDE AND CONQUER 348

as 1 − c > 0.

Base: We must still show the claim for 0 < n < n2 . In other words, we
need b ≥ f (n)/n for 0 < n < n2 . We can satisfy this constraint and the one
above if b = max{a/(1 − c), f (n)/n | 0 < n < n2 } (note that because this
set is finite and nonempty, it must have a maximum element).

Returning to recurrence 10.4, we see that Theorem 10.2 applies if M > 4.

Thus, if we set M = 5, we have f (n) ∈ O(n). The entire algorithm is shown
in Figure 10.5.
Now that we have described the algorithm precisely, let us analyze its
running time more carefully to be sure that it is, in fact in Θ(n). It is easily
seen that the running time is in Ω(n). We need for the recurrence
f (n) ∈ f (⌊n/5⌋) + f (⌊3n/4⌋) + O(n) (10.5)
to give an upper bound on the running time of the algorithm for sufficiently
large n. Clearly, the number of elements in the first recursive call is ⌊n/5⌋.
Furthermore, if we ignore the recursive calls, the time needed is in O(n) (note
that the time needed to sort 5 elements is bounded by a constant because
the number of elements is constant). However, the number of elements in
the second recursive call is not always bounded above by ⌊3n/4⌋. Consider,
for example, the array A[1..13] with A[i] = i for 1 ≤ i ≤ 13. The values 3
and 8 will be placed in T [1] and T [2], respectively. The value assigned to
p will therefore be 3. If k > 3, ten elements will be passed to the second
recursive call, but ⌊3 · 13/4⌋ = 9.
Returning to Figure 10.4, and taking the number of rows to be 5, we
see that for each two columns (ten elements) that we add, at least three
elements are excluded from the second recursive call. For example, when
5 ≤ n ≤ 14, at least three elements are excluded, and when 15 ≤ n ≤ 24, at
least six elements are excluded. In general, at least 3i elements are excluded
if 10i − 5 ≤ n ≤ 10i + 4. We need to find the largest value of n such that
3i < n/4, if such an n exists. If we can do so, then for all larger values of
n, the number of elements in the second recursive call will be no more than
3n/4. Because the number of elements must be an integer, it will then be
bounded above by ⌊3n/4⌋.
We therefore need to find the largest i such that
3i < (10i + 4)/4
12i < 10i + 4
i < 2.
CHAPTER 10. DIVIDE AND CONQUER 349

Figure 10.5 A linear-time algorithm for the selection problem

LinearSelect(A[1..n], k)
if n ≤ 4
Sort(A[1..n])
return A[k]
else
m ← ⌊n/5⌋; T ← new Array[1..m]
for i ← 1 to m
Sort(A[5i − 4..5i]); T [i] ← A[5i − 2]
p ← LinearSelect(T [1..m], ⌈m/2⌉); r ← 0; w ← 0; b ← 0
// Invariant: r, w, b ∈ N, r + w + b ≤ n, and A[i] < p for 1 ≤ i ≤ r,
// A[i] = p for n − b − w < i ≤ n − b, and A[i] > p for n − b < i ≤ n.
while r + w + b < n
j ←n−b−w
if A[j] < p
r ← r + 1; A[j] ↔ A[r]
else if A[j] = p
w ←w+1
else
A[j] ↔ A[n − b]; b = b + 1
if r ≥ k
return LinearSelect(A[1..r], k)
else if r + w ≥ k
return p
else
return LinearSelect(A[r + w + 1..n], k − r − w)
CHAPTER 10. DIVIDE AND CONQUER 350

Figure 10.6 Multiplication and division functions for use with the BigNum
ADT defined in Figure 4.18 on page 146

Precondition: u and v are BigNums.

Postcondition: Returns a BigNum representing the product of the values
of u and v.
Multiply(u, v)
Precondition: u is a BigNum and v is a positive BigNum.
Postcondition: Returns a BigNum representing the value ⌊u/v⌋.
Divide(u, v)

The largest such n is therefore 10 + 4 = 14. Then for all n ≥ 15, recurrence
(10.5) gives an upper bound on the running time of LinearSelect. From
Theorem 10.2, the running time is in O(n), and hence in Θ(n).
Various performance improvements can be made to LinearSelect. For
example, if n = 5, there is no reason to apply the Dutch national flag
algorithm after sorting the array — we can simply return A[k]. In other
words, it would be better if the base case included n = 5, and perhaps some
larger values as well. Furthermore, sorting is not the most efficient way to
solve the selection problem for small n. We explore some alternatives in the
exercises.
Even with these performance improvements, however, LinearSelect
does not perform nearly as well as the randomized algorithm outlined at
the beginning of this section. Better still is using a quick approximation
of the median, such as finding the median of the first, middle, and last
elements, as the value of p. This approach yields an algorithm whose worst-
case running time is in Θ(n2 ), but which typically performs better than even
the randomized algorithm.

10.5 Integer Division

Exercise 4.14 on page 145 discussed an implementation of a BigNum ADT
(specified in Figure 4.18, page 146) for arbitrary-precision natural num-
bers. This ADT is rather limited, in that its only arithmetic operations are
addition and subtraction. Figure 10.6 specifies multiplication and division Figure 10.6
corrected
functions for operating on BigNums. We leave it as an exercise to show that 4/12/12.
CHAPTER 10. DIVIDE AND CONQUER 351

the polynomial multiplication algorithm of Section 10.1 can be adapted to

form a BigNum multiplication algorithm that runs in Θ(nlg 3 ) time, where
n is the number of bits in the product. In this section and the next, we
consider implementations of Divide.
Let us first consider the familiar long division algorithm from elementary
school. Suppose the dividend u has m digits and the divisor v has n digits.
We begin by finding the smallest prefix of the significant digits of u that Digits (or bits)
are significant if
gives us a number no smaller than v. If there is no such prefix, the quotient they are not to
is 0. Otherwise, the next step is to obtain an approximation for the most the left of the
significant digit of the result. Let k be the number of digits in the prefix leftmost nonzero
digit (or bit,
formed above. We obtain the approximation by dividing the k − n + 1 most respectively).
significant digits of u by the most significant digit of v. (Note that because The most
significant digit
k will always be either n or n + 1, k − n + 1 will be either 1 or 2.) This (or bit) is the
quotient is our approximation of the first significant digit of the result. leftmost of the
significant digits
Because we only used one digit of v in approximating the first signif- (or bits,
icant digit of the result, this approximation may be too large. We deter- respectively).
mine whether this is the case by multiplying this approximation by v, then
comparing the product with the prefix of u. If the product is larger, we
repeatedly subtract 1 from the approximation and recompute the product
until the product is no larger than the prefix of u. The resulting digit is the
first significant digit of the result.
Having obtained the first significant digit of the result, we need to obtain
the rest of the result. We begin by subtracting the last product obtained
above from the prefix of u. We then append the rest of u to this difference.
Finally, we obtain the remaining digits of the quotient by dividing the result-
ing value by v. If we ensure that the result of this last division has exactly
m − k digits (by padding with zeros if necessary), then the digits we obtain
complete the quotient. We have therefore reduced the division problem to a
smaller instance of itself. The reduction is not quite a transformation, but
it can easily be expressed as a loop.
This algorithm can easily be applied to the division of an m-bit binary
number u by an n-bit binary number v. In fact, we really don’t need to
approximate the first significant bit — it can’t be a 0, so it must be a 1. We
do, however, need to determine the prefix of u as described above. We then
subtract v from the prefix and append the remainder of u. We obtain the
remaining bits by dividing this value by v and inserting zeros if necessary.
In the above algorithm, we perform a Θ(n)-bit subtraction for each 1
bit of the quotient. Furthermore, each subtraction is preceded by at least
one Θ(n)-bit comparison. Because the number of 1 bits in the quotient
may be as many as m − n + 1, the worst-case number of comparisons and
CHAPTER 10. DIVIDE AND CONQUER 352

subtractions of Θ(n)-bit numbers is (m−n+1). The running time is therefore

in Θ(n(m − n)), which is worse than what we would like.
We can instead group the bits of u and v into digits of some radix r,
where r is a power of 2. However, for any fixed radix r, this decreases the
number of comparisons by at most a constant factor, so that the asymptotic
running time does not improve. Applying the divide-and-conquer principle,
we might try breaking v into two digits. For now, let’s assume the number of
bits n in v is even. Our radix is therefore 2n/2 . To obtain an approximation
of the first digit of the quotient, we recursively divide the first one or two
digits of u by the first digit of v.
If n is odd and greater than 1, we can multiply both u and v by 2 without
changing the quotient. These multiplications have the effect of appending a
zero as the least significant bit, so that the resulting length is even. Because
n is greater than 1, dividing the length of the resulting divisor by 2 yields
a smaller subproblem. For n = 1, v = 1, so the quotient is simply u. We
have therefore reduced the division problem to smaller instances of itself.
The resulting algorithm is shown in Figure 10.7. We assume the existence
of two constants zero and one, which refer to BigNums with values of 0 and
1, respectively.
We will now analyze the running time of DivideDC. We begin with the
while loop. In order to analyze this loop, we need to know how large prod,
rem, and approx might be. From the invariant of the main loop, rem ≤ v −1
at the top of the main loop. Before the while loop is executed, the value
of rem is multiplied by 2n/2 , and next is added. Because next contains at
most n/2 significant bits, next < 2n/2 . Thus, when the while loop executes,
rem < v2n/2 . Because v contains n significant bits, rem contains at most
3n/2 significant bits. Likewise, it is not hard to show that at the beginning
of the while loop, approx contains at most n/2 + 1 significant bits, and that
prod contains at most 3n/2 + 1 significant bits. The body of the while loop
therefore runs in Θ(n) time in the worst case.
In order to get a tight bound on the number of iterations of the while
loop, we need a tighter bound on approx. In particular, we need to know
how close approx is to ⌊rem/v⌋. Let r = ⌊rem × 2−n/2 ⌋. We first observe
that
$ %
r × 2n/2

r
=
vFirst + 1 (vFirst + 1) × 2n/2

rem
= ,
(vFirst + 1) × 2n/2
CHAPTER 10. DIVIDE AND CONQUER 353

Figure 10.7 Divide-and-conquer implementation of Divide, specified in

Figure 10.6

DivideDC(u, v)
m ← u.NumBits(); n ← v.NumBits()
if n = 1
return u
else
if n mod 2 = 1
u ← u.Shift(1); m ← m + 1
v ← v.Shift(1); n ← n + 1
digLen ← n/2; numDig ← ⌈m/digLen⌉
qLen ← digLen × (numDig − 1)
qBits ← new Array[0..qLen − 1]
vFirst ← new BigNum(v.GetBits(digLen, digLen))
rem ← new BigNum(u.GetBits(qLen, digLen))
// Invariant: rem < v and
// v × new BigNum(qBits[i + digLen..qLen − 1]) + rem =
// new BigNum(u.GetBits(i + digLen, m − i − digLen)).
for i ← qLen − digLen to 0 by −digLen
next ← new BigNum(u.GetBits(i, digLen))
rem ← rem.Shift(digLen).Add(next)
if rem.CompareTo(v) < 0
qDig ← zero.GetBits(0, digLen)
else
approx ← DivideDC(rem.Shift(−digLen), vFirst)
prod ← Multiply(v, approx)
// Invariant: prod = v × approx, approx ≥ ⌊rem/v⌋, and the
// invariant for the outer loop.
while prod.CompareTo(rem) > 0
approx ← approx.Subtract(one)
prod ← prod.Subtract(v)
qDig ← approx.GetBits(0, digLen)
Copy(qDig[0..digLen − 1], qBits[i..i + digLen − 1])
return new BigNum(qBits)
CHAPTER 10. DIVIDE AND CONQUER 354

because the n/2 low-order bits of the numerator do not affect the value of
the expression. Furthermore, the right-hand side above is no larger than
⌊rem/v⌋. Thus,
j r k r

approx − ⌊rem/v⌋ ≤ −
vFirst vFirst + 1
r r − vFirst
≤ −
vFirst vFirst + 1
r(vFirst + 1) − vFirst(r − vFirst)
=
vFirst(vFirst + 1)
r + vFirst2
= .
vFirst(vFirst + 1)

Now because rem < v2n/2 , it follows that r < vFirst×2n/2 . We therefore
have
r + vFirst2
approx − ⌊rem/v⌋ ≤
vFirst(vFirst + 1)
vFirst × 2n/2 + vFirst2
<
vFirst(vFirst + 1)
2n/2 + vFirst
=
vFirst + 1
2n/2 vFirst
= + .
vFirst + 1 vFirst + 1
Because vFirst contains n/2 significant bits, its value must be at least
2n/2−1 . The value of the first term on the right-hand side above is therefore
strictly less than 2. Clearly, the value of the second term is strictly less than
1, so that the right-hand side is strictly less than 3. Because the left-hand
side is an integer, its value must therefore be at most 2. It follows from
the while loop invariant that the loop terminates when approx = ⌊rem/v⌋.
Because this loop decrements approx by 1 each iteration, it must iterate at
most twice. Its running time is therefore in Θ(n).
It is now easily seen that, excluding the recursive call, the running time
of the body of the main loop is dominated by the running time of the
multiplication. Because the result of the multiplication contains at most
3n/2 + 1 significant bits, this multiplication can be done in Θ(nlg 3 ) time
using the multiplication algorithm suggested at the beginning of this section.
CHAPTER 10. DIVIDE AND CONQUER 355

If m ≥ n, the number of iterations of the main loop is easily seen to be

numDig − 1 = ⌈m/digLen⌉ − 1

m
= −1
n/2
= ⌈2m/n⌉ − 1.

Thus, the running time of the main loop, excluding the recursive call, is in

Θ(nlg 3 m/n) = Θ(mnlg 3−1 ).

We now observe that for even n, there are in the worst case ⌈2m/n⌉ − 1
recursive calls. For odd n, the worst-case number of recursive calls is
⌈2(m + 1)/(n + 1)⌉ − 1. The resulting recurrence is therefore quite com-
plicated. However, consider the parameters of the recursive call. We have
already shown that rem < v2n/2 , and vFirst = ⌊v2−n/2 ⌋. This recursive call
therefore divides a value strictly less than v by ⌊v2−n/2 ⌋. Thus, in any of
these calls, the dividend is less than the divisor plus 1, multiplied by 2n ,
where n is the number of bits in the divisor. In addition, it is easily seen
that the dividend is never less than the divisor. Furthermore, if these rela-
tionships initially hold for odd n, they hold for the recursive call in this case
as well. We therefore will first restrict our attention to this special case.
Let n, the number of significant bits in v, be even. If

v ≤ u < (v + 1)2n ,

then m, the number of significant bits in u, is at most 2n. The number of

iterations of the outer loop is therefore at most

⌈2m/n⌉ − 1 ≤ ⌈4n/n⌉ − 1
= 3.

Because each iteration may contain a recursive call, this suggests that there
are a total of at most 3 recursive calls. However, note that whenever a
recursive call is made, the dividend is no less than the divisor, so that a
nonzero digit results in the quotient. Suppose the first of the three digits
of the quotient is nonzero. Because the first n bits of u are at most v, the
only possible nonzero result for the first digit is 1. The remainder of the
quotient is then formed by dividing a value strictly less than 2n by v, which
is at least 2n−1 . This result is also at most 1, so that the second digit must
be 0. We conclude that no more than two recursive calls are ever made. In
each of these recursive calls, the divisor has n/2 bits.
CHAPTER 10. DIVIDE AND CONQUER 356

If n is odd and greater than 1, we increase the number of bits in v by

1. The above reasoning then applies to n + 1, where n denotes the original
number of bits in v. We can therefore express the overall running time in
terms of n via the recurrence

f (n) ∈ 2f (⌈n/2⌉) + Θ(nlg 3 )

for n > 1. Applying Theorem 3.32, we see that f (n) ∈ Θ(nlg 3 ).

Let us now turn to the more general case. If m < n, it is easily seen
that the running time is in Θ(n). If m ≥ n, as we have already shown, the
worst-case number of recursive calls is in Θ(m/n), and the overhead is in
Θ(mnlg 3−1 ). Because each of these recursive calls satisfies the special case
analyzed above, each runs in Θ(nlg 3 ) time. Thus, the overall running time
is in Θ(mnlg 3−1 ).
If we wish to express the running time in terms of the number of bits in
the larger of the two operands, it is easily seen that the worst case occurs
when m is larger, but n is a fixed fraction of m. If N denotes the number of
bits in the larger operand, we then see that the running time is in Θ(N lg 3 ),
which is asymptotically the same as multiplication. Furthermore, it is not
hard to see that if we can improve the running time of multiplication to
O(N 1+ǫ ) for any fixed positive ǫ, the running time of division will also be in
O(N 1+ǫ ) (or O(mnǫ )).
In a later chapter, we will show that the running time of multiplication
can be improved to O(N lg N lg lg N ). When we use this running time, the
recurrence for the case in which

v ≤ u < (v + 1)2n ,

becomes
f (n) ∈ 2f (⌈n/2⌉) + O(n lg n lg lg n)
for n > 1. Applying Theorem 3.32 to this recurrence yields

f (n) ∈ O(n lg2 n lg lg n).

The running time of the division algorithm is then in O(N lg2 N lg lg N ),

which is slightly worse than the running time of the multiplication algo-
rithm used. In the next section, we will design a division algorithm whose
asymptotic running time matches that of multiplication even for the asymp-
totically fastest known multiplication algorithm.
CHAPTER 10. DIVIDE AND CONQUER 357

10.6 * Newton’s Method

We can reduce division to multiplication in a straightforward way if we
can compute a reciprocal. A reciprocal of an arbitrary positive integer is
a fraction that may or may not have a finite binary representation. We
therefore will have to settle for an approximation. In order to simplify the
discussion of fixed-point fractions, it helps to scale the value of the divisor
v to an appropriate range. Specifically, suppose v consists of n bits; i.e.,
2n−1 ≤ v < 2n . Then
juk
1

−n
= u2 .
v v2−n
Note that 1/2 ≤ v2−n < 1. Thus, if we can compute a close fixed-point
approximation for the reciprocal of a value y such that 1/2 ≤ y < 1, we can
reduce the integer division problem to the integer multiplication problem.
Note that due to the given range of y, the binary encoding of y has no bits to
the left of the radix point, and the first bit to the right is a 1. Furthermore,
because 1 < 1/y ≤ 2, we can approximate 1/y with a value z such that
1 ≤ z < 2; i.e., z has a single 1 bit to the left of the radix point. We can
therefore use BigNums to represent both y and 1/y with the interpretation
that each contains a radix point at the appropriate position.
Because multiplying an approximation of the reciprocal of v2−n by u2−n
gives only an approximation of ⌊u/v⌋, we may need to correct our result.
Suppose we can approximate the reciprocal to enough accuracy that the
result of the multiplication is a value q such that
u
q − ≤ 1.

v
Then it is not hard to see that
j u k
⌊q⌋ − ≤ 1.

v
Then if v⌊q⌋ ≤ u−v, we know that the actual quotient is ⌊q⌋+1. If v⌊q⌋ > u,
we know that the quotient is ⌊q⌋ − 1. Otherwise, the quotient is ⌊q⌋.
Suppose an error of ǫ is introduced in approximating the reciprocal.
Then we need

−n
u2 1 u
−n
+ ǫ − ≤ 1
v2 v
u u
+ ǫu2−n − ≤ 1

v v
|ǫ| ≤ 2n /u.
CHAPTER 10. DIVIDE AND CONQUER 358

Figure 10.8 Partial implementation of Divide, specified in Figure 10.6,

using an approximate reciprocal

DivideRecip(u, v)
if u.CompareTo(v) < 0
return zero
else
m ← u.NumBits(); n ← v.NumBits()
r ← Reciprocal(v, m − n)
q ← Multiply(u, r).Shift(1 − r.NumBits() − n);
prod ← Multiply(v, q)
if prod.CompareTo(u.Subtract(v)) ≤ 0
q ← q.Add(one)
else if prod.CompareTo(u) > 0
q ← q.Subtract(one)
return q

Precondition: y refers to a nonzero BigNum, and k refers to a natural

number.
Postcondition: Returns a BigNum z such that

1−z.NumBits() 1
− −y.NumBits() ≤ 2−k .

z2
y2

Reciprocal(y, k)

Suppose u consists of m bits, so that u < 2m . Then we can ensure that

the approximation of u/v differs from the actual value of u/v by no more
than 1 if our approximation of the reciprocal of v2−n differs from the actual
reciprocal by at most 2n−m .
The resulting algorithm is shown in Figure 10.8. We handle the case
in which u < v separately in order to ensure that the precondition for
u.Subtract(v) is met. As in the previous section, we use constants zero
and one, which refer to BigNum representing 0 and 1, respectively. It is
easily seen that the running time is simply the time for Reciprocal plus the
time to do the two multiplications. The time to do the first multiplication
depends on the size of the value returned by Reciprocal. Because the
accuracy of the approximation is 2n−m , we would expect the value to be not
CHAPTER 10. DIVIDE AND CONQUER 359

Figure 10.9 The Newtonian iteration

x0 x1 f (x)

much more than m − n significant bits.

In the remainder of this section, we will consider how to implement the
Reciprocal function specified in Figure 10.8. The technique we apply is
Newton’s method for approximating a root of a function. Let I be some
interval of the real numbers, and suppose f : I → R has at least one root
— a value x ∈ I such that f (x) = 0. For example, if y is a fixed positive
real number, the function f (x) = 1/x − y over R>0 has exactly one root,
namely, x = 1/y. Newton’s method is an iterative approach to finding an
approximation of a root of f .
Newton’s method begins with an initial estimate x0 of the root. If f (x0 )
is not sufficiently close to 0, a better approximation is found using the deriva-
tive of f , which we will denote by f ′ . Recall that f ′ (x0 ) gives the slope of the
line tangent to f at x0 (see Figure 10.9). We can easily find the intersection
x1 of this line with the x-axis, and for many functions, this intersection will
be a better approximation to the root than the initial estimate. We then
apply Newton’s method using x1 as the initial estimate. For many functions,
this approach is guaranteed to approach a root very quickly. As we will see,
the function f (x) = 1/x − y is such a function.
The line tangent to f at x0 has slope f ′ (x0 ) and includes the point
(x0 , f (x0 )). To find its x-intercept, we need to go to the left of x0 a distance
of f (x0 )/f ′ (x0 ) (or if this value is negative, we go to the right a distance of
CHAPTER 10. DIVIDE AND CONQUER 360

−f (x0 )/f ′ (x0 )). The new estimate x1 is therefore given by

x1 = x0 − f (x0 )/f ′ (x0 ).

If f (x) = 1/x − y, then f ′ (x) = −x−2 . The new estimate is therefore

1/x0 − y
x1 = x0 −
−x−2
0
= x0 + x0 − yx20
= 2x0 − yx20 .

In order to see how quickly the Newtonian iteration converges to 1/y,

suppose we have an estimate x0 = (1 + ǫ)/y, where ǫ is some real number.
Applying the iteration, we have

x1 = 2x0 − yx20
1+ǫ 2

2(1 + ǫ)
= −y
y y
2 + 2ǫ − 1 − 2ǫ − ǫ2
=
y
1−ǫ 2
= . (10.6)
y

Thus, each iteration squares the error term ǫ. For 1/2 ≤ y < 1, it is not
hard to show that an initial estimate of 3/2 yields

−1/4 ≤ ǫ < 1/2;

hence, the number of bits of accuracy doubles with each iteration.

Although this technique does converge rapidly to 1/y, the number of
iterations depends on the degree of accuracy we require. Unfortunately, each
iteration requires a multiplication by y, so that the time required cannot be
proportional to the time for a single multiplication of values having roughly
the same size as y. However, note that successive iterations give successively
better approximations. For the earlier approximations, which will probably
not be very accurate anyway, we need not use all of the bits of y in the
computation.
This suggests the following approach. Suppose we need an approxima-
tion that differs from the actual reciprocal by no more than 2−k . We will
use k as the size of this problem instance. If k is not too small, we first
CHAPTER 10. DIVIDE AND CONQUER 361

solve a smaller instance in order to obtain a less-accurate approximation.

The accuracy that we require of this approximation needs to be such that a
single application of the Newtonian iteration will yield an accuracy of within
2−k . In applying this iteration, we only use as many bits of y as we need in
order to ensure the required accuracy. Finally, in order to keep the number
of bits in the approximation from growing too rapidly, we return only as
many bits as we need to ensure the required accuracy.
Let α ∈ R denote the absolute error of some estimate; i.e., our estimate is
1/y + α. Let β ∈ R≥0 denote the absolute error introduced by truncating y,
so that the value we use for y in the iteration is y − β. Finally, let γ ∈ R≥0
denote the absolute error introduced by truncating the result. The value
computed by the iteration is therefore
2
1 1 1 β 2αβ
2 + α − (y − β) +α −γ = + 2 + + α2 β − yα2 − γ.
y y y y y

We need for this value to differ from 1/y by at most 2−k ; i.e., we need

β 2αβ
+ α β − yα − γ ≤ 2−k .
2 2

+ (10.7)
y2 y
Note that because y > 0, β ≥ 0, and γ ≥ 0, all terms except the second
are always nonnegative. In order to ensure that the inequality holds when
the value inside the absolute value bars is nonnegative, we can therefore
ignore the last two terms. We therefore need
β 2αβ
2
+ + α2 β ≤ 2−k .
y y
If we replace α by |α| in the above inequality, the left-hand side does not
decrease. For fixed α and β, the resulting left-hand side is maximized when
y is minimized. Setting y to its minimum possible value of 1/2, it therefore
suffices to ensure that

4β + 4|α|β + α2 β ≤ 2−k .

In order to keep the first term sufficiently small, we need β < 2−k−2 . In
order to leave room for the other two terms, let us take β ≤ 2−k−3 . In other
words, we will use the first k + 3 bits of y in applying the iteration. Then
as long as |α| ≤ 1/2, we have

4β + 4αβ + α2 β ≤ 2−k−1 + 2−k−2 + 2−k−5

≤ 2−k .
CHAPTER 10. DIVIDE AND CONQUER 362

Let us now consider the case in which the value inside the absolute value
bars in (10.7) is negative. We can now ignore the first and third terms. We
therefore need
2αβ
yα2 + γ − ≤ 2−k .
y
Here, we can safely replace α by −|α|. For fixed α, β, and γ in the resulting
inequality, the first term is maximized when y is maximized, but the third
term is maximized when y is minimized. It therefore suffices to ensure that
α2 + γ + 4|α|β ≤ 2−k .
Again taking β ≤ 2−k−3 , we only need |α| ≤ 2−(k+1)/2 and γ ≤ 2−k−2 . We
then have
α2 + γ + 4|α|β ≤ 2−k−1 + 2−k−2 + 2k−1−(k+1)/2
≤ 2−k ,
provided k ≥ 1.
We can satisfy the constraints on α and γ by finding an approximation
within 2−⌈(k+1)/2⌉ , and returning k + 3 bits of the result of applying the
iteration (recall that the result has one bit to the left of the radix point).
Note that if we take k as the size of the problem instance, we are reducing
the problem to an instance roughly half the original size. We therefore have
a divide-and-conquer algorithm.
In order to complete the algorithm, we need to handle the base cases.
Because ⌈(k + 1)/2⌉ < k only when k > 2, these cases occur for k ≤ 2. It
turns out that these cases are important for ensuring that the approximation
is at least 1 and strictly less than 2. From (10.6), the result of the iteration
is never more than 1/y (here y denotes the portion we are actually using
in computing the iteration). Thus, if y > 1/2, the estimate is less than 2.
Furthermore, if y = 1/2, an initial estimate less than 2 will ensure that some
error remains, so that the result is still strictly less than 2. Finally, provided
ǫ < 1, the result is always closer to 1/y than the initial estimate. Thus, if
we make sure that our base case gives a value that is less than 2 and no
worse an estimate than 1 would be, the approximation will always be in the
proper range.
We leave it as an exercise to show that the estimate
11 − ⌊8y⌋
4
satisfies the specification and the requirements discussed above for k ≤ 2.
⌊8y⌋ is simply the first 3 bits of y. Because 1/2 ≤ y < 1, 4 ≤ ⌊8y⌋ < 8. The
CHAPTER 10. DIVIDE AND CONQUER 363

Figure 10.10 Implementation of Reciprocal, specified in Figure 10.8,

using Newton’s method with divide-and-conquer

RecipNewton(y, k)
n ← y.NumBits(); len ← k + 3
if k ≤ 2
return eleven.Subtract(y.Shift(3 − n))
else
x0 ← RecipNewton(y, ⌈(k + 1)/2⌉)
y ← y.Shift(len − n)
t ← x0 .Shift(len + x0 .NumBits())
x1 ← t.Subtract(Multiply(y, Multiply(x0 , x0 )))
return x1 .Shift(len − x1 .NumBits())

numerator is therefore always a 3-bit natural number. The final division by

4 simply puts the radix point in the proper place.
The algorithm is shown in Figure 10.10. We use the variable len to store
the value k + 3, which, except in the base case, is both the number of bits
we use from y and the number of bits we return. We assume the existence
of a constant eleven referring to a BigNum with value 11. Before we do the
subtraction, we must make sure the radix points in the operands line up.
The approximation x0 has one bit to the left of the implicit radix point.
The multiplication of x0 by 2 simply moves the radix point to the right
one place. As a result, the implicit radix point in 2x0 is x0 .NumBits() − 2
from the right in x0 . The implicit radix point in the product yx20 is len +
2(x0 .NumBits()−1) bits from the right. In order for the radix points to line
up, we therefore need to pad the value stored in x0 with len+x0 .NumBits()
zeros prior to subtracting.
Let us now analyze the running time of RecipNewton. Suppose we
use a multiplication algorithm that runs in O(M (n)) time, where n is the
number of bits in the product. For now, we will assume that M (n) is a
smooth function in Ω(n), but we will strengthen this assumption as the
analysis proceeds. It is easily seen that for k ≥ 3, the number of bits
returned by RecipNewton is k + 3. Therefore, for k ≥ 4, the worst-case
number of bits in the first product is 2k + 6. Because we use k + 3 bits of y,
the worst-case number of bits in the second product is 3k + 9. Because M
CHAPTER 10. DIVIDE AND CONQUER 364

is smooth, from Exercise 3.18, the time required for the two multiplications
is in O(M (k)).
Because the remainder of the operations, excluding the recursive call,
run in linear time, the total time excluding the recursive call is in O(M (k)).
The total running time is therefore given by the recurrence

k+1
f (k) ∈ f + O(M (k))
2

for k ≥ 4. We can simplify this recurrence by defining f1 (k) = f (k + 1).

Thus, for k ≥ 4,

f1 (k) = f (k + 1)

k+2
∈f + O(M (k + 1))
2

k
=f + 1 + O(M (k + 1))
2
= f1 (⌈k/2⌉) + O(M (k + 1))
= f1 (⌈k/2⌉) + O(M (k)),

because M is smooth.
In order to be able to apply Theorem 3.32 to f1 , we need additional
assumptions on M . We therefore assume that M (k) = k q g(k), where q ≥ 1
and g1 (k) = g(2k+2 ) is smooth. (Note that the functions k lg 3 and k lg k lg lg k
both satisfy these assumptions on M .) Then from Theorem 3.32, f1 (k) ∈
O(M (k)). Because M is smooth, f (k) = f1 (k − 1) ∈ O(M (k)).
We can now analyze the running time of DivideRecip. If m < n, the
running time is clearly in Θ(1). Suppose m ≥ n. Then the value r returned
by Reciprocal(v, m − n) has m − n + 3 bits in the worst case. Hence,
the result of the first multiplication has 2m − n + 3 bits in the worst case.
The worst-case running time of this multiplication is therefore in O(M (m)).
q then has m − n + 1 bits in the worst case. The result of the second
multiplication therefore has m + 1 bits in the worst case, and hence runs
in O(M (m)) time. Because Reciprocal runs in O(M (m − n)) time, and
the remaining operations run in O(m) time, the overall running time is in
O(M (m)). The running time for DivideRecip is therefore the same as for
multiplication, even if our multiplication algorithm runs in O(n lg n lg lg n)
time.
CHAPTER 10. DIVIDE AND CONQUER 365

10.7 Summary
The divide-and-conquer technique involves reducing large instances of a
problem to one or more smaller instances, each of which is a fraction of
the size of the original problem. The running time of the resulting algo-
rithm can typically be analyzed by deriving a recurrence to which Theorem
3.32 applies. Theorem 3.32 can also suggest how to improve a divide-and-
conquer algorithm.
Some variations of the divide-and-conquer technique don’t completely fit
the above description. For example, quick sort does not necessarily produce
subproblems whose sizes are a fraction of the size of the original array. As a
result, Theorem 3.32 does not apply. However, we still consider quick sort to
be a divide-and-conquer algorithm because its goal is to partition an array
into two arrays of approximately half the size of the input array, and to
sort these arrays recursively. Likewise, in LinearSelect, the sizes of the
two recursive calls are very different, but because they are both fractions
of the original size, the analysis ends up being related to that of a more
standard divide-and-conquer algorithm. Finally, DivideDC does not divide
the problem into a bounded number of subproblems; however, all of the
recursive calls in turn yield at most two recursive calls, so we can analyze
these calls using standard divide-and-conquer techniques.

10.8 Exercises
Exercise 10.1 Prove that PolyMult, shown in Figure 10.1, meets its
specification.

Exercise 10.2 PolyMult is not particularly efficient when one polyno-

mial has a degree much larger than that of the other. For example, if p has
degree n and q has degree 1, a straightforward implementation of the def-
inition of the product yields Θ(n) running time. Devise an algorithm that
runs in Θ(mnlg 3−1 ) time on polynomials of degree m and n with m ≥ n.
Your algorithm may use PolyMult. Analyze the running time of your al-
gorithm. [Hint: If m > n, divide the larger polynomial into polynomials of
degree at most n.]

* Exercise 10.3 Construct a divide-and-conquer polynomial multiplica-

tion algorithm that performs 5 recursive calls on polynomials of 1/3 the
size of the original polynomials. Show that your algorithm has a running
time in Θ(nlog3 5 ). (Note that log3 5 < lg 3.)
CHAPTER 10. DIVIDE AND CONQUER 366

** Exercise 10.4 Generalize Exercise 10.3 by showing that for sufficiently

large n and any k ≥ 2, the product of two degree-(n − 1) polynomials can be
computed from the products of 2k − 1 polynomials of degree approximately
(n/k)−1. Using this result, show that for any ǫ ∈ R>0 , there is an algorithm
to multiply two degree-(n − 1) polynomials in O(n1+ǫ ) time.

Exercise 10.5 Adapt PolyMult to implement Multiply, as specified in

Figure 10.6, in Θ(nlg 3 ) time, where n is the number of bits in the product.

Exercise 10.6 Prove that MergeSort, shown in Figure 10.2, meets its
specification.

Exercise 10.7 Suppose we are given a tape containing a large number of

Keyed items to be sorted. The number of items is too large to fit into main
memory, but we have three additional tapes we can use, and we can rewrite
the input tape. Give a bottom-up version of merge sort that produces the
sorted output on one of the tapes. You may not assume that data items on
the tapes can be accessed “randomly” — they must be accessed in sequence.
Your algorithm must make at most O(lg n) passes through each tape.

Exercise 10.8 Prove that QuickSort, shown in Figure 10.3, meets its
specification.

Exercise 10.9 Notice that one of the recursive calls in QuickSort is tail
recursion. Taking advantage of this fact, convert one of the recursive calls
to iteration. Notice that the calls can be made in either order, and so
either may be converted to iteration. Make the proper choice so that the
resulting algorithm uses Θ(lg n) stack space in the worst case on an array
of n elements.

* Exercise 10.10 The goal of this exercise is to prove Theorem 10.1.

a. For x ≤ y, let [x, y] denote the set of all real numbers a such that
x ≤ a ≤ y. For natural numbers m < n, let f : [m, n] → R≥0 be a
continuous function such that whenever m ≤ x < y ≤ n, f (x) ≥ f (y)
(i.e., f is nonincreasing). Prove that
n
X Z n n−1
X
f (i) ≤ f (x)dx ≤ f (i).
i=m+1 m i=m

b. Use the result of part a. to prove Theorem 10.1.

CHAPTER 10. DIVIDE AND CONQUER 367

Exercise 10.11 Prove that for an array of size n, QuickSort (shown in

Figure 10.3) makes a total of at most n + 1 calls (including the initial call
and all recursive calls, as appropriate) in which lo ≥ hi.

Exercise 10.12 A randomized algorithm for the selection problem can be

obtained by replacing the the first assignment statement of SelectBy-
Median (Figure 2.7 on page 43) with the statement:

p ← A[RandomInteger(1, n)]

Show that the expected running time of this algorithm is in Θ(n). [Hint:
Your analysis should be similar to the analysis of QuickSort in Section
10.3.]

Exercise 10.13 Let n0 ≥ 1 and m ≥ 1 be integers, let q be a positive real

number, and let c1 , . . . , cm be positive real numbers such that
m
cqi < 1.
X

i=1

Let f : N → R≥0 be an eventually nondecreasing function satisfying

m
X
f (n) ∈ f (⌊ci n⌋) + X(nq )
i=1

whenever n ≥ n0 , where X is either O, Ω, or Θ. Prove that f (n) ∈ X(nq ).

* Exercise 10.14 Let n0 ≥ 1 and m ≥ 2 be integers, let q be a positive

real number, and let c1 , . . . , cm be positive real numbers such that
m
cqi = 1.
X

i=1

Let f : N → R≥0 be an eventually nondecreasing function satisfying

m
X
f (n) ∈ f (⌊ci n⌋) + O(nq )
i=1

whenever n ≥ n0 . Prove that f (n) ∈ O(nq lg n).

Exercise 10.15 Prove that LinearSelect, shown in Figure 10.5, meets

the specification given in Figure 1.2 (p. 6).
CHAPTER 10. DIVIDE AND CONQUER 368

Exercise 10.16 Determine the number of comparisons used by each of the

following algorithms when sorting 4 elements.

a. InsertionSort, shown in Figure 1.7 on page 11.

b. MergeSort, shown in Figure 10.2.

Exercise 10.17 Repeat Exercise 10.16 for 5 elements.

Exercise 10.18 Show that it is possible to find either the smallest or largest
of n elements using at most n − 1 comparisons.

* Exercise 10.19 Show that it is possible to find either the second largest
or second smallest of n elements using at most n + ⌈lg n⌉ − 2 comparisons.

* Exercise 10.20 Show that it is possible to find the median of five ele-
ments using at most six comparisons.

* Exercise 10.21 Prove that DivideDC, shown in Figure 10.7, meets is

specification as given in Figure 10.6.

Exercise 10.22 Prove that DivideRecip, shown in Figure 10.8, meets its
specification as given in Figure 10.6.

* Exercise 10.23 Let 1/2 ≤ y < 1.

a. Show that if
11 − ⌊8y⌋
x0 = ,
4
then
1 1

y − x0 ≤ .
4

b. Show that if x0 is as defined in part a, then

1
− x0 ≤ 1 − 1 .

y y

c. Prove that RecipNewton meets is specification as given in Figure

10.8.
CHAPTER 10. DIVIDE AND CONQUER 369

Figure 10.11 Additional functions for BigNums

Precondition: u is a BigNum and k is a Nat.

Postcondition: Returns a BigNum representing the value of uk . We
assume that 00 = 1.
Power(u, k)
Precondition: u is a BigNum.
Postcondition: Returns a String (see Figure 4.17 and Exercise 4.13)
containing the decimal representation u.
ToString(u)

Exercise 10.24 Design a divide-and-conquer algorithm that implements

Power as specified in Figure 10.11. Your algorithm should run in O(M (n)) Figure 10.11
corrected
time, where n is the number of bits in the result and M (n) is the time 4/12/12.
needed for Multiply when the product contains n bits. You may make
reasonable assumptions about M (n), provided nlg 3 and n lg n lg lg n satisfy
these assumptions.

* Exercise 10.25 Design a divide-and-conquer algorithm that implements

ToString as specified in Figure 10.11. Your algorithm should run in O(nq )
time, where n is the number of bits in u, assuming Multiply needs O(nq )
time to produce an n-bit product. You may assume q is a real number
strictly larger than 1.

* Exercise 10.26 Given two natural numbers u and v which are not both
0, the greatest common divisor of u and v (or gcd(u, v)) is the largest integer
that evenly divides both u and v.

a. Prove that for any positive integers u and v, gcd(u, v) = gcd(v, u mod
v).

b. Design a divide-and-conquer algorithm that takes as input two positive

integers u and v and returns gcd(u, v). Your algorithm should run in
O(lg max(u, v)) time.
CHAPTER 10. DIVIDE AND CONQUER 370

* Exercise 10.27 Given two positive integers u and m such that u < m,
a multiplicative inverse of u mod m is any positive integer v such that 1 ≤
v < m and (uv) mod m = 1.
a. Prove that for any positive integers u and v, there exist integers a and
b such that au + bv = gcd(u, v).
b. Prove that u has a multiplicative inverse mod m iff gcd(u, m) = 1.
[Hint: See Theorem 7.4 on page 265.]
c. Prove that for 1 ≤ u < m, u has at most one multiplicative inverse
mod m.
d. Give a efficient divide-and-conquer algorithm that takes as input pos-
itive integers u and m such that u < m and returns the multiplicative
inverse of u mod m, or nil if no inverse exists. Your algorithm should
run in O(lg m) time. [Hint: Modify the algorithm for Exercise 10.26
to find a and b as described in part a.]

Exercise 10.28 The Manhattan Skyline Problem can be stated as follows.

We are given a description of n rectangular buildings on the horizon. Each
description is a triple, hli , wi , hi i, where li is the x-coordinate of the build-
ing’s left-hand edge, wi is the width of the building, and hi is the height
above the horizon of the building’s roof. (Note that the buildings may over-
lap.) We wish to construct the skyline produced by these buildings. The
skyline is represented by a sequence of points h(x1 , y1 ), . . . , (xk , yk )i, ordered
by x-coordinate, representing the locations where a vertical segment of the
skyline meets a horizontal segment leading to the right (see Figure 10.12).
Note that the value of yk must always be 0. Give a divide-and-conquer al-
gorithm to compute the Manhattan skyline, and show that your algorithm
runs in Θ(n lg n) time.

Exercise 10.29 A majority element of an array A[1..n] is an element that

occurs more than n/2 times in the array. Construct an efficient divide-
and-conquer algorithm to find the majority element of A if one exists. Your
algorithm may only compare elements for equality (hence, it may not sort the
elements). Analyze the worst-case running time of your algorithm. (Θ(n) is
possible.)

Exercise 10.30 Give a divide-and-conquer algorithm to construct a round-

robin tournament involving n competitors. The tournament consists of a se-
ries of rounds. In each round, every competitor plays one other competitor
CHAPTER 10. DIVIDE AND CONQUER 371

Figure 10.12 Manhattan skyline problem example.

(x4 , y4 )

(x2 , y2 )

(x3 , y3 ) (x5 , y5 )
(x1 , y1 )

(x6 , y6 )

if n is even; if n is odd, exactly one competitor is idle each round. Every

competitor must play every other competitor exactly once in the tourna-
ment. Therefore, the number of rounds is n − 1 if n is even, or n if n is odd.
Let the competitors be identified by the natural numbers 0, . . . , n − 1. Your
algorithm should produce a 2-dimensional array A of natural numbers such
that A[i, j] indicates j’s opponent in round i; if j is idle in round i, A[i, j]
should be n. Your algorithm should run in Θ(n2 ) time. A possible output
for n = 5 is shown below.
0 1 2 3 4
0 1 0 5 4 3
1 2 4 0 5 1
2 3 2 1 0 5
3 4 5 3 2 0
4 5 3 4 1 2

* Exercise 10.31 Give a Θ(n lg n) divide-and-conquer algorithm for de-

termining the closest pair of points in a given collection. You may assume
that the points are given via two arrays: x[1..n] and y[1..n], where n ≥ 2.
The n points are then (x[1], y[1]), (x[2], y[2]), . . . , (x[n], y[n]). The distance
CHAPTER 10. DIVIDE AND CONQUER 372

between a pair of points (x, y) and (x′ , y ′ ) is given by

p
(x − x′ )2 + (y − y ′ )2 .

Your algorithm should return the minimum distance separating any two
distinct points.
√
* Exercise 10.32 Give a divide-and-conquer algorithm for computing ⌊ n⌋,
where n is a BigNum. Your algorithm’s running time should be in O(M (lg n)),
where M (n) is as defined in Section 10.6.

** Exercise 10.33 Give a Θ(nlg 7 ) divide-and-conquer algorithm for mul-

tiplying two n × n matrices of real numbers. [Hint: Find a way to multiply
two 2 × 2 matrices using only 7 scalar multiplications. Use this technique
as the basis for a divide-and-conquer algorithm.]

* Exercise 10.34 A Hamiltonian path in a (directed or undirected) graph

is a path that contains each vertex exactly once. A directed graph is said to
be complete if for each pair of distinct vertices i and j, either (i, j) or (j, i)
is an edge in the graph. It turns out that every complete directed graph
has a Hamiltonian path. Give a divide-and-conquer algorithm that finds a
Hamiltonian path in a given complete directed graph. Your algorithm should
run in O(n lg n) time in the worst case, where n is the number of vertices in
the graph, assuming the graph is implemented as a MatrixGraph.

10.9 Chapter Notes

The PolyMult algorithm is based on a Θ(nlg 3 ) large-integer multiplica-
tion algorithm by Karatsuba and Ofman [73]. The DivideDC algorithm is
due to Burnikel and Ziegler [19]. The RecipNewton algorithm is a top-
down adaptation of an algorithm given by Knuth [77]; he credits the idea
to Cook. Solutions to Exercises 10.3, 10.4, 10.24, and 10.25, can be found
in Knuth [77]. Implementations of variations on the DivideRecip and Re-
cipNewton, as well as the algorithms suggested by Exercises 10.24 and
10.25, can be found on this textbook’s web site. Furthermore, the calcula-
tor demo posted there provides a framework for testing implementations of
arbitrary-precision arithmetic algorithms.
Merge sort was one of the earliest algorithms developed for electronic
computers, being developed by von Neumann in 1945 [106, 78]. Exercise 10.7
is based on work by Eckert and Mauchly [32]. Quick sort was developed by
CHAPTER 10. DIVIDE AND CONQUER 373

Hoare [59]. See Bentley and McIlroy [13] for a good practical implementation
of quick sort.
Algorithm LinearSelect is due to Blum, et al. [15]. The solution to
Exercise 10.19 is due to Aigner [4].
The solution to Exercise 10.31 is due to Bentley [11]. The solution to
Exercise 10.33 is due to Strassen [101].
Chapter 11

Optimization I: Greedy
Algorithms

In this chapter and the next, we consider algorithms for optimization prob-
lems. We have already seen an example of an optimization problem — the
maximum subsequence sum problem from Chapter 1. We can characterize
optimization problems as admitting a set of candidate solutions. In the max-
imum subsequence sum problem, the candidate solutions are the contiguous
subsequences in the input array. An objective function then typically maps
these candidate solutions to numeric values. The objective function for the
maximum subsequence sum problem maps each contiguous subsequence to
its sum. The goal is to find a candidate solution that either maximizes or
minimizes, depending on the problem, the objective function. Thus, the
goal of the maximum subsequence problem is to find a candidate solution
that maximizes the objective function.
In this chapter, we will examine optimization problems which admit
greedy solutions. A greedy algorithm builds a specific candidate solution
incrementally. The aspect of a greedy algorithm that makes it “greedy” is
how it chooses from among the different ways of incrementing the current
partial solution. In general, the different choices are ordered according to
some criterion, and the best choice according to this criterion is taken. Thus,
the algorithm builds the solution by always taking the step that appears to
be most promising at that moment. Though there are many problems for
which greedy strategies do not produce optimal solutions, when they do,
they tend to be quite efficient. In the next chapter, we will examine a more
general technique for solving optimization problems when greedy strategies
fail.

374
CHAPTER 11. OPTIMIZATION I: GREEDY ALGORITHMS 375

11.1 Job Scheduling

Consider the job scheduling problem discussed in Chapter 8. Recall that
we are given n jobs, each requiring one unit of execution time and having
its own deadline. Suppose that, in addition, each job has a positive integer
value. We wish to schedule the jobs on a single server so as to maximize the
total value of those jobs which meet their deadlines. Because jobs which do
not meet their deadlines do not contribute any value, we will assume that no
jobs are scheduled after their deadlines — if a job can’t meet its deadline, we
simply don’t schedule it. At this point, we are not assuming any particular
scheduling strategy, such as the one given in Chapter 8; instead, we are
trying to find an optimal strategy.
In deriving a greedy algorithm in a top-down fashion, the first step is
to generalize the problem so that a partial solution is given as input. We
assume as a precondition that this partial solution can be extended to an
optimal solution. Our task is then to extend it in some way so that the
resulting partial solution can be extended to an optimal solution. If we
characterize the size of such an instance as the difference between the size
of a complete solution and the given partial solution, we will have reduced
a large instance to a smaller instance.
The input to the generalized scheduling problem is a set X = {x1 , . . . , xm }
of jobs and a partial schedule sched of these jobs. To be more precise, let
sched[1..n] be an array of natural numbers such that if sched[t] = 0, then
no job has been scheduled in the time slot ending at time t; otherwise, if
sched[t] = i, then job xi is scheduled in this time slot. If all the jobs in
X either have been scheduled or cannot be scheduled, we are finished —
the precondition that this schedule can be extended to an optimal sched-
ule implies that it must be an optimal schedule. Otherwise, our task is to
schedule some job xi so that the resulting partial schedule can be extended
to a schedule of maximum value. If we take the size of a partial schedule
to be the number of unscheduled jobs in X, we will have reduced a large
instance to a smaller instance.
We must now decide upon the criterion to use to extend a partial sched-
ule. Of the remaining jobs that can meet their deadlines, it would make
sense to schedule the one with the highest value. Furthermore, in order to
impact the fewest deadlines of other jobs, it would make sense to schedule it
as late as possible. In what follows, we will show that this selection criterion
always results in an optimal schedule.
In order to simplify reasoning about this strategy, let us observe that
because we will not be changing any scheduling decisions that have already
CHAPTER 11. OPTIMIZATION I: GREEDY ALGORITHMS 376

been made, the values of the jobs scheduled so far have no effect on future
decisions — their values are simply added to the total value of the schedule.
As a result, all we really need to know about the schedule constructed so far
is what time slots are still available. Furthermore, maximizing the values of
jobs scheduled in the remaining slots will maximize the total value, because
the values of all scheduled jobs are simply added together.
We can therefore focus our attention on the following version of the
problem. The input consists of a set X of (unscheduled) jobs and an array
avail[1..n] of boolean values. A valid schedule either assigns a job xi into a
time slot t such that t is no more than the deadline of xi and avail[t] = true,
or it does not schedule xi . The goal is to maximize the total value of
scheduled jobs. The following theorem shows that an optimal schedule can
be constructed by selecting the job with maximum value and scheduling it
at the latest possible time, assuming it can be scheduled.

Theorem 11.1 Let X = {x1 , . . . , xm } be a set of jobs, and let avail[1..n]

be an array of boolean values indicating the time slots at which jobs may
be scheduled. Let xk be a job having maximum value, and suppose there is
some t no greater than the deadline of xk such that avail[t] = true. Let t0
be the maximum such t. Then there is an optimal schedule in which xk is
scheduled at the time slot ending at time t0 .

Proof: Let sched[1..n] be an optimal schedule and suppose sched[t0 ] 6= k.

We consider two cases.

Case 1: sched[t1 ] = k. Because t0 is the latest available time slot for

xk , t1 < t0 . Therefore, by swapping the values of sched[t1 ] and sched[t0 ],
we violate no deadlines and do not change the value of the schedule. The
resulting schedule must therefore be optimal.

Case 2: xk is not scheduled in sched. Let j = sched[t0 ]. We first observe

that j 6= 0, because in this case we could obtain a schedule with higher value
by scheduling xk in sched[t0 ]. Because xk is a job having maximum value,
the value of xk is at least the value of xj . Therefore, by scheduling xk at
sched[t0 ] instead of xj , we retain an optimal schedule.

Theorem 11.1 tells us that our greedy strategy results in an optimal

schedule. To implement this strategy, we need to consider the jobs in nonin-
creasing order of their values, and schedule each schedulable job at the latest
time possible. Therefore, we should first sort the jobs in nonincreasing order
CHAPTER 11. OPTIMIZATION I: GREEDY ALGORITHMS 377

of their values. Using heap sort or merge sort, this can be done in Θ(m lg m)
time. Schedule, shown in Figure 8.2, then implements the greedy strat-
egy. Because Schedule can be implemented to run in O(n + m lg n) time,
if m ∈ Θ(n), the entire algorithm runs in Θ(n lg n) time.

11.2 Minimum-Cost Spanning Trees

Suppose we wish to construct a communications network connecting a given
set of nodes. Given the distances separating each pair of nodes, we wish to
find the network topology that connects all of the nodes using as little cable
as possible.
We can state the above problem as a graph problem. In Exercise 9.6,
we defined a tree to be connected, acyclic, undirected graph. (Note that a
tree is different from a rooted tree as defined on page 153, though we can
form a rooted tree from a tree by selecting any vertex as the root.) Given a
connected undirected graph G = (V, E), a spanning tree is a tree (V, T ) such
that T ⊆ E; i.e., a spanning tree is a tree consisting of all of the vertices of
G and a subset of the edges. Let cost : E → N give a cost for each edge. We
wish to find a minimum-cost spanning tree (MST) for G — i.e., a spanning
tree whose edges have minimum total cost.
In order to develop a greedy algorithm, we first generalize the problem
so that a portion of an MST is given as input. This partial MST will be a
subset E ′ ⊆ E such that (V, E ′ ) is acyclic, but not necessarily connected.
In order to keep the cost as small as possible, we will use as our selection
criterion the cost of the edge; i.e., we will always select a least-cost edge that
does not introduce a cycle.
We need to show that the above strategy will result in an MST. In order
to state the theorem that guarantees this fact, we need one definition. Let
G = (V, E) be an undirected graph. A connected component of G is any
connected subset C ⊆ V such that no vertex in C is adjacent to any vertex
in V \ C. Thus, the connected component containing a vertex v is the set
of all vertices reachable from v using zero or more edges. We can now state
the following theorem, which validates our selection strategy.

Theorem 11.2 Let G = (V, E) be a connected undirected graph with cost

function cost : E → N, and let E ′ ⊂ E be such that for some MST (V, T ) of
G, E ′ ⊆ T . Suppose that (V, E ′ ) is not connected, and let C be a connected
component of (V, E ′ ). If {u, v} ∈ E \ E ′ is a minimum-cost edge such that
u ∈ C and v 6∈ C, then there is an MST of G containing all the edges in
E ′ ∪ {{u, v}}.
CHAPTER 11. OPTIMIZATION I: GREEDY ALGORITHMS 378

Before we prove Theorem 11.2, we note that it is stronger than we re-

quire. We state it in this way in order to justify a second greedy algorithm,
which we will discuss a bit later. Note, however, that a minimum-cost edge
that does not introduce a cycle certainly qualifies as the edge {u, v} in the
statement of the theorem.

Proof of Theorem 11.2: If {u, v} ∈ T , then there is nothing to show.

Suppose {u, v} 6∈ T . We will show how to construct a set T ′ such that
E ′ ∪ {{u, v}} ⊆ T ′ and (V, T ′ ) is an MST.
Because (V, T ) is a tree, there is a path from u to v in T . However, this
path cannot contain the edge {u, v}, because {u, v} 6∈ T . This path must
therefore consist of:
• a (possibly empty) path in C from u to some vertex w;
• an edge {w, x}, where x 6∈ C; and
• a (possibly empty) path from x to v.
Note that even though either path above might be empty, they cannot both
be empty, or we would have {w, x} = {u, v}. By the choice of {u, v},
cost({w, x}) ≥ cost({u, v}). Let T ′ = (T ∪ {{u, v}}) \{{w, x}}. Then the
total cost of T ′ is no more than the total cost of T . From Exercise 9.6 a, T ,
and hence T ′ , has exactly |V | − 1 edges. Furthermore, (V, T ′ ) is connected,
as any path in T that contains {w, x} can be modified to use the edge {u, v}
and the paths from u to w and x to v in T . From Exercise 9.6 b, (V, T ′ ) is
a tree. Because its cost is no more than the cost of the MST (V, T ), (V, T ′ )
is an MST containing all of the edges in E ′ ∪ {{u, v}}.

In order to implement this algorithm, we need an efficient way to de-

termine whether two vertices belong to the same connected component of
(V, E ′ ), where E ′ is the set of edges we have collected so far. The con-
nected components form disjoint subsets of V . We can therefore main-
tain these connected components using a DisjointSets structure. In order
to determine whether u and v belong to the same component, we see if
Find(u) = Find(v). If not, we include {u, v} and merge the two compo-
nents.
The algorithm, known as Kruskal’s algorithm, is shown in Figure 11.1.
Note that in using an InvertedPriorityQueue, we process the edges in
nondecreasing order of cost. We could have achieved the same effect by
sorting the edges by cost, but this presentation is somewhat simpler, and in
fact amounts to sorting the edges with heap sort.
CHAPTER 11. OPTIMIZATION I: GREEDY ALGORITHMS 379

Figure 11.1 Kruskal’s algorithm for finding an MST

Precondition: G refers to a Graph which is undirected and connected,

and whose edges contain their costs as data.
Postcondition: Returns a ConsList of Edges representing an MST of
G. Each edge will occur once in the list.
Kruskal(G)
n ← G.Size(); comp ← new DisjointSets(n)
q ← new InvertedPriorityQueue(); T ← new ConsList()
for i ← 0 to n − 1
L ← G.AllFrom(i)
while not L.IsEmpty()
e ← L.Head(); q.Put(e, e.Data()); L ← L.Tail()
// Invariant: T is a subset of the edges of an MST of G, the sets in
// comp are the connected components of T , and q contains a subset of
// the edges of G ordered by cost, including at least all {u, v} such that
// u and v are in different sets in comp.
while not q.IsEmpty()
e ← q.RemoveMin()
c1 ← comp.Find(e.Source()); c2 ← comp.Find(e.Dest())
if c1 6= c2
T ← new ConsList(e, T ); comp.Merge(c1 , c2 )
return T

For the purpose of analyzing Kruskal, let n be the number of vertices

in G, and let a be the number of edges. Let us first assume that G is
implemented using ListGraph. In the initialization code preceding the
first loop, Θ(n) time is required to construct a new DisjointSets structure,
whereas the other operations each run in Θ(1) time. The for loop with the
nested while simply traverses G in a manner similar to the loops found in
TopSort (Figure 9.6). If we ignore for the moment the cost of the Put
operations on q, we see that this nested structure runs in Θ(n+a) time. Each
edge is inserted into q; hence, because InvertedPriorityQueue.Put runs
in Θ(lg i) time when there are i elements in the queue, the total time for all
insertions is in Θ(a lg a). Because G is connected, n − 1 ≤ a ≤ n(n − 1)/2.
Furthermore, lg(n(n−1)/2) < 2 lg n, so that Θ(n+a) + Θ(a lg a) = Θ(a lg n).
The last while loop is easily seen to run in Θ(a lg n) time as well.
CHAPTER 11. OPTIMIZATION I: GREEDY ALGORITHMS 380

If G is implemented using MatrixGraph, the AllFrom operation re-

quires Θ(n) time, so that the for loop requires Θ(n2 ). The total running time
is therefore in Θ(n2 + a lg n), which is worse than Θ(a lg n) for sufficiently
sparse graphs (i.e., when a ∈ o(n2 / lg n)). Kruskal’s algorithm therefore
tends to be better-suited for the ListGraph implementation, particularly
when the graph is sparse.
As we suggested earlier, Kruskal’s algorithm isn’t the only greedy algo-
rithm for finding MSTs. We arrive at a different algorithm if we generalize
the original problem in a slightly different way. Rather than allowing our
input to consist of any set of edges that can be extended to an MST, we
instead require that this set of edges form a spanning tree on some subset
of the vertices. Thus, when we add an edge, it must extend this spanning
tree to another vertex; i.e., it must connect a vertex in the spanning tree
to one that is not in the spanning tree. Our selection criterion will be to
select such an edge having minimum cost. Theorem 11.2 tells us that such
a strategy results in an MST.
The data structures needed to implement this algorithm, which is known
as Prim’s algorithm, are simpler than those needed to implement Kruskal’s
algorithm. We need to partition the vertices into two disjoint sets — the
set of vertices in the spanning tree and those not in the spanning tree. A Corrected
5/7/12.
boolean array inTree[0..n − 1] will suffice for this purpose. For each vertex
k not in the spanning tree, we need an efficient way to find a least-cost
edge {i, k} such that i is in the spanning tree. For this purpose, we use two
arrays:

• an array bestCost[1..n − 1] such that if k is not in the spanning tree,

then bestCost[k] is the minimum cost of any edge to k from a vertex
in the spanning tree, or ∞ if there is no such edge; and

• an array best[1..n − 1] such that if k is not in the spanning tree and

bestCost[k] 6= ∞, then {best[k], k} is a least-cost edge from the span-
ning tree to k.

The spanning tree will initially contain only the vertex 0; hence, it is
unnecessary to include the index 0 for the arrays best and bestCost. We
can then initialize each best[k] to 0 and each bestCost[k] to the cost of edge
{0, k}, or to ∞ if there is no such edge. In order to find an edge to add
to the spanning tree we can find the minimum bestCost[k] such that k is
not in the spanning tree. If we denote this index by next, then the edge
{best[next], next} is the next edge to be added, thus connecting next to
the spanning tree. For each k that is still not in the spanning tree, we must
CHAPTER 11. OPTIMIZATION I: GREEDY ALGORITHMS 381

then update bestCost[k] by comparing it to the cost of {next, k}, and update
best[k] accordingly. The algorithm is shown in Figure 11.2.
It is easily seen that if G is a MatrixGraph, the running time is in
Θ(n2 ). This is an improvement over Kruskal’s algorithm when a Matrix-
Graph is used. If a ListGraph is used, however, the running time is still
in Ω(n2 ), and can be as bad as Θ(n3 ) for dense graphs. Thus, Kruskal’s
algorithm is preferred when a ListGraph is used, but Prim’s algorithm is
preferred when a MatrixGraph is used. If we have the freedom of choosing
the Graph implementation, we should choose a ListGraph and Kruskal’s
algorithm for sparse graphs, but a MatrixGraph and Prim’s algorithm for
dense graphs.

11.3 Single-Source Shortest Paths

Suppose we wish to drive from Los Angeles to Boston. A cursory glance
at a road atlas tells us there are many different routes we can take. We
decide that we wish to take a route that optimizes a particular objective
function — perhaps total length or total expected time. We can model such
a problem as an instance of a shortest path problem. We model the various
road segments as edges of a directed graph G whose vertices are the road
intersections. We represent Los Angeles with a start vertex u and Boston
with an end vertex v. A cost (representing either length or expected time)
is associated with each edge. We wish to find a least-cost path from u to v
in G.
We will use the term length to refer to the cost of an edge, regardless of
what kind of cost is being represented. A least-cost path is then a shortest
path. We will assume that edge lengths are positive integers. Note that if
we have found a shortest path from u to v, and if vertex w occurs on this
path, then the subpath from u to w is also a shortest path from u to w.
As a result, when we find a shortest path from u to v, we typically find
many other shortest paths from u as well. For this reason, it simplifies the
discussion to generalize the problem to that of finding, for every vertex w,
a shortest path from u to w.
We first observe that for each vertex w 6= u, if we consider only a single
shortest path from u to w, then there is a unique predecessor x of w on
this path. Furthermore, we can select the shortest paths in such a way
that x precedes w on any shortest path on which w occurs. Thus, for each
vertex in G, there is a unique sequence of these predecessors leading back
to u. This predecessor relationship therefore gives the parent relationship
CHAPTER 11. OPTIMIZATION I: GREEDY ALGORITHMS 382

Figure 11.2 Prim’s algorithm for finding an MST

Precondition: G refers to a Graph which is undirected and connected,

and whose edges contain their costs as data.
Postcondition: Returns a ConsList of Edges representing an MST of
G. Each edge will occur once in the list.
Prim(G)
n ← G.Size(); inTree ← new Array[0..n − 1]; inTree[0] ← true
best ← new Array[1..n − 1]; bestCost ← new Array[1..n − 1]
T ← new ConsList()
for k ← 1 to n − 1
inTree[k] ← false; best[k] ← 0; bestCost[k] ← G.Get(0, k)
if bestCost[k] = nil
bestCost[k] ← ∞
// Invariant: T contains count edges forming a spanning tree for the
// vertices k such that inTree[k] is true, and there is an MST of G
// containing all of the edges in T . For each k such that inTree[k] is false,
// bestCost[k] is the minimum cost of any edge {i, k} such that inTree[i]
// is true, or ∞ if there is no such edge. For each k such that inTree[k]
// is false and bestCost[k] 6= ∞, {best[k], k} is a least-cost edge
// leading to k from any vertex i such that inTree[i] is true.
for count ← 0 to n − 2
m←∞
for k ← 1 to n − 1
if not inTree[k] and bestCost[k] < m
next ← k; m ← bestCost[k]
e ← new Edge(best[next], next, m)
T ← new ConsList(e, T ); inTree[next] ← true
for k ← 1 to n − 1
if not inTree[k]
d ← G.Get(next, k)
if d 6= nil and d < bestCost[k]
best[k] ← next; bestCost[k] ← d
return T
CHAPTER 11. OPTIMIZATION I: GREEDY ALGORITHMS 383

of a tree rooted at u. This rooted tree can be used to represent the shortest
paths.
Let us now generalize the problem so that a tree T rooted at u and
containing a subset of the edges and vertices of the graph is provided as
additional input. Suppose that this tree is a proper subtree of a shortest
path tree; i.e., suppose that for each vertex w in the tree, the path from u to
w in the tree is a shortest path from u to w in G. We need to add a vertex
x and an edge (w, x), where w is a vertex in T , so that the path from u to
x in the resulting tree is a shortest path in G from u to x.
For each vertex w in T , let dw give the length of the path from u to w
in T . For each edge (x, y) in G, let len(x, y) give the length of (x, y). Let
(w, x) be an edge in G such that

• w is in T ;

• x is not in T ; and

• dw + len(w, x) is minimized.

Clearly, the path from u to w in T , followed by the edge (w, x) is a shortest

path from u to x in G, as there can be no shorter paths from u to any vertex
not in T .
Building a shortest path tree in this way is very similar to the way
Prim’s algorithm builds an MST. While Prim’s algorithm is for undirected
graphs, this algorithm — Dijkstra’s algorithm — is for directed graphs. The
computation for updating the bestCost array must use dw +len(w, x), rather
than the cost of (w, x) (we assume that len(w, x) will be stored in the data
variable for edge (w, x)). Hence, the value dw must be computed and stored
as vertices are added to the tree. The resulting algorithm runs in Θ(n2 )
time, the same as for Prim’s algorithm; we leave the details as an exercise.

11.4 Huffman Codes

Compression algorithms are often used for data archival or for improving
data transmission rates. In this section, we examine one of the key com-
ponents of data compression. In order to simplify the discussion, we will
assume we are dealing with character data, though the techniques apply
more generally.
Typical character encodings, such as ASCII or Unicode, are fixed-width
encodings — all characters are encoded using the same number of bits.
However, in a typical natural-language text, some characters like ‘e’ occur
CHAPTER 11. OPTIMIZATION I: GREEDY ALGORITHMS 384

Figure 11.3 A Huffman tree for the string, “Mississippi”

M p

much more frequently than other characters like ‘X’. It might make sense,
then, to use a variable-width encoding, so that the more frequently occurring
characters have shorter codes. Furthermore, if the encoding were chosen for
a particular text document, we could use shorter codes overall because we
would only need to encode those characters that actually appear in the
document.
The difficulty with variable-width encodings is choosing the encoding so
that it is clear where one character ends and the next begins. For example,
if we encode ‘n’ with 11 and ‘o’ with 111, then the encoding 11111 would
be ambiguous — it could encode either “no” or “on”. To overcome this
difficulty, we arrange the characters as the leaves of a binary tree in which
each non-leaf has two nonempty children (see Figure 11.3). The encoding of
a character is determined by the path from the root to the leaf containing
that character: each left child on the path denotes a 0 in the encoding, and
each right child on the path denotes a 1 in the encoding. Thus, in Figure
11.3, ‘M’ is encoded as 100. Because no path from the root to a leaf is a
proper prefix of any other path from the root to a leaf, no ambiguity results.

Example 11.3 For example, we can use the tree in Figure 11.3 to en-
code “Mississippi” as 100011110111101011010. We parse this encoding by
traversing the tree according to the paths specified by the encoding. Start-
ing at the root, we go right-left-left, arriving at the leaf ‘M’. Starting at the
root again, we go left, arriving at the leaf ‘i’. Continuing in this manner, we
CHAPTER 11. OPTIMIZATION I: GREEDY ALGORITHMS 385

see that the bit-string decodes into “Mississippi”. Note that because there
are four distinct characters, a fixed-width encoding would require at least
two bits per character, yielding a bit string of length 22. However, the bit
string produced by the given encoding has length 21.
The specific problem we wish to address in this section is that of pro-
ducing a tree that yields a minimum-length encoding for a given text. We
will not concern ourselves with the counting of the characters in the text;
rather, we will assume that a frequency table has been produced and is pro-
vided as our input. This frequency table gives the number of occurrences of
each character in the text. To simplify matters, we will assume that none
of the characters in the table has a frequency of 0. Furthermore, we will
not concern ourselves with producing the encoding from the tree; i.e., our
output consists solely of a binary tree storing the information we need in
order to extract each character’s code.
We first need to consider how we can determine the length of an encoded
string for a particular encoding tree. Note that when we decode a bit string,
we traverse exactly one edge in the tree for each bit of the encoding. One
way to determine the length of the encoding is therefore to compute the
number of times each edge would be traversed during decoding. A given
edge (u, v) is traversed once for each occurrence of each character in the
subtree rooted at v. For a subtree t, let us therefore define weight(t) to be
the total number of occurrences of all characters in t. For an encoding tree
T , we can then define cost(T ) to be the sum of the weights of all proper
subtrees of T . (Note that weight(T ) will always be the length of the given
text.) cost(T ) then gives the length of the encoding based on T . For a
given frequency table, we define a Huffman tree to be an encoding tree with
minimum cost for that table.
Let us now generalize the problem so that the input is a collection of
trees, t1 , . . . , tn , each of which encodes a portion of the frequency table.
We assume that each character in the frequency table occurs in exactly one
of the input trees, and that the frequency table has a Huffman tree that
contains all of the input trees as subtrees. Note that if all of the trees are
single nodes, this input is just the information from the frequency table. If
the input consists of more than one tree, we need to merge two of the trees
by making them the children of a new root. Furthermore, we need to be
able to do this so that the frequency table has a Huffman tree containing
all of the resulting trees as subtrees. We claim that merging two trees of
minimum weight will produce such a tree.
CHAPTER 11. OPTIMIZATION I: GREEDY ALGORITHMS 386

Theorem 11.4 Let T be a Huffman tree for a frequency table F , and let
t1 , . . . , tn be subtrees of T such that n > 1 and each leaf of T occurs in exactly
one of t1 , . . . , tn . Suppose weight(t1 ) ≤ weight(t2 ) ≤ · · · ≤ weight(tn ). Let
tn+1 be the binary tree formed by making t1 and t2 the left and right children,
respectively, of a new root. Then there is a Huffman tree T ′ for F containing
t3 , t4 , . . . , tn+1 as subtrees.

Proof: If tn+1 is a subtree of T , then we can let T ′ = T . Furthermore, if

T has a subtree with t1 as the right child and t2 as the left child, we can
simply swap t1 with t2 and let the resulting tree be T ′ . Otherwise, t1 and
t2 are not siblings in T . Furthermore, neither can be a subtree of the other
because they have no leaves in common. Let node x be the lowest common
ancestor of t1 and t2 in T ; i.e., x is an ancestor of both t1 and t2 , but neither
child of x is. We consider two cases.

Case 1: The path from x to t1 is no longer than the path from x to t2 . Let
t be the sibling of t2 in T . Without loss of generality, assume t is the left
child and t2 is the right child (otherwise, we can swap them). Clearly, t can
be neither t1 nor t2 . Furthermore, it cannot be a proper subtree of any of
t1 , . . . , tn , because then t2 would also be a proper subtree of the same tree.
Finally, t cannot contain t1 as a proper subtree, because then the path from
x to t1 would be longer than the path from x to t2 . We conclude that t must
contain one or more of t3 , . . . , tn . We can therefore swap t1 with t, letting
the result be T ′ .
Because t contains one or more of t3 , . . . , tn , weight(t1 ) ≤ weight(t);
hence, weight(t) − weight(t1 ) ≥ 0. The swap then causes the weights of all
nodes except x on the path from x to the parent of t1 in T to increase by
weight(t)−weight(t1 ). Furthermore, it causes the weights of all nodes except
x on the path from x to the parent of t2 in T to decrease by weight(t) −
weight(t1 ). No other nodes change weight. Because there are at least as
many nodes on the path from x to t2 in T as on the path from x to t1 in
T , the swap cannot increase the cost of the tree. Therefore T ′ is a Huffman
tree.

Case 2: The path from x to t1 is longer than the path from x to t2 . In

this case we assume without loss of generality that t1 is a left child, and
we swap its sibling with t2 . Because Case 1 doesn’t rely on the fact that
weight(t1 ) ≤ weight(t2 ), the same reasoning holds for this case.

We assume the frequency table is provided via two arrays:

CHAPTER 11. OPTIMIZATION I: GREEDY ALGORITHMS 387

Figure 11.4 Algorithm for constructing a Huffman tree

Precondition: chars[1..n] is an array of Chars, n ≥ 1, and freq[1..n] is an

array of positive Nats.
Postcondition: Returns a BinaryTreeNode representing a Huffman tree
for text including freq[i] occurrences of char[i], for 1 ≤ i ≤ n.
HuffmanTree(chars[1..n], freq[1..n])
q ← new InvertedPriorityQueue()
for i ← 1 to n
t ← new BinaryTreeNode(); t.SetRoot(chars[i])
q.Put(t, freq[i])
// Invariant: q contains BinaryTreeNodes which are subtrees of
// some Huffman tree for the given frequency table, and whose priorities
// are their weights as defined by the frequencies of the characters
// contained in their leaves. Each character in chars[1..n] occurs in
// exactly one leaf of one tree.
while q.Size() > 1
w1 ← q.MinPriority(); t1 ← q.RemoveMin()
w2 ← q.MinPriority(); t2 ← q.RemoveMin()
t ← new BinaryTreeNode(); t.SetLeft(t1 ); t.SetRight(t2 )
q.Put(t, w1 + w2 )
return q.RemoveMin()

• chars[1..n], which contains the characters in the table; and

• freq[1..n], which contains positive integers giving the frequencies of the

corresponding characters.

The algorithm should then return a BinaryTreeNode representing a Huff-

man tree for the frequency table. The data in the leaves are characters, and
all other data items in the tree are nil. The algorithm is shown in Figure
11.4. We maintain the trees in an InvertedPriorityQueue using the
weights of the trees as priorities.
Because each iteration of the for loop adds an element to an initially
empty priority queue, iteration i runs in Θ(lg i) time. The for loop therefore
runs in Θ(n lg n) time. After the for loop completes, q contains n elements.
Each iteration of the while loop removes two elements from q and adds one
CHAPTER 11. OPTIMIZATION I: GREEDY ALGORITHMS 388

element. The loop therefore iterates n − 1 times. Each iteration runs in

O(lg n) time. The running time of the while loop is therefore in O(n lg n),
so that the algorithm runs in Θ(n lg n) time.

11.5 Summary
Greedy algorithms provide an efficient mechanism for solving certain opti-
mization problems. The major steps involved in the construction of a greedy
algorithm are:

• Generalize the problem so that a partial solution is given as input.

• Decide upon a selection criterion for incrementally extending partial

solutions.

• Prove that if a given partial solution can be extended to an optimal

solution, then after extending this partial solution using the chosen
selection criterion, the resulting partial solution can also be extended
to an optimal solution.

• Implement the transformation suggested by the incremental extension

using a loop.

Priority queues are often useful in facilitating quick access to the best
extension, as determined by the selection criterion. In many cases, the
extension involves joining pieces of a partial solution in a way that can be
modeled effectively using a DisjointSets structure.
Proving that the incremental extension can be extended to an optimal
solution is essential, because it is not true for all selection criteria. In fact,
there are optimization problems for which there is no greedy solution. In
the next chapter, we will examine a more general, though typically more
expensive, technique for solving optimization problems.

11.6 Exercises
Exercise 11.1 Prove that Kruskal, shown in Figure 11.1, meets its spec-
ification.

Exercise 11.2 Prove that Prim, shown in Figure 11.2, meets its specifica-
tion.
CHAPTER 11. OPTIMIZATION I: GREEDY ALGORITHMS 389

Exercise 11.3 Instead of using the arrays best and bestCost, Prim’s algo-
rithm could use a priority queue to store all of the edges from vertices in the
spanning tree. As vertices are added to the spanning tree, all edges from
these vertices would be added to the priority queue. As edges are removed
from the priority queue, they would need to be checked to see if they con-
nect a vertex in the spanning tree with one that is not in the spanning tree.
Implement this algorithm and analyze its running time assuming the graph
is implemented as a ListGraph.

Exercise 11.4 Implement the single-source shortest path algorithm as out-

lined in Section 11.3. Your algorithm should take a Graph G and a natural
number u < G.Size() and return an array pred[0..G.Size() − 1] such that
pred[i] gives the parent of i in the shortest paths tree; pred[u] should be −1.

Exercise 11.5 Modify your algorithm from Exercise 11.4 to use a priority
queue as suggested in Exercise 11.3. Analyze its running time assuming the
graph is implemented as a ListGraph.

Exercise 11.6 Suppose we wish to solve the single-source shortest path

problem for a graph with unweighted edges; i.e., each edge is understood to
have a length of 1. Prove that the algorithm for Exercise 11.5 can be modified
by replacing the priority queue with a queue (see Exercise 4.11, page 143) to
yield an algorithm for the unweighted single-source shortest path problem.
Analyze the running time of the resulting algorithm, assuming the graph
is implemented as a ListGraph. (This algorithm is known as breadth-first
search.)

Exercise 11.7 Construct a Huffman tree for the string, “banana split”,
and give its resulting encoding in binary. Don’t forget the blank character.

Exercise 11.8 Prove that HuffmanTree, shown in Figure 11.4, meets its
specification.

Exercise 11.9 Suppose we have a set of jobs, each having a positive integer
execution time. We must schedule all of the jobs on a single server so that
at most one job occupies the server at any given time and each job occupies
the server for a length of time equal to its execution time. Our goal is to
minimize the sum of the finish times of all of the jobs. Design a greedy
algorithm to accomplish this and prove that it is optimal. Your algorithm
should run in O(n lg n) time, where n is the number of jobs.
CHAPTER 11. OPTIMIZATION I: GREEDY ALGORITHMS 390

Exercise 11.10 Extend the above exercise to k servers, so that each job is
scheduled on one of the servers.

Exercise 11.11 Suppose we are given a set of events, each having a start
time and a finish time. Each event requires a single room. We wish to assign
events to rooms using as few rooms as possible so that no two events in the
same room overlap (they may, however, be scheduled “back-to-back” with
no break in between). Give a greedy algorithm to accomplish this and prove
that it is optimal. Your algorithm should run in O(n lg n) time.

Exercise 11.12 Repeat the above exercise with the constraint that only
one room is available. The goal is to schedule as many events as possible.

Exercise 11.13 We wish to plan a trip across country in a car that can go
d miles on a full tank of gasoline. We have identified all of the gas stations
along the proposed route. We wish to plan the trip so as to make as few stops
for gasoline as possible. Design a greedy algorithm that gives an optimal
set of stops when given d and an array dist[1..n] such that dist[i] gives the
distance from the starting point to the ith gas station. Your algorithm
should operate in O(n) time.

* Exercise 11.14 The fractional knapsack problem is as follows. We are

given a set of n items, each having a positive weight wi ∈ N and a positive
value vi ∈ N. We are also given a weight bound W ∈ N. We wish to carry
some of these items in a knapsack without exceeding the weight bound. Our
goal is to maximize the total value of the items we carry. Furthermore, the
items are such that we can take a fraction of the item if we wish. Thus, we
wish to maximize
Xn
ai vi
i=1

for rational a1 , . . . , an such that for 1 ≤ i ≤ n, 0 ≤ ai ≤ 1, and subject to

the constraint that
Xn
ai wi ≤ W.
i=1

a. Give a greedy algorithm to find an optimal packing, and prove that

your algorithm is correct. Your algorithm should run in O(n lg n)
time.
CHAPTER 11. OPTIMIZATION I: GREEDY ALGORITHMS 391

b. Show using a specific example that this greedy algorithm does not
always give an optimal solution if we require that each ai be either 0
or 1.

c. Using techniques from Chapter 10, improve the running time of your
algorithm to O(n).

11.7 Chapter Notes

Greedy algorithms were first identified in 1971 by Edmonds [34], though
they actually existed long before then. The theory that underlies greedy
algorithms — matroid theory — was developed by Whitney [112] in the
1930s. See, e.g., Lawler [84] or Papadimitriou and Steiglitz [90] for more
information on greedy algorithms and matroid theory.
The first MST algorithm was given by Boru̇vka [16] in 1926. What is
now known as Prim’s algorithm was first discovered by Jarnı́k [67], and over
25 years later rediscovered independently by Prim [92] and Dijkstra [27];
the latter paper also includes the single-source shortest paths algorithm
outlined in Section 11.3. Kruskal’s algorithm was given by Kruskal [83].
Other MST algorithms have been given by Yao [116], Cheriton and Tarjan
[21], and Tarjan [104]. Other improvements for single-source shortest paths
have been given by Johnson [71, 72], Tarjan [104], and Fredman and Tarjan
[44].
Huffman coding was developed by Huffman [65]. See Lelewer and Hirsch-
berg [85] for a survey of compression algorithms. On the website for this
textbook is a tool for constructing and displaying a Huffman tree for a given
text.
Chapter 12

Optimization II: Dynamic

Programming

In the last chapter, we saw that greedy algorithms are efficient solutions to
certain optimization problems. However, there are optimization problems
for which no greedy algorithm exists. In this chapter, we will examine a more
general technique, known as dynamic programming, for solving optimization
problems.
Dynamic programming is a technique of implementing a top-down solu-
tion using bottom-up computation. We have already seen several examples
of how top-down solutions can be implemented bottom-up. Dynamic pro-
gramming extends this idea by saving the results of many subproblems in
order to solve the desired problem. As a result, dynamic programming algo-
rithms tend to be more costly, in terms of both time and space, than greedy
algorithms. On the other hand, they are often much more efficient than
straightforward recursive implementations of the top-down solution. Thus,
when greedy algorithms are not possible, dynamic programming algorithms
are often the most appropriate.

12.1 Making Change

Suppose we wish to produce a specific value n ∈ N from a given set of coin
denominations d1 < d2 < · · · < dk , each of which is a positive integer. Our
goal is to achieve a value of exactly n using a minimum number of coins. To
ensure that it is always possible to achieve a value of exactly n, we assume
that d1 = 1 and that we have as many coins in each denomination as we
need.

392
CHAPTER 12. OPTIMIZATION II: DYNAMIC PROGRAMMING 393

An obvious greedy strategy is to choose at each step the largest coin that
does not cause the total to exceed n. For some sets of coin denominations,
this strategy will result in the minimum number of coins for any n. However,
suppose n = 30, d1 = 1, d2 = 10, and d3 = 25. The greedy strategy first
takes 25. At this point, the only denomination that does not cause the total
to exceed n is 1. The greedy strategy therefore gives a total of six coins: one
25 and five 1s. This solution is not optimal, however, as we can produce 30
with three 10s.
Let us consider a more direct top-down solution. If k = 1, then dk = 1,
so the only solution contains n coins. Otherwise, if dk > n, we can reduce
the size of the problem by removing dk from the set of denominations, and
the solution to the resulting problem is the solution to the original problem.
Finally, suppose dk ≤ n. There are now two possibilities: the optimal
solution either contains dk or it does not. In what follows, we consider these These two
possibilities are
two cases separately. not exclusive —
Let us first consider the case in which the optimal solution does not there could be
contain dk . In this case, we do not change the optimal solution if we remove one optimal
solution that
dk from the set of denominations. We therefore have reduced the original contains dk and
problem to a smaller problem instance. another that
does not.
Now suppose the optimal solution contains dk . Suppose we remove one
dk coin from this optimal solution. What remains is an optimal solution
to the instance with the same set of denominations and a target value of
n − dk . Now working in the other direction, if we have the optimal solution
to the smaller instance, we can obtain an optimal solution to the original
instance by adding a dk coin. Again, we have reduced the original problem
to a smaller problem instance.
To summarize, when dk ≤ n, the optimal solution can be obtained from
the optimal solution to one of two smaller problem instances. We have no
way of knowing in advance which of these smaller instances is the right
one; however, if we obtain both of them, we can compare the two resulting
candidate solutions. The one with fewer coins is the optimal solution. In
fact, if we could quickly determine which of these smaller instances would
yield fewer coins, we could use this test as the selection criterion for a greedy
algorithm. Therefore, let us focus for now on the more difficult aspect of
this problem — that of determining the minimum number of coins in an
optimal solution.
Based on the above discussion, the following recurrence gives the mini-
mum number of coins needed to obtain a value of n from the denominations
CHAPTER 12. OPTIMIZATION II: DYNAMIC PROGRAMMING 394

d1 , . . . , dk :

n
 if k = 1
C(n, k) = C(n, k − 1) if k > 1, dk > n (12.1)

min(C(n, k − 1), C(n − dk , k) + 1) if k > 1, n ≥ dk .


This recurrence gives us a recursive algorithm for computing C(n, k).

However, the direct recursive implementation of this recurrence is inefficient.
In order to see this, let us consider the special case in which di = i for
1 ≤ i ≤ k and n ≥ k 2 . Then for k > 1, the computation of C(n, k)
requires the computation of C(n, k − 1) and C(n − k, k). The computation
of C(n−k, k) then requires the computation of C(n−k, k −1). Furthermore,

n − k ≥ k2 − k
= k(k − 1)
≥ (k − 1)2

for k ≥ 1. Thus, when n ≥ k 2 , the computation of C(n, k) requires the

computation of two values, C(n1 , k−1) and C(n2 , k−1), where n1 ≥ (k−1)2
and n2 ≥ (k − 1)2 . It is then easily shown by induction on k that C(n, k)
requires the computation of 2k−1 values C(ni , 1), where ni ≥ 1 for 1 ≤ i ≤
2k−1 . In such cases, the running time is exponential in k.
A closer look at the above argument reveals that a large amount of
redundant computation is taking place. For example, the subproblem C(n−
2k + 2, k − 2) must be computed twice:

1. C(n, k) requires the computation of C(n, k − 1), which requires C(n −

k + 1, k − 1), which requires C(n − 2k + 2, k − 1), which requires
C(n − 2k + 2, k − 2); and

2. C(n, k) also requires C(n − k, k), which requires C(n − k, k − 1), which
requires C(n − k, k − 2), which requires C(n − 2k + 2, k − 2).

Applying this reasoning again to both computations of C(n − 2k + 2, k − 2),

we can see that C(n − 4k + 8, k − 4) must be computed four times. More
generally, for even i < k, C(n − ik + i2 /2, k − i) must be computed 2i/2 times
for n ≥ k 2 .
In order to avoid the redundant computation that leads to exponential
running time, we can compute all values C(i, j) for 0 ≤ i ≤ n, 1 ≤ j ≤ k,
saving them in an array. If we compute recurrence (12.1) bottom-up, rather
than top-down, we will have all of the values we need in order to compute
CHAPTER 12. OPTIMIZATION II: DYNAMIC PROGRAMMING 395

each C(i, j) in constant time. All (n + 1)k of these values can therefore be
computed in Θ(nk) time. Once all of these values have been computed, then
the optimal collection of coins can be constructed in a greedy fashion, as
suggested above. The algorithm is shown in Figure 12.1. This algorithm is
easily seen to use Θ(nk) time and space.
A characteristic of this problem that is essential in order for the dynamic
programming approach to work is that it is possible to decompose a large
problem instance into smaller problem instances in a way that optimal so-
lutions to the smaller instances can be used to produce an optimal solution
to the larger instance. This is, of course, one of the main principles of the
top-down approach. However, this characteristic may be stated succinctly
for optimization problems: For any optimal solution, any portion of that
solution is itself an optimal solution to a smaller instance. This principle
is known as the principle of optimality. It applies to the change-making
problem because any sub-collection of an optimal collection of coins is it-
self an optimal collection for the value it yields; otherwise, we could replace
the sub-collection with a smaller sub-collection yielding the same value, and
obtain a better solution to the original instance.
The principle of optimality usually applies to optimization problems, but
not always in a convenient way. For example, consider the problem of finding
a longest simple path in a graph from a given vertex u to a given vertex v. A simple path in
a graph is a path
If we take a portion of the longest path, say from x to y, this subpath is in which each
not necessarily the longest simple path from x to y in the original graph. vertex appears at
However, it is guaranteed to be the longest simple path from x to y in the most once.

subgraph consisting of only those vertices on that subpath and all edges
between them in the original graph. Thus, a subproblem consists of a start
vertex, a final vertex, and a subset of the vertices. Because a graph with
n vertices has 2n subsets of vertices, there are an exponential number of
subproblems to solve. Thus, in order for dynamic programming to be an
effective design technique, the principle of optimality must apply in a way
that yields relatively few subproblems.
One characteristic that often leads to relatively few subproblems, while
at the same time causing direct recursive implementations to be quite ex-
pensive, is that the top-down solution results in overlapping subproblems.
As we have already discussed, the top-down solution for the change-making
problem can result in two subproblems which have a subproblem in common.
This overlap results in redundant computation in the direct recursive imple-
mentation. On the other hand, it reduces the total number of subproblems,
so that the dynamic programming approach is more efficient.
CHAPTER 12. OPTIMIZATION II: DYNAMIC PROGRAMMING 396

Figure 12.1 Algorithm for computing the minimum number of coins needed
to achieve a given value

Precondition: d[1..k] is an array of Ints such that 1 = d[1] < d[2] < · · · <
d[k], and n is a Nat.
Postcondition: Returns an array A[1..k] such that A[i] gives the number
of coins of denomination d[i] in a minimum-sized collection of coins with
value n.
Change(d[1..k], n)
C ← new Array[0..n, 1..k]; A ← new Array[1..k]
for i ← 0 to n
C[i, 1] ← i
for i ← 0 to n
for j ← 2 to k
if i < d[j]
C[i, j] ← C[i, j − 1]
else
C[i, j] ← Min(C[i, j − 1], C[i − d[j], j] + 1)
for j ← 1 to k
A[j] ← 0
i ← n; j ← k P
k
// Invariant: l=1 A[l]d[l] = n − i, and there is an optimal solution
// that includes all of the coins in A[1..k], but no additional coins from
// d[j + 1..k].
while j > 1
if i < d[j] or C[i, j − 1] < C[i − d[j], j] + 1
j ←j−1
else
A[j] ← A[j] + 1; i ← i − d[j]
A[1] ← i
return A[1..k]
CHAPTER 12. OPTIMIZATION II: DYNAMIC PROGRAMMING 397

12.2 Chained Matrix Multiplication

Recall that the product AB, where A is a k × m matrix and B is an m × n
matrix, is the k × n matrix C such that
m
X
Cij = Ail Blj for 1 ≤ i ≤ k, 1 ≤ j ≤ n.
l=1

If we were to compute the matrix product by directly computing each of the

kn sums, we would perform a total of kmn scalar multiplications.
Now suppose we wish to compute the product,

M1 M2 · · · Mn ,

where Mi is a di−1 ×di matrix for 1 ≤ i ≤ n. Because matrix multiplication is

associative, we have some choice over the order in which the multiplications
are performed. For example, to compute M1 M2 M3 , we may either

• first compute M1 M2 , then multiply on the right by M3 ; or

• first compute M2 M3 , then multiply on the left by M1 .

In other words, we may compute either (M1 M2 )M3 or M1 (M2 M3 ).

Now suppose d0 = 2, d1 = 3, d2 = 4, and d3 = 1. Then the three
matrices are dimensioned as follows:

• M1 : 2 × 3;

• M2 : 3 × 4; and

• M3 : 4 × 1.

If we compute (M1 M2 )M3 , we first multiply a 2×3 matrix by a 3×4 matrix,

then we multiply the resulting 2 × 4 matrix by a 4 × 1 matrix. The total
number of scalar multiplications is

2 · 3 · 4 + 2 · 4 · 1 = 32.

On the other hand, if we compute M1 (M2 M3 ), we first multiply a 3 × 4

matrix by a 4 × 1 matrix, then we multiply the resulting 3 × 1 matrix by a
2 × 3 matrix. The total number of scalar multiplications is

3 · 4 · 1 + 2 · 3 · 1 = 18.
CHAPTER 12. OPTIMIZATION II: DYNAMIC PROGRAMMING 398

Thus, the way in which the matrices are parenthesized can affect the
number of scalar multiplications performed in computing the matrix prod-
uct. This fact motivates an optimization problem: Given a sequence of
positive integer dimensions d0 , . . . , dn , determine the minimum number of
scalar multiplications needed to compute the product M1 · · · Mn , assuming
Mi is a di−1 × di matrix for 1 ≤ i ≤ n, and that the number of scalar
multiplications required to multiply two matrices is as described above.
Various greedy strategies might be applied to this problem, but none can
guarantee an optimal solution. Let us therefore look for a direct top-down
solution to the problem of finding the minimum number of scalar multipli-
cations for a product Mi · · · Mj . Let us focus on finding the last matrix
multiplication. This multiplication will involve the products Mi · · · Mk and
Mk+1 · · · Mj for some k, 1 ≤ k < n. The sizes of these two matrices are
di−1 × dk and dk × dj . Therefore, once these two matrices are computed, an
additional di−1 dk dj scalar multiplications must be performed. The principle
of optimality clearly holds for this problem, as a better way of computing
either sub-product results in fewer total scalar multiplications. Therefore,
the following recurrence gives the minimum number of scalar multiplications
needed to compute Mi · · · Mj :

0 if i = j
m(i, j) = (12.2)
 min (m(i, k) + m(k + 1, j) + di−1 dk dj ) if i < j
i≤k<j

In order to compute m(i, j), 2(j − i) subproblems need to be solved. It is

easily seen that there is a great deal of overlap between these subproblems.
Therefore, dynamic programming is appropriate for computing m(i, j). We
need a matrix m[1..n, 1..n]. In order to compute m[i, j], we need to use
values in row i to the left of column j and values in column j below row i.
It therefore makes sense to compute m by rows from bottom to top, and left
to right within each row. The algorithm is given in Figure 12.2. It is easily
seen to run in Θ(n3 ) time and to use Θ(n2 ) space.

12.3 All-Pairs Shortest Paths

In Section 11.3, we discussed the single-source shortest paths problem for
directed graphs. In this section, we generalize the problem to all pairs of
vertices; i.e., we wish to find, for each pair of vertices u and v, a shortest
path from u to v. An obvious solution is to apply Dijkstra’s algorithm n
times, each time using a different vertex as the source. This would result
CHAPTER 12. OPTIMIZATION II: DYNAMIC PROGRAMMING 399

Figure 12.2 Chained matrix multiplication algorithm

Precondition: d[0..n] is an array of positive Nats, and n is a positive Nat.

Postcondition: Returns the minimum number of scalar multiplications
needed to compute the product M1 · · · Mn , where Mi is a d[i − 1] × d[i]
matrix for 1 ≤ i ≤ n.
ChainedMatrixMult(d[0..n])
m ← new Array[1..n, 1..n]
for i ← n to 1 by −1
m[i, i] ← 0
for j ← i + 1 to n
m[i, j] ← ∞
for k ← i to j − 1
m[i, j] ← Min(m[i, j], m[i, k] + m[k + 1, j] + d[i − 1]d[k]d[j])
return m[1, n]

in an algorithm with running time in Θ(n3 ). Although the algorithm we

present in this section is no faster asymptotically, it serves as a good example
of how certain space optimizations can sometimes be made for dynamic
programming algorithms. It also serves as an illustration of how dynamic
programming can be applied to problems that are not optimization problems
in the strictest sense of the word.
Let G = (V, E) be a directed graph, and let len : V 2 → N ∪ {∞} be a
function giving the length of each edge, so that
• len(u, u) = 0 for u ∈ V ; and

• len(u, v) = ∞ iff u 6= v and (u, v) 6∈ E, for (u, v) ∈ V 2 .

We wish to find, for each ordered pair (u, v) ∈ V 2 , the length of the shortest
path from u to v; if there is no such path, we define the length to be ∞.
Note that we have simplified the problem so that instead of finding the
actual paths, we will only be finding their lengths.
This optimization problem is somewhat nonstandard in that the objec-
tive function is not a numeric-valued function. Instead, its range can be
thought of as a matrix of values. However, the optimum is well-defined, as
it occurs when all values are simultaneously minimized, and this is always
possible.
CHAPTER 12. OPTIMIZATION II: DYNAMIC PROGRAMMING 400

Let p be a shortest path from i to j, and consider any vertex k other

than i or j. Then either k is in p or it isn’t. If k is not in p, then p remains
the shortest path from i to j if we remove k from the graph. Otherwise, we
can break p into a path from i to k and a path from k to j. Clearly, each
of these paths are shortest paths between their endpoints. Thus, if we can
find the shortest path from i to k and the shortest path from k to j, we can
determine the shortest path from i to j.
A shortcoming to this approach is that we haven’t actually reduced the
size of the problem, as the shortest paths from i to k and k to j are with
respect to the original graph. One way to avoid this shortcoming is to
generalize the problem so that a set of possible intermediate vertices is given
as additional input. The problem is then to find, for each ordered pair (i, j)
of vertices, the length of the shortest path from i to j such that all vertices
other than i and j on this path belong to the given set. If the given set is
V , then the result is the solution to the all-pairs shortest paths problem.
In order to keep the number of subproblems from being too large, we can
restrict the sets we allow as input. Specifically, our additional input can be
a natural number k, which denotes the set of all natural numbers strictly
less than k.
Let Lk (i, j) denote the length of the shortest path from i to j with
intermediate vertices strictly less than k, where 0 ≤ i < n, 0 ≤ j < n, and
0 ≤ k ≤ n. Using the above reasoning, we have the following recurrence for
Lk (i, j):
(
len(i, j) if k = 0
Lk (i, j) =
min(Lk−1 (i, j), Lk−1 (i, k − 1) + Lk−1 (k − 1, j)) if k > 0.
(12.3)
We can then implement a dynamic programming algorithm to compute
all Lk (i, j) using a 3-dimensional array. However, we can save a great deal
of space by making some observations. Note that in order to compute an
entry Lk (i, j), for k > 0, we only use entries Lk−1 (i, j), Lk−1 (i, k − 1),
and Lk−1 (k − 1, j). We claim that Lk−1 (i, k − 1) = Lk (i, k − 1) and that
Lk−1 (k − 1, j) = Lk (k − 1, j). To see this, note that
Lk (i, k − 1) = min(Lk−1 (i, k − 1), Lk−1 (i, k − 1) + Lk−1 (k − 1, k − 1))
= Lk−1 (i, k − 1)
and
Lk (k − 1, j) = min(Lk−1 (k − 1, j), Lk−1 (k − 1, k − 1) + Lk−1 (k − 1, j))
= Lk−1 (k − 1, j).
CHAPTER 12. OPTIMIZATION II: DYNAMIC PROGRAMMING 401

Figure 12.3 Floyd’s algorithm for all-pairs shortest paths

Precondition: G refers to a Graph in which the data associated with each

edge is a Nat giving its length.
Postcondition: Returns an array L[0..n − 1, 0..n − 1] such that L[i, j] is
the length of the shortest path from i to j in G.
Floyd(G)
n ← G.Size(); L ← new Array[0..n − 1, 0..n − 1]
for i ← 0 to n − 1
for j ← 0 to n − 1
if i = j
L[i, j] ← 0
else
d ← G.Get(i, j)
if d = nil
L[i, j] ← ∞
else
L[i, j] ← d
for k ← 1 to n
for i ← 0 to n − 1
for j ← 0 to n − 1
L[i, j] ← Min(L[i, j], L[i, k − 1] + L[k − 1, j])
return L[0..n − 1, 0..n − 1]

As a result, we can use a 2-dimensional array L[0..n − 1, 0..n − 1] to

represent Lk−1 . We can then transform this array into Lk by updating each
value in turn. The algorithm, known as Floyd’s algorithm, is shown in Figure
12.3. We assume that the length of each edge is given by its key. It is easily Figure 12.3
corrected
seen that, regardless of whether G is implemented as a MatrixGraph or 5/7/12.
a ListGraph, the algorithm runs in Θ(n3 ) time and uses only a constant
amount of space other than what is required for input and output.

12.4 The Knapsack Problem

In Exercise 11.14 (page 390), we introduced the fractional knapsack problem.
One part of this exercise was to show that the greedy algorithm for the
CHAPTER 12. OPTIMIZATION II: DYNAMIC PROGRAMMING 402

fractional knapsack problem does not extend to the so-called 0-1 knapsack
problem — the variation in which the items cannot be broken. Specifically,
in this variation we are given a set of n items, each having a positive weight
wi ∈ N and a positive value vi ∈ N, and a weight bound W ∈ N. We wish
to find a subset S ⊆ {1, . . . , n} that maximizes
X
vi
i∈S

subject to the constraint that

X
wi ≤ W.
i∈S

To solve this problem, first note that either item n is in an optimal

solution, or it isn’t. If it is, then we can obtain an optimal solution by
solving the problem in which item n has been removed and the weight bound
has been decreased by wn . Otherwise, we can obtain an optimal solution
by solving the problem in which item n has been removed. We therefore
have the following recurrence giving the optimal value Vi (j) that can be
obtained from the first i items with a weight bound of j, where 0 ≤ i ≤ n
and 0 ≤ j ≤ W :

0
 if i = 0
Vi (j) = Vi−1 (j) if i > 0, j < wi (12.4)

max(Vi−1 (j), Vi−1 (j − wi ) + vi ) otherwise.


The optimal value is then given by Vn (W ).

It is not hard to see that the optimal value can be computed in Θ(nW )
time and space using dynamic programming — the details are left as an
exercise. However, suppose W is much larger than the values of the items.
In this case, another approach might be more appropriate. Let
n
X
V = vi .
i=1

Let us then compute the minimum weight required to achieve each possible
value v ≤ V . The largest value v yielding a minimum weight no larger than
W is then our optimal value.
Taking this approach, we observe that item n is either in the set of items
for which value v can be achieved with minimum weight, or it isn’t. If it
CHAPTER 12. OPTIMIZATION II: DYNAMIC PROGRAMMING 403

is, then the minimum weight can be computed by removing item n and
finding the minimum weight needed to achieve a value of v − vn . Otherwise,
the minimum weight can be computed by removing item n. The following
recurrence therefore gives the minimum weight Wi (j) needed to achieve a
value of exactly j from the first i items, for 0 ≤ i ≤ n, 0 ≤ j ≤ V :



 0 if j = 0

∞ if i = 0, j > 0
Wi (j) = (12.5)
Wi−1 (j)
 if i > 0, 0 < j < vi


min(W (j), W (j − v ) + w ) otherwise.
i−1 i−1 i i

The optimal value is then the maximum j ≤ V such that Wn (j) ≤ W .

It is not hard to see that a dynamic programming algorithm based on
this recurrence can find the optimal value in Θ(nV ) time and space. Again,
we leave the details as an exercise. Note that we could potentially improve
the algorithm further if we could first find a better upper bound than V for
the optimal value. This would allow us to reduce the number of columns we
need to compute in our array. We will explore this idea further in Chapter
17.

12.5 Summary
Dynamic programming algorithms provide more power for solving optimiza-
tion problems than do greedy algorithms. Efficient dynamic programming
algorithms can be found when the following conditions apply:
• The principle of optimality can be applied to decompose the problem
into subinstances of the same problem.

• There is significant overlap between the subinstances.

• The total number of subinstances, including those obtained by recur-

sively decomposing subinstances, is relatively small.
Although dynamic programming algorithms proceed bottom-up, the first
step in formulating a dynamic programming algorithm is to formulate a top-
down solution. This top-down solution usually takes the form of a recurrence
for computing the optimal value of the objective function. The top-down
solution is then implemented bottom-up, storing all of the solutions to the
subproblems. In some cases, we optimize the space usage by discarding some
of these solutions.
CHAPTER 12. OPTIMIZATION II: DYNAMIC PROGRAMMING 404

Because dynamic programming algorithms typically solve subinstances

that are not used in the optimal solution, they tend to be less efficient than
greedy algorithms. Hence, greedy algorithms are preferred when they exist.
However, for many problems, there are no greedy algorithms that guarantee
optimal solutions. In such cases, dynamic programming algorithms may be
the most efficient.
Although the examples in this chapter have all been optimization prob-
lems, it is not hard to see that dynamic programming can be applied to other
problems as well. Any computation that can be expressed as a recurrence
can be computed bottom-up, yielding a dynamic programming solution. We
explore some examples in the exercises.

12.6 Exercises
Exercise 12.1 Prove by induction on n + k that C(n, k), as defined in
recurrence (12.1), gives the minimum number of coins needed to give a
value of exactly n if the denominations are d1 < d2 < · · · < dk and d1 = 1.

Exercise 12.2 Prove that Change, shown in Figure 12.1, meets its spec-
ification. You do not need to focus on the first half of the algorithm; i.e.,
you can assume that C(i, j), as defined in recurrence (12.1), is assigned to
C[i, j]. Furthermore, you may use the result of Exercise 12.1 in your proof.

* Exercise 12.3 As we have seen, the greedy algorithm suggested in Sec-

tion 12.1 works for some sets of coin denominations but not for others.

a. Prove that for denominations d1 < d2 < · · · < dk , where k > 1, if the
greedy algorithm fails for some value, then it must fail for some value
n < dk + dk−1 .

b. Devise an efficient dynamic programming algorithm which takes as

input a set of denominations and returns true if the greedy algorithm
always works for this set, or returns false otherwise. You may assume
that the denominations are given in increasing order. Your algorithm
should use O(M k) time and space, where M is the largest denomina-
tion and k is the number of denominations.

Exercise 12.4 Prove by induction on j − i that m(i, j), as defined in re-

currence 12.2, is the minimum number of scalar multiplications needed to
compute a product Mi · · · Mj , where Mk is a dk−1 × dk matrix for i ≤ k ≤ j.
CHAPTER 12. OPTIMIZATION II: DYNAMIC PROGRAMMING 405

Exercise 12.5 Prove by induction on k that Lk (i, j), as defined in recur-

rence 12.3, gives the length of the shortest path from i to j in which all
intermediate vertices are strictly less than k.

Exercise 12.6
a. Modify Floyd’s algorithm (Figure 12.3) so that it returns an array
S[0..n − 1, 0..n − 1] such that for i 6= j, S[i, j] gives the vertex k such
that (i, k) is the first edge in a shortest path from i to j. If there is
no path from i to j, or if i = j, then S[i, j] should be −1.
b. Give an algorithm that takes the array S[0..n − 1, 0..n − 1] defined
above, along with i and j such that 0 ≤ i < n and 0 ≤ j < n, and
prints the vertices along a shortest path from i to j. The first vertex
printed should be i, followed by the vertices in order along the path,
until the last vertex j is printed. If i = j, only i should be printed. If
there is no path from i to j, a message to that effect should be printed.
Your algorithm should run in O(n) time.

Exercise 12.7 Give an algorithm for the 0-1 knapsack problem that runs
in O(nW ) time and space, where n is the number of items and W is the
weight bound. Your algorithm should use dynamic programming to compute
recurrence (12.4) for 0 ≤ i ≤ n and 0 ≤ j ≤ W , then use these values to
guide a greedy algorithm for selecting the items to put into the knapsack.
Your algorithm should return an array selected[1..n] of booleans such that
selected[i] is true iff item i is in the optimal packing.

Exercise 12.8 Repeat Exercise 12.7 using recurrence (12.5) instead of (12.4).
Your algorithm should use Θ(nV ) time and space, where n is the number
of items and V is the total value of all the items.

Exercise 12.9 Let A[1..n] be an array of integers. An increasing subse-

quence of A is a sequence of indices hi1 , . . . , ik i such that ij < ij+1 and
A[ij ] < A[ij+1 ] for 1 ≤ j < k. (Note that the indices in the subsequence
are not necessarily contiguous.) A longest increasing subsequence of A is an
increasing subsequence of A with maximum length.
a. Give a recurrence for L(i), the length of the longest increasing subse-
quence of A[1..i] that ends with i, where 1 ≤ i ≤ n.
b. Give a dynamic programming algorithm that prints the indices of
a longest increasing subsequence of A[1..n]. Your algorithm should
operate in O(n2 ) time.
CHAPTER 12. OPTIMIZATION II: DYNAMIC PROGRAMMING 406

Exercise 12.10 Let A[1..m] and B[1..n] be two arrays. An array C[1..k]
is a common subsequence of A and B if there are two sequences of indices
hi1 , . . . , ik i and hj1 , . . . , jk i such that

• i1 < i2 < · · · < ik ;

• j1 < j2 < · · · < jk ; and

• C[l] = A[il ] = B[jl ] for 1 ≤ l ≤ k.

A longest common subsequence of A and B is a common subsequence of A

and B with maximum size.

a. Give a recurrence for L(i, j), the length of the longest common subse-
quence of A[1..i] and B[1..j].

b. Give a dynamic programming algorithm that returns the longest com-

mon subsequence of A[1..m] and B[1..n]. Your algorithm should op-
erate in O(mn) time.

Exercise 12.11 A palindrome is a string that reads the same from right
to left as it does from left to right (“abcba”, for example). Give a dynamic
programming algorithm that takes a String (see Figure 4.17 on page 144)
s as input, and returns a longest palindrome contained as a substring within
s. Your algorithm should operate in O(n2 ) time, where n is the length of
s. You may use the results of Exercise 4.13 (page 143) in analyzing your
algorithm. [Hint: For each pair of indices i ≤ j, determine whether the
substring from i to j is a palindrome.]

* Exercise 12.12 Suppose we have two k-dimensional boxes A and B

whose k dimensions are a1 , . . . , ak and b1 , . . . , bk , respectively. We say that
A fits inside of B if there is a permutation ai1 , . . . , aik of the dimensions
of A such that aij < bj for 1 ≤ j ≤ k. Design a dynamic programming
algorithm that takes as input a positive integer k and the dimensions of n k-
dimensional boxes, and returns the maximum size of any subset of the boxes
that can be ordered such that each box (except the last) in the ordering fits
inside of the next. Your algorithm should run in O(max(n2 k, nk lg k)) time
in the worst case. Note that your algorithm doesn’t need to return a subset
or an ordering — only the size of the subset.

Exercise 12.13 Let G = (V, E) be a directed graph. The transitive closure

of G is a directed graph G′ = (V, E ′ ), where E ′ is the set of all (u, v) ∈ V 2
CHAPTER 12. OPTIMIZATION II: DYNAMIC PROGRAMMING 407

Figure 12.4 Polygons illustrating Exercise 12.14

a) b) c)

such that u 6= v and there is a path from u to v in G. Give an O(n3 ) dynamic

programming algorithm to produce a MatrixGraph that is the transitive
closure of a given MatrixGraph.

Exercise 12.14 A convex polygon is a polygon whose interior angles are

all less than 180 degrees. For example, in Figure 12.4, polygon a is convex,
but polygon b is not. A triangulation of a convex polygon is a set of non-
intersecting diagonals that partition the polygon into triangles, as shown in
Figure 12.4 c. Give a dynamic programming algorithm that takes as input
a convex polygon and produces a triangulation that minimizes the sum of
the lengths of the diagonals, where the length of an edge (x1 , y1 ), (x2 , y2 ) is
given by p
(x1 − x2 )2 + (y1 − y2 )2 .
You may assume that the polygon is represented as a sequence of points in
the Cartesian plane hp1 , p2 , . . . , pn i such that the edges of the polygon are
(p1 , p2 ), (p2 , p3 ), . . . , (pn−1 , pn ), and (pn , p1 ). You may further assume that
n ≥ 3. Your algorithm should run in O(n3 ) time.

* Exercise 12.15 A chain is a rooted tree with exactly one leaf. We are
given a chain representing a sequence of n pipelined processes. Each node i
in the chain represents a process and has a positive execution time ei ∈ N.
Each edge (i, j) has a positive communication cost cij ∈ N. For edge (i, j),
if processes i and j are executed on separate processors, the time needed to
send data from process i to process j is cij ; if the processes are executed on
the same processor, this time is 0. We wish to assign processes to processors
such that each processor has total weight no more than a given value B ∈ N.
The weight of a processor is given by the sum of the execution times of the
processes assigned to that processor, plus the sum of the communication
CHAPTER 12. OPTIMIZATION II: DYNAMIC PROGRAMMING 408

Figure 12.5 An example illustrating Exercise 12.15

3
Processor 1: Weight = 8
2

B = 20
3 Communication cost = 2
5

Processor 2: Weight = 17
8

costs of edges between tasks on that processor and tasks on other processors
(see Figure 12.5). The communication cost of an assignment is the sum of
the communication costs of edges that connect nodes assigned to different
processors.
Give a dynamic programming algorithm that finds the minimum com-
munication cost of any assignment of processes to processors such that each
processor has weight no more than B. Note that we place no restriction on
the number of processors used. Your algorithm should run in O(n2 ) time.
Prove that your algorithm is correct.

Exercise 12.16 Given two strings x and y, we define the edit distance from
x to y as the minimum number of operations required to transform x into
y, where the operations are chosen from the following:

• insert a character;

• delete a character; or
CHAPTER 12. OPTIMIZATION II: DYNAMIC PROGRAMMING 409

• change a character.

** a. Prove that there is an optimal sequence of operations transforming x

to y in which the edits proceed from left to right.

b. Using the above result, give a dynamic programming algorithm that

takes as input x and y and returns the edit distance from x to y. Your
algorithm should run in O(mn) time, where m is the length of x and
n is the length of y. Prove that your algorithm is correct.

Exercise 12.17 Suppose we wish to store n keys, k1 < k2 < · · · < kn , in a

binary search tree (see Chapter 6). We define the cost of a look-up of key
ki as di + 1, where di is the depth of ki in the tree. Thus, the cost of a The depth of a
node in a tree is
look-up is simply the number of nodes examined. Suppose further that we the number of
are given, for 1 ≤ i ≤ n, the probability pi ∈ R>0 that any given look-up is edges in the path
for key ki . We assume that from the root to
n that node.
X
pi = 1.
i=1

We say that a binary search tree containing these keys is optimal if the
expected cost of a look-up in this tree is minimum over the set of all binary
search trees containing these keys.

a. Let us extend the definition of the cost of a look-up to pertain to

a specific subtree, so that the cost with respect to subtree T is the
number of nodes in T examined during that look-up. For i ≤ j, let Sij
be the set of all binary search trees with keys k1 , . . . , kn such that there
is a subtree containing exactly the keys ki , . . . , kj . Let Cij denote the
minimum over Sij of the expected cost of a look-up with respect to
the subtree containing keys ki , . . . , kj . Prove that

pi if i = j


j
Cij = X
 min
i≤k≤j
 (C i,k−1 + C k+1,j ) + pk if i < j
k=i

* b. Give a dynamic programming algorithm that takes as input p1 , . . . , pn

and returns the expected cost of a look-up for an optimal binary search
tree whose keys k1 < k2 < · · · < kn have the given probabilities. (Note
that we don’t need the values of the keys in order to compute this
value.) Your algorithm should run in O(n3 ) time and O(n2 ) space.
CHAPTER 12. OPTIMIZATION II: DYNAMIC PROGRAMMING 410

** c. Suppose rij is the root of an optimal binary search containing the

keys ki , . . . , kj , where i ≤ j. Prove that ri,j−1 ≤ rij ≤ ri+1,j for
1 ≤ i < j ≤ n.

* d. Using the above result, improve your algorithm to run in O(n2 ) time.

Exercise 12.18 Give a dynamic programming algorithm that takes as in-

put two natural numbers k ≤ n and returns the probability that flipping a
fair coin n times yields at least k heads. Your algorithm should run in O(n)
time. Prove that your algorithm is correct.

* Exercise 12.19 Give a dynamic programming algorithm that takes as

input a natural number n and returns the number of different orderings of n
elements using < and/or =. For example, for n = 3, there are 13 orderings:

x<y<z x<z<y y<x<z y<z<x

z<x<y z<y<x x=y<z z<x=y
x=z<y y<x=z y=z<x x<y=z
x=y=z

Your algorithm should run in O(n2 ) time and use O(n) space. Prove that
your algorithm is correct.

* Exercise 12.20 Suppose we have a mathematical structure containing

three elements, a, b, and c, and a multiplication operation given by the
following table:
a b c
a a c a
b c b b
c a c b
Note that this multiplication operation is neither commutative nor associa-
tive. Give a dynamic programming algorithm that takes as input a string
over a, b, and c, and returns a boolean indicating whether it is possible to
parenthesize the string so that the result is a. (For example, if we paren-
thesize abca as (a(bc))a, we get a result of a.) Your algorithm should run
in O(n3 ) time, where n is the length of the input string. Prove that your
algorithm is correct.

* Exercise 12.21 Suppose we are given an array L[1..n] of positive integers

representing the lengths of successive words in a paragraph. We wish to
format the paragraph so that each line contains no more than m characters,
CHAPTER 12. OPTIMIZATION II: DYNAMIC PROGRAMMING 411

including a single blank character between adjacent words on the same line.
Furthermore, we wish to minimize a “sloppiness” criterion. Specifically, we
wish to minimize the following objective function:
k−1
X
f (m − ci ),
i=1

where k is the total number of lines used, f : N → N is some nondecreasing

function, and ci is the number of characters (including blanks between adja-
cent words) on line i. Give an efficient dynamic programming algorithm for
computing the optimal arrangement. Your algorithm should run in O(n2 )
time and use O(n) space. [Hint: Reduce this problem to the problem for
which the measure of sloppiness includes the last line — i.e., the optimiza-
tion function is as above, but with k − 1 replaced by k.]

* Exercise 12.22 We are given a set of n points (xi , yi ), where each xi

and yi is a real number and all the xi s are distinct. A bitonic tour of these
points is a cycle that begins at the rightmost point, proceeds strictly to
the left to the leftmost point, then proceeds strictly to the right to return
to the rightmost point; furthermore, this cycle contains every point exactly
once (see Figure 12.6). We wish to find the bitonic tour having minimum
Euclidean length; i.e., the distance between two points (x1 , y1 ) and (x2 , y2 )
is given by p
(x1 − x2 )2 + (y1 − y2 )2 .
Give an efficient dynamic programming algorithm for finding a minimum-
length bitonic tour. Your algorithm should use O(n2 ) time and space. [Hint:
Reduce this problem to that of finding a minimum-length bitonic path that
includes all the points exactly once, but does not return to the starting
point.]

* Exercise 12.23 Suppose we were to modify the scheduling problem of

Section 11.1 so that each job also has a natural number execution time,
which must be fulfilled without interruption. Thus, a schedule including job
i must have job i scheduled for ei contiguous time units lying between time
0 and time di , where ei is the execution time and di is the deadline of job
i. Give an efficient dynamic programming algorithm to generate a schedule
with maximum value. Your algorithm should use O(n(m + lg n)) time and
O(mn) space, where n is the number of jobs and m is the maximum deadline.
CHAPTER 12. OPTIMIZATION II: DYNAMIC PROGRAMMING 412

Figure 12.6 Bitonic and non-bitonic tours

a) A bitonic tour. b) A tour that is not bitonic.

12.7 Chapter Notes

The mathematical foundation for dynamic programming was given by Bell-
man [10]. The Change algorithm in Figure 12.1 is due to Wright [115].
The ChainedMatrixMult algorithm in Figure 12.2 is due to Godbole
[54]. Floyd’s algorithm (Figure 12.3) is due to Floyd [38], but is based on
a theorem due to Warshall [110] for computing the transitive closure of a
boolean matrix. Because a boolean matrix can be viewed as an adjacency
matrix for a directed graph, this is the same as finding the transitive closure
of a directed graph (Exercise 12.13).
The algorithm suggested by Exercise 12.3 is due to Kozen and Zaks
[81]. Exercise 12.10 is solved by Chvatal, Klarner, and Knuth [22]. Wagner
and Fischer [109] solved Exercise 12.16 and provided an alternative solution
to Exercise 12.10. Exercise 12.17 is solved by Gilbert and Moore [53] and
Knuth [79], but a more elegant solution is given by Yao [117]. Exercises
12.19 and 12.20 are from Brassard and Bratley [18].
Part IV

Common Reduction Targets

413
Chapter 13

Depth-First Search

As we have seen in earlier chapters, many problems can be characterized as

graph problems. Graph problems often require the searching of the graph
for certain structural characteristics. For example, one problem which we
will examine in this chapter is searching a connected undirected graph for
vertices whose removal would disconnect the graph. Such vertices are called
articulation points.
In order to find articulation points, an algorithm must extract a great
deal of information involving the paths connecting the various vertices in the
graph. One way of organizing this information is by constructing a certain
kind of rooted spanning tree, called a depth-first spanning tree. One advan-
tage to processing a rooted spanning tree as opposed to a graph is that a
rooted tree fits more naturally with the top-down approach. Furthermore,
as we will see later in this chapter, a depth-first spanning tree has proper-
ties that are helpful for extracting connectivity information. We therefore
find that many problems can be reduced to finding one or more depth-first
spanning trees. These spanning trees are found using a technique called
depth-first search.
Because algorithms using depth-first search operate on rooted trees, we
begin by studying the problem of determining ancestry in rooted trees. The
technique we use to solve this problem will motivate the depth-first search
technique. We will then show how depth-first search can be used to find
articulation points in a connected undirected graph. Finally, we will show
how the technique can be extended to problems for directed graphs.

414
CHAPTER 13. DEPTH-FIRST SEARCH 415

13.1 Ancestry in Rooted Trees

Suppose we are given two nodes, x and y, in a rooted tree T . We wish to
determine whether x is an ancestor of y. By traversing the subtree rooted
at x (see Section 6.1), we can determine whether y is in that subtree. In the
worst case, y is not in the subtree, so we have to traverse the entire subtree.
If the subtree contains most of the nodes in T , the traversal will take Θ(n)
time, where n is the number of nodes in the tree.
It seems unlikely that we would be able to solve this problem in o(n)
time. However, note that we can traverse the entire tree in Θ(n) time. This
fact might help us to solve efficiently the problem of determining ancestry
for several pairs of nodes. Perhaps we can do a single traversal in Θ(n) time,
and save enough information that only constant additional time is needed
to decide ancestry for any pair of nodes in the tree.
We can get some insight into how to accumulate the necessary informa-
tion by reviewing preorder and postorder traversals, as outlined in Section
6.1. A preorder traversal visits a node x before visiting any of its proper
descendants, whereas a postorder traversal visits x after visiting all of its
proper descendants. Therefore, if x is a proper ancestor of y, a preorder
traversal will visit x before visiting y, and a postorder traversal will visit x
after visiting y. If we could combine a preorder traversal with a postorder
traversal and keep track of the order in which the visits were made, we could
then efficiently check a necessary condition for x being a proper ancestor of
y.
We will show that the above condition is also sufficient for x being a
proper ancestor of y. First, however, let us present an algorithm for calcu-
lating this information in order to be able to reason about it more precisely.
In order to describe the algorithm, we need a simple data structure called
a VisitCounter, whose definition is shown in Figure 13.1. It has two
representation variables, an integer count and an array num[0..n − 1]. Its
structural invariant is that for 0 ≤ i < n, num[i] is a natural number, and
that
count = max num[i].
0≤i<n

We interpret the size of the structure to be the size of num, and we interpret
the value of num[i] as the value associated with i. Clearly, the constructor
runs in Θ(n) time, and the operations all run in Θ(1) time.
The algorithm shown in Figure 13.2 combines the preorder and postorder
traversals of the tree T . We use a (directed) Graph to represent T . pre is
a VisitCounter that records the order in which nodes are visited in the
CHAPTER 13. DEPTH-FIRST SEARCH 416

Figure 13.1 The data structure VisitCounter

Structural Invariant: For 0 ≤ i < n, num[i] is a Nat, and

count = max num[i].

0≤i<n

Precondition: n is a Nat.
Postcondition: Constructs a VisitCounter of size n, all of whose
values are 0.
VisitCounter(n)
count ← 0; num ← new Array[0..n − 1]
for i ← 0 to n − 1
num[i] ← 0

Precondition: i is a Nat strictly less than Size().

Postcondition: Associates with i a value of m + 1, where m is the largest
value initially associated with any j, 0 ≤ j < Size().
VisitCounter.Visit(i)
count ← count + 1; num[i] ← count

Precondition: i is a Nat strictly less than Size().

Postcondition: Returns the value associated with i.
VisitCounter.Num(i)
return num[i]

Precondition: true.
Postcondition: Returns the size of this VisitCounter.
VisitCounter.Size()
return SizeOf(num)
CHAPTER 13. DEPTH-FIRST SEARCH 417

Figure 13.2 Preprocessing algorithm for ancestry test

Precondition: T is a Graph representing a rooted tree with edges directed

from parents to children. pre and post are VisitCounters whose size n is
the number of nodes in T , and i is a Nat strictly less than n.
Postcondition: Let S be the set of descendants of i. For every j ∈ S
and every node k 6∈ S, pre.Num(j) > pre.Num(k) and post.Num(j) >
post.Num(k). For any j, k ∈ S, j is a proper ancestor of k iff pre.Num(j) <
pre.Num(k) and post.Num(j) > post.Num(k). If node k 6∈ S, then
pre.Num(k) and post.Num(k) are unchanged.
PrePostTraverse(T, i, pre, post)
pre.Visit(i); L ← T.AllFrom(i)
// Invariant: The postcondition holds with S denoting the set of
// proper descendants of i that are not descendants of any node in L,
// except that pre.Num(i) has been changed to be larger than
// pre.Num(k) for every k 6∈ S ∪ {i}.
while not L.IsEmpty()
next ← L.Head().Dest(); L ← L.Tail()
PrePostTraverse(T, next, pre, post)
post.Visit(i)

preorder traversal, and post is a VisitCounter that records the order in

which nodes are visited in the postorder traversal.
Note that PrePostTraverse is different from most recursive algo-
rithms in that there is no explicit base case. However, a base case does exist
— the case in which i has no outgoing edges. In this case, the loop will
not iterate, and no recursive call will be made. The lack of an explicit base
case is reflected in the following correctness proof, which likewise does not
contain a separate base case.

Theorem 13.1 PrePostTraverse satisfies its specification.

Proof: By induction on n, the size of T .

Induction Hypothesis: Assume that PrePostTraverse satisfies its

specification for every tree with strictly fewer than n nodes.
CHAPTER 13. DEPTH-FIRST SEARCH 418

Induction Step: Assume the precondition is satisfied. We must show the

correctness of the invariant.

Initialization: The call to pre.Visit(i) makes pre.Num(i) larger than any

other values in pre. Otherwise, no other values in pre or post have been
changed from their initial values. At the beginning of the loop, L contains
the children of i and S = ∅. The invariant is therefore satisfied.

Maintenance: Suppose the invariant holds at the beginning of an iteration.

Clearly, the precondition holds for the recursive call. Because the subtree
rooted at next has strictly fewer nodes than does T , from the Induction
Hypothesis, the recursive call satisfies the postcondition with S denoting
the set of descendants of next. Let R be the set of descendants of next. Let
S ′ denote the value of S at the end of the iteration; i.e., S ′ = S ∪ R. We
must show that the invariant holds for S ′ at the end of the iteration.
Let us first determine the values in pre and post that have changed
from their initial values by the time the iteration completes. From the
invariant, only pre.Num(i), pre.Num(j), and post.Num(j) such that j ∈ S
have changed prior to the beginning of the iteration. From the Induction
Hypothesis, the recursive call only changes the values of pre.Num(j) and
post.Num(j) for j ∈ R. Thus, the only values to have changed from their
initial values are pre.Num(i), pre.Num(j), and post.Num(j) such that j ∈
S ′ . Furthermore because only values for j ∈ R are changed by the iteration,
it is still the case that pre.Num(i) > pre.Num(k) for all k 6∈ S ′ ∪ {i}.
Let j ∈ S ′ and k 6∈ S ′ . If j 6∈ R, then pre.Num(j), post.Num(j),
pre.Num(k) and post.Num(k) are unchanged by the iteration. Therefore,
because j ∈ S, from the invariant, it is still the case that pre.Num(j) >
pre.Num(k) and post.Num(j) > post.Num(k). On the other hand, sup-
pose j ∈ R. Because k 6∈ R, by the Induction Hypothesis, pre.Num(j) >
pre.Num(k) and post.Num(j) > post.Num(k) at the end of the iteration.
Now let j, k ∈ S ′ . We must show that j is a proper ancestor of k iff
pre.Num(j) < pre.Num(k) and post.Num(j) > post.Num(k).

⇒: Suppose j is a proper ancestor of k. Then it is either j and k are

both in R or neither is in R. If neither j nor k is in R, then the iter-
ation changes none of their pre or post values. Hence, from the invari-
ant, pre.Num(j) < pre.Num(k) and post.Num(j) > post.Num(k). On the
other hand, if j, k ∈ R then from the Induction Hypothesis, pre.Num(j) <
pre.Num(k) and post.Num(j) > post.Num(k).
CHAPTER 13. DEPTH-FIRST SEARCH 419

⇐: Suppose pre.Num(j) < pre.Num(k) and post.Num(j) > post.Num(k).

If j ∈ R, then from the Induction Hypothesis, k ∈ R, for if not, pre.Num(j) >
pre.Num(k). Thus, from the Induction Hypothesis, j is a proper ancestor
of k. If j 6∈ R, then from the Induction Hypothesis, k 6∈ R, for other-
wise, post.Num(k) > post.Num(j). Then none of their pre or post values is
changed by the iteration. From the invariant, j is a proper ancestor of k.

Termination: Because L contains finitely many elements and each iteration

removes one element from L, the loop must eventually terminate.

Correctness: Assume the invariant holds and that L is empty when the
loop terminates. We need to show that the postcondition holds when the
algorithm finishes. Let S denote the set of descendants of i.
Let us first consider which values in pre and post have been changed
by the algorithm. From the invariant, only pre.Num(i), pre.Num(j), and
post.Num(j), where j ∈ S \{i}, have changed by the time the loop ter-
minates. The final call to post.Visit(i) changes post.Num(i). Therefore,
the only values to have been changed by the algorithm are pre.Num(j) and
post.Num(j) such that j ∈ S.
Let j ∈ S and k 6∈ S. If j 6= i, then from the invariant, pre.Num(j) >
pre.Num(k) and post.Num(j) > post.Num(k). If j = i, then from the
invariant pre.Num(j) > pre.Num(k). Furthermore, the call to post.Visit(i)
makes post.Num(j) > post.Num(k).
Now let j, k ∈ S. We must show that j is a proper ancestor of k iff
pre.Num(j) < pre.Num(k), and post.Num(j) > post.Num(k).

⇒: Suppose j is a proper ancestor of k. Then k 6= i, because i is an

ancestor of every node in S. If j 6= i, then it follows from the invariant
that pre.Num(j) < pre.Num(k) and post.Num(j) > post.Num(k). If j = i
then from the invariant pre.Num(j) < pre.Num(k), and the final call to
post.Visit(i) makes post.Num(j) > post.Num(k).

⇐: Suppose pre.Num(j) < pre.Num(k) and post.Num(j) > post.Num(k).

If neither j = i nor k = i, then from the invariant, j is a proper ancestor of
k. If j = i, then clearly j is a proper ancestor of k. Finally, k 6= i because the
final call to post.Visit(i) would then make post.Num(k) > post.Num(j).

In order to analyze the running time of PrePostTraverse, let us

assume that T is represented as a ListGraph. Then the call to AllFrom
CHAPTER 13. DEPTH-FIRST SEARCH 420

Figure 13.3 Algorithm for testing ancestry for multiple pairs of nodes in a
rooted tree

Precondition: T refers to a Graph representing a tree rooted at i with

edges directed from parents to children, and x[1..m] and y[1..m] are arrays
of Nats less than T.Size().
Postcondition: Returns an array ancestor[1..m] of Bools such that
ancestor[i] is true iff x[i] is a proper ancestor of y[i].
Ancestors(T, i, x[1..m], y[1..m])
n ← T.Size(); ancestor ← new Array[1..m]
pre ← new VisitCounter(n); post ← new VisitCounter(n)
PrePostTraverse(T, i, pre, post)
for j ← 1 to m
condPre ← pre.Num(x[j]) < pre.Num(y[j])
condPost ← post.Num(x[j]) > post.Num(y[j])
ancestor[j] ← condPre and condPost
return ancestor

runs in Θ(1) time. Furthermore, the while loop iterates exactly m times,
where m is the number of children of i. Because each iteration of the while
loop results in one recursive call, it is easily seen that the running time is
proportional to the total number of calls to PrePostTraverse. It is easily
shown by induction on n, the number of nodes in the subtree rooted at i,
that a call in which the second parameter is i results in exactly n total calls.
The running time is therefore in Θ(n).
The algorithm for testing ancestry for multiple pairs of nodes is given
in Figure 13.3. The initialization prior to the call to PrePostTraverse
clearly runs in Θ(n) time, as does the call to PrePostTraverse. The
body of the loop runs in Θ(1) time. Because the loop iterates m times, the
entire algorithm runs in Θ(n + m) time.

13.2 Reachability in a Graph

We will now show how the technique used in the last section can be applied
to graph problems. Consider the problem of determining whether there is
a path from a given vertex i to a given vertex j in an undirected graph.
CHAPTER 13. DEPTH-FIRST SEARCH 421

Viewing the problem top-down, we first note that there is a path if i = j.

Otherwise, we can retrieve all of the vertices adjacent to i and remove i from
the graph. For each vertex k that was adjacent to i, we can then determine
whether there is a path from k to j in the resulting graph. We must be
careful, however, because each of these tests is destructive — it removes all
of the vertices it reaches. As a result, we must be sure a node k is still in the
graph before we solve that subproblem. Note that if it has been removed,
then it must have been reachable from one of the other vertices adjacent
to i; hence all nodes reachable from k have been removed. Thus, if j were
reachable from k, we would have already found that it was reachable from
i.
In order to avoid deleting vertices from the graph, we need a mechanism
for selecting a subgraph based on a given subset of the vertices. More
precisely, let G = (V, E) be a (directed or undirected) graph, and let V ′ ⊆ V .
We define the subgraph of G induced by V ′ to be G′ = (V ′ , E ′ ), where E ′ is
the set of edges connecting vertices from V ′ . We therefore need a mechanism
for selecting a subset of the vertices in the graph.
For this purpose, we define the data structure Selector. A Selector
represents a set of n elements numbered 0, . . . , n − 1, each of which is ei-
ther selected or unselected. The constructor and operations for Selector
are specified in Figure 13.4. It is a straightforward matter to implement
Selector using an array of booleans so that the constructor, SelectAll,
and UnselectAll run in Θ(n) time, where n is the number of elements
represented, and so that the remaining operations run in Θ(1) time.
We can now traverse the graph using almost the same algorithm as Pre-
PostTraverse — the only differences are that pre and post are not needed,
and we must check that a vertex has not already been visited before we tra-
verse it. We call this traversal a depth-first search (DFS). The entire algo-
rithm is shown in Figure 13.5. We retain pre and post in order to maintain
a close relationship between ReachDFS and PrePostTraverse.
Let G be an undirected Graph, and let i ∈ N such that i < G.Size().
Further let sel be a Selector of size G.Size() in which all elements are se-
lected, and let pre and post be VisitCounters of size G.Size() in which all
values are 0. Suppose we invoke ReachDFS(G, i, sel, pre, post). We define
a directed graph G′ as follows, based on the behavior of this invocation:

• G′ has the same vertices as G;

• G′ has the edge (j, k) iff a call ReachDFS(G, j, sel, pre, post) is made,
which in turn calls ReachDFS(G, k, sel, pre, post).
CHAPTER 13. DEPTH-FIRST SEARCH 422

Figure 13.4 Specification of the Selector data structure

Precondition: n is a Nat.
Postcondition: Constructs a Selector of size n, all of whose elements
are selected.
Selector(n)
Precondition: true.
Postcondition: Selects all elements.
Selector.SelectAll()
Precondition: true.
Postcondition: Unselects all elements.
Selector.UnselectAll()
Precondition: i is a Nat less than the number of elements.
Postcondition: Selects element i.
Selector.Select(i)
Precondition: i is a Nat less than the number of elements.
Postcondition: Unselects element i.
Selector.Unselect(i)
Precondition: i is a Nat less than the number of elements.
Postcondition: Returns true if element i is selected, or false otherwise.
Selector.IsSelected(i)

Let us consider the structure of G′ . We first observe that for each

vertex k 6= i, there is some edge (j, k) in G′ iff k is reachable from i in
G. Furthermore, a call to ReachDFS(G, k, sel, pre, post) can be made
only if sel.IsSelected(k) = true. Because this call immediately unse-
lects k, and because the algorithm never selects a vertex, it follows that
ReachDFS(G, k, sel, pre, post) can be called at most once. Hence, each
vertex in G′ has at most one incoming edge. Finally, i can have no incoming
edges. Therefore, G′ forms a tree rooted at i. The vertices of G′ are exactly
the vertices reachable from i in the subgraph of G induced by the selected
vertices. G′ is therefore a rooted spanning tree of the connected component
CHAPTER 13. DEPTH-FIRST SEARCH 423

Figure 13.5 Reachability algorithm for undirected graphs

Precondition: G refers to an undirected Graph, and i and j are Nats

strictly less than G.Size().
Postcondition: Returns true iff there is a path in G from i to j.
Reachable(G, i, j)
n ← G.Size(); sel ← new Selector(n)
pre ← new VisitCounter(n); post ← new VisitCounter(n)
ReachDFS(G, i, sel, pre, post)
return not sel.IsSelected(j)

Precondition: G refers to an undirected graph, i is a Nat such that

i < G.Size(), sel refers to a Selector of size G.Size(), pre and post refer
to VisitCounters of size G.Size(), and sel.IsSelected(i) = true.
Postcondition: Unselects each j such that j is reachable from i in G′ ,
where G′ denotes the subgraph of G induced by the set of selected vertices.
ReachDFS(G, i, sel, pre, post)
sel.Unselect(i); pre.Visit(i); L ← G.AllFrom(i)
while not L.IsEmpty()
next ← L.Head().Dest(); L ← L.Tail()
if sel.IsSelected(next)
ReachDFS(G, next, sel, pre, post)
post.Visit(i)

containing i in the subgraph of G induced by the selected vertices.

It should now be clear that the calls to ReachDFS(G, i, sel, pre, post)
and PrePostTraverse(G′ , i, pre, post) produce exactly the same values
in pre and post. In essence, ReachDFS performs both a preorder traversal
and a postorder traversal on a rooted tree. This rooted tree is a spanning
tree of a particular connected component of a given graph. The given graph
is the subgraph of the input graph G induced by the selected vertices, and
the connected component is specified by the input vertex i. The spanning
tree is not specified, but is implied by the behavior of ReachDFS. We
call this spanning tree the depth-first spanning tree generated by the call to
ReachDFS(G, i, sel, pre, post).
We can use the correspondence between ReachDFS and PrePostTra-
CHAPTER 13. DEPTH-FIRST SEARCH 424

verse in order to analyze the running time of ReachDFS. Suppose G is

implemented as a ListGraph. Let G′ be the subgraph of G induced by the
selected vertices, and let G′′ be the connected component of G′ containing i;
thus, the vertices in G′′ are the vertices visited by ReachDFS. Let n be the
number of vertices in G′′ . Certainly, ReachDFS runs in Ω(n) time. The
only difference in the two algorithms is that in ReachDFS, the loop may
iterate more times. Thus, if we ignore the iterations in which no recursive
call is made, the running time is the same as that of PrePostTraverse:
Θ(n).
In the call ReachDFS(G, j, sel, pre, post), the loop iterates m times,
where m is the number of vertices adjacent to j in G. The total number of
iterations in all recursive calls is therefore 2a1 + a2 , where a1 is the number
of edges in G′′ and a2 is the number of edges in G from vertices in G′′ to
vertices not in G′′ . The time for a single iteration that does not make a
recursive call is in Θ(1). Because G′′ is connected, a1 ≥ n − 1; hence, the
total running time of ReachDFS(G, i, sel, pre, post) is in Θ(a), where a is
the number of edges in G incident on vertices in G′′ .

13.3 A Generic Depth-First Search

Due to its hierarchical nature, a rooted tree is more amenable to the top-
down approach to algorithm design than is a graph. Furthermore, as we will
see shortly, a depth-first spanning tree has several additional properties that
can prove useful for designing graph algorithms. For this reason, it makes
sense to generalize ReachDFS to a general-purpose depth-first search al-
gorithm. With such an algorithm, we can then design our algorithms as
traversals of depth-first spanning trees.
In order to generalize this algorithm, we need an ADT for defining
various ways of processing a depth-first spanning tree. Upon examining
ReachDFS, we see that there are five places where processing might occur:

• Preorder processing of vertices can occur prior to the loop.

• Preorder processing of tree edges might occur prior to the recursive

call.

• Postorder processing of tree edges might occur following the recursive

call.

• Though the if statement in ReachDFS has no else-block, we might

include an else-block for processing other edges.
CHAPTER 13. DEPTH-FIRST SEARCH 425

• Postorder processing of vertices can occur following the loop.

We therefore have the ADT specified in Figure 13.6. The generic depth-first
search is shown in Figure 13.7.
Let us now consider the useful properties of depth-first spanning trees.
These properties concern the non-tree edges. First, we show the following
theorem regarding undirected graphs.

Theorem 13.2 Let G be a connected undirected graph with n vertices, and

let sel be a Selector of size n in which all elements are selected. Suppose
we call Dfs(G, i, sel, s), where s is a Searcher of size n. Then for every
edge {j, k} processed as a non-tree edge, either j is an ancestor of k or k is
an ancestor of j.

Proof: Without loss of generality, assume j is unselected before k is. Con-

sider the call to Dfs on vertex j. Initially, j is preorder processed while k
is still selected. We consider two cases.

Case 1: {j, k} is processed as a non-tree edge in the call to Dfs on j. Then

when this happens, k must be unselected. There must therefore have been
a call to Dfs on k which unselected k. This call resulted in k being both
preorder processed and postorder processed after j was preorder processed,
but before j was postorder processed. j is therefore a proper ancestor of k.

Case 2: {j, k} is processed as a tree edge in the call to Dfs on j, but is

processed as a non-tree edge in the call to Dfs on k. In this case, k is by
definition a child of j.

The above theorem gives the property of depth-first spanning trees that
makes depth-first search so useful for connected undirected graphs. Given
a connected undirected graph G and a depth-first spanning tree T of G, let
us refer to edges of G that correspond to edges in T as tree edges. We will
call all other edges back edges. By definition, tree edges connect parents
with children. Theorem 13.2 tells us that back edges connect ancestors with
descendants.
However, Theorem 13.2 does not apply to depth-first search on a directed
graph. To see why, consider the graph shown in Figure 13.8. The solid
edges in part (b) show a depth-first search tree for the graph in part (a);
the remaining edges of the graph are shown with dashed lines in part (b).
Because 0 is the root and all other vertices are reachable from 0, all other
CHAPTER 13. DEPTH-FIRST SEARCH 426

Figure 13.6 The Searcher ADT for facilitating depth-first search

Precondition: n is a Nat.
Postcondition: Constructs a new Searcher of size n.
Searcher(n)
Precondition: i is a Nat less than the size of this Searcher.
Postcondition: true.
Searcher.PreProc(i)
Precondition: i is a Nat less than the size of this Searcher.
Postcondition: true.
Searcher.PostProc(i)
Precondition: e is an Edge whose vertices are less than the size of this
Searcher.
Postcondition: true.
Searcher.TreePreProc(e)
Precondition: e is an Edge whose vertices are less than the size of this
Searcher.
Postcondition: true.
Searcher.TreePostProc(e)
Precondition: e is an Edge whose vertices are less than the size of this
Searcher.
Postcondition: true.
Searcher.OtherEdgeProc(e)
CHAPTER 13. DEPTH-FIRST SEARCH 427

Figure 13.7 A generic depth-first search algorithm

Precondition: G refers to a Graph, i is a Nat less than G.Size(), sel

refers to a Selector with size G.Size() such that i is selected, and s refers
to a Searcher with size G.Size().
Postcondition: Traverses a depth-first spanning tree rooted at i on
the connected component containing i in the subgraph of G induced by
the selected vertices. Each vertex j in this tree is processed by calling
s.PreProc(j) before any of j’s proper descendants are processed and by
calling s.PostProc(j) after all of j’s descendants are processed. Each
edge (j, k) in the tree is processed by calling s.TreePreProc((j, k)) be-
fore k is processed and by calling s.TreePostProc((j, k)) after k is pro-
cessed. All other edges e from j to any node in G are processed by calling
s.OtherEdgeProc(e).
Dfs(G, i, sel, s)
sel.Unselect(i); s.PreProc(i); L ← G.AllFrom(i)
while not L.IsEmpty()
edge ← L.Head(); next ← edge.Dest(); L ← L.Tail()
if sel.IsSelected(next)
s.TreePreProc(edge)
Dfs(G, next, sel, s)
s.TreePostProc(edge)
else
s.OtherEdgeProc(edge)
s.PostProc(i)

vertices are descendants of 0. Suppose (0, 1) is the first edge from 0 to be

processed. Because 2 is the only vertex reachable from 1, it is the only
proper descendant of 1. Of the remaining edges from 0, only (0, 3) leads to
a vertex that has not yet been reached, so 3 is the only other child of 0.
Finally, because 4 is reachable from 3, 3 is the parent of 4. In the resulting
depth-first spanning tree, 3 is neither an ancestor nor a descendant of 1, but
there is an edge (3, 1) in the graph.
The point at which the proof of Theorem 13.2 fails for directed graphs
is the initial assumption that j is unselected before k is. For an undirected
graph, one of the endpoints of the edge will be unselected first, and it doesn’t
CHAPTER 13. DEPTH-FIRST SEARCH 428

Figure 13.8 Example of depth-first search on a directed graph

(a) 0 (b) 0

1 3 1 3

2 4 2 4

matter which endpoint we call j. However, with a directed edge, either the
source or the destination may be unselected first, and we must consider
both cases. Given the assumption that the source is unselected first, the
remainder of the proof follows. We therefore have the following theorem.

Theorem 13.3 Let G be a directed graph with n vertices such that all
vertices are reachable from i, and let sel be a Selector of size n in which
all elements are selected. Suppose we call Dfs(G, i, sel, s), where s is a
Searcher of size n. Then for every edge (j, k) processed as a non-tree
edge, if j is unselected before k is, then j is an ancestor of k.
Thus, if we draw a depth-first spanning tree with subtrees listed from
left to right in the order we unselect them (as in Figure 13.8), there will be
no edges leading from left to right. As we can see from Figure 13.8, all three
remaining possibilities can occur, namely:

• edges from ancestors to descendants (we call these forward edges if

they are not in the tree);

• edges from descendants to ancestors (we call these back edges); and

• edges from right to left (we call these cross edges).

Theorem 13.3 gives us the property we need to make use of depth-first search
with directed graphs.
As a final observation, we note that back edges in directed graphs always
form cycles, because there is always a path along the tree edges from a vertex
to any of its descendants. Hence, a directed acyclic graph cannot have back
edges.
CHAPTER 13. DEPTH-FIRST SEARCH 429

In the next three sections, we will show how to use depth-first search to
design algorithms for connected undirected graphs, directed acyclic graphs,
and directed graphs.

13.4 Articulation Points

As we mentioned at the beginning of this chapter, an articulation point in
a connected undirected graph is any vertex whose removal yields a discon-
nected graph. If, for example, a given graph represents a communications
network, then an articulation point is a node whose failure would partition
the network. It would therefore be desirable to know if a given network
contains any articulation points.
Let G be a connected undirected graph, and let T be a depth-first span-
ning tree for G. We first note that it is easy to tell if the root of T is an
articulation point. If T has only one child, then the removal of its root from
G cannot disconnect G — the tree edges continue to connect the graph. On
the other hand, from Theorem 13.2, G can have no edges between differ-
ent subtrees of T . Thus, if T has more than one child, its root must be
an articulation point of G. We therefore conclude that the root of T is an
articulation point of G iff it has more than one child.
The above property suggests the following algorithm. Let n be the num-
ber of vertices and a be the number of edges in G. We do n separate
depth-first searches, using a different vertex as the root for each search.
As we process the tree edges (either in preorder or in postorder), we count
the number of children of the root. After the search completes, we then
determine whether the root is an articulation point by examining the num-
ber of children it has. It is not hard to see that we could construct such
an algorithm with running time in Θ(na), provided G is implemented as a
ListGraph.
In order to obtain a more efficient algorithm, let us consider some vertex
i other than the root of T . If there are no back edges in G, then the removal
of i disconnects G if i has at least one child. In this case, G is partitioned
into one connected component for each child of i, plus a connected com-
ponent containing all vertices that are not descendants of i. If, however,
there are back edges, these back edges may connect all of these partitions
together. Note, however, that from Theorem 13.2, no back edge can connect
partitions containing two different children of i. In particular, if a back edge
does connect a partition containing a child of i with another partition, it
leads from a proper descendant of i to a proper ancestor of i. We therefore
CHAPTER 13. DEPTH-FIRST SEARCH 430

conclude that i is an articulation point iff i has at least one child j in T such
that no descendant of j is adjacent to a proper ancestor of i.
If we can efficiently test the above property, then we should be able
to find all articulation points with a single depth-first search. Note that
it is sufficient to know, for each vertex j other than the root, the highest
ancestor k of j that is adjacent to some descendant of j. The parent i of
j is an articulation point if k = i. On the other hand, if for each child
j of i, the highest ancestor k adjacent to some descendant of j is a proper
ancestor of i, then i is not an articulation point. Because a vertex is preorder
processed before all of its descendants, we can determine which of a given
set of ancestors of a vertex is the closest to the root by determining which
was preorder processed first. Thus, let us use a VisitCounter pre to keep
track of the order the vertices are preorder processed. We then need to
compute the following value for each vertex i other than the root:

highest[i] = min{pre.Num(k) | k is adjacent to a descendant of i}. (13.1)

Let us consider a top-down approach to computing highest[i]. We first

observe that if j is a child of i, then highest[i] ≤ highest[j]. Furthermore,
the vertex k determining highest[i] is adjacent to either i or some proper
descendant of i. If k is adjacent to a proper descendant of i, then highest[i] =
highest[j] for some child j of i. Finally, because i is adjacent to its parent,
which has a smaller value in pre than does any child of i, we can ignore
the pre values of the children of i in computing highest[i]. We therefore
conclude that

highest[i] = min({pre.Num(k) | {i, k} is a back edge} ∪

{highest[j] | j is a child of i}). (13.2)

We can now build a Searcher s so that Dfs(G, 0, sel, s) will find the
articulation points of G, where sel is an appropriate Selector. (Note that
it doesn’t matter which node is used as the root of the depth-first search,
so we will arbitrarily use 0.) Let n be the number of vertices in G. We
need as representation variables a VisitCounter pre of size n, an array
highest[0..n − 1], a readable array artPoints[0..n − 1] of booleans to store the
results, and a natural number rootChildren to record the number of children
of the root. Note that making artPoints readable makes this data structure
insecure, because code that can read the reference to the array can change
values in the array. We will discuss this issue in more detail shortly.
To implement the Searcher operations, we only need to determine
when the various calculations need to be done. Initialization should go in the
CHAPTER 13. DEPTH-FIRST SEARCH 431

constructor; however, because the elements of the arrays are not needed until
the corresponding vertices are processed, we can initialize these elements in
PreProc. We want the processing of a vertex i to compute highest[i]. In
order to use recurrence (13.2), we need pre.Num(k) for each back edge {i, k}
and highest[j] for each child j of i. We therefore include the code to compute
highest[i] in OtherEdgeProc and TreePostProc. The determination
of whether a vertex i other than the root is an articulation point needs to
occur once we have computed highest[j] for each child j of i; hence, we
include this code in TreePostProc. To be able to determine whether the
root is an articulation point, we count its children in TreePostProc. We
can then make the determination once all of the processing is complete, i.e.,
in the call to PostProc for the root.
The implementation of ArtSearcher is shown in Figure 13.9. We have
not given an implementation of the TreePreProc operation — it does
nothing. We have also not specified any preconditions or postconditions
for the constructor or any of the operations. The reason for this is that
we are only interested in what happens when we use this structure with a
depth-first search. It therefore doesn’t make sense to prove its correctness
in every context. As a result, we don’t need to make this structure secure.
Furthermore, the code in each of the operations is so simple that specifying
preconditions and postconditions is more trouble than it is worth. As we
will see, it will be a straightforward matter to prove that the algorithm that
uses this structure is correct.
We can now construct an algorithm that uses depth-first search to find
the articulation points in a connected undirected Graph G. The algorithm
is shown in Figure 13.10. Let n be the number of vertices and a be the
number of edges in G, and suppose G is implemented as a ListGraph.
Because each of the operations in ArtSearcher runs in Θ(1) time, it is
easily seen that the call to Dfs runs in Θ(a) time, the same as ReachDFS.
The remaining statements run in Θ(n) time. Because G is connected, n ∈
O(a), so the entire algorithm runs in Θ(a) time.
To prove that ArtPts is correct, we need to show that the call to Dfs
results in s.artPoints containing the correct boolean values. In order to
prove this, it is helpful to prove first that s.highest contains the correct
values. This proof uses the fact that Dfs performs a traversal of a depth-
first spanning tree of G.

Lemma 13.4 Let G be a connected undirected Graph with n vertices. Let

sel be a Selector of size n in which all elements are selected, and let s be
a newly-constructed ArtSearcher of size n. Then Dfs(G, 0, sel, s) results
CHAPTER 13. DEPTH-FIRST SEARCH 432

Figure 13.9 ArtSearcher implementation of Searcher

ArtSearcher(n)
artPoints ← new Array[0..n − 1]; highest ← new Array[0..n − 1]
pre ← new VisitCounter(n); rootChildren ← 0

ArtSearcher.PreProc(i)
pre.Visit(i); artPoints[i] ← false; highest[i] ← ∞

ArtSearcher.TreePostProc(e)
i ← e.Source(); j ← e.Dest(); highest[i] ← Min(highest[i], highest[j])
if i = 0
rootChildren ← rootChildren + 1
else if highest[j] = pre.Num(i)
artPoints[i] ← true

ArtSearcher.OtherEdgeProc(e)
i ← e.Source(); k ← e.Dest()
highest[i] ← Min(highest[i], pre.Num(k))

ArtSearcher.PostProc(i)
if i = 0 and rootChildren > 1
artPoints[i] ← true

in s.highest[i] having the value specified in Equation (13.1) for 1 ≤ i < n.

Proof: Let 1 ≤ i < n. We first observe that s.highest[i] is only changed by

PreProc(i), TreePostProc(e), and OtherEdgeProc(e), where e is an
edge from i; i.e., s.highest[i] is only changed during the processing of vertex
i. We will show by induction on m, the number of descendants of i in the
depth-first spanning tree, that the processing of vertex i gives s.highest[i]
the correct value.

Induction Hypothesis: Assume that for any j with fewer than m descen-
CHAPTER 13. DEPTH-FIRST SEARCH 433

Figure 13.10 Algorithm for finding articulation points in a connected undi-

rected graph

Precondition: G refers to a connected undirected Graph.

Postcondition: Returns an array A[0..G.Size() − 1] of booleans such that
A[i] is true iff i is an articulation point in G.
ArtPts(G)
n ← G.Size(); s ← new ArtSearcher(n); sel ← new Selector(n)
Dfs(G, 0, sel, s)
return s.ArtPoints()

dants, the processing of vertex j gives s.highest[j] the correct value.

Induction Step: Suppose i has m descendants. It is easily seen that the

processing of vertex i assigns to s.highest[i] the value,

min({s.pre.Num(k) | {i, k} is a back edge} ∪

{s.highest[j] | j is a child of i}).

We must show that s.pre.Num(k) and s.highest[j] have the correct values
when they are used. s.pre.Num(k) is used in OtherEdgeProc(e), where
e = (i, k). This operation is only called when k is unselected, and hence after We are denoting
edges as directed
PreProc(k) has been called. s.pre.Num(k) has therefore been set to its edges because a
correct value. s.highest[j] is used in TreePostProc(e), where e = (i, j). Graph uses only
Hence, j is a child of i that has been processed. Because j is a child of i, it directed edges
(see p. 311).
has strictly fewer than m descendants. Thus, by the Induction Hypothesis,
its processing sets it to its correct value. Thus, the processing of vertex i
sets s.highest[i] to the value given in Equation (13.2), which we have shown
to be equivalent to Equation (13.1).

The proof of Lemma 13.4 relies heavily on known properties of depth-first

search and depth-first spanning trees. As a result, it is quite straightforward.
Often proofs of correctness for algorithms using depth-first search don’t even
require induction. Such is the case for the proof of the following theorem.

Theorem 13.5 ArtPts satisfies its specification.

CHAPTER 13. DEPTH-FIRST SEARCH 434

Proof: Let 0 ≤ i < n. We must show that the call to Dfs results in
s.artPoints[i] being true if i is an articulation point, or false otherwise. We
first note that artPoints[i] is changed only during the processing of i. Fur-
thermore, it is initialized to false in PreProc(i). We must therefore show
that it is set to true iff i is an articulation point. We consider two cases.

Case 1: i = 0. Then artPoints[i] is set to true iff rootChildren > 1 in the

call to PostProc(0). rootChildren is only changed during the processing
of vertex 0. It is initialized in PreProc(0) to 0 and incremented by 1 in
TreePostProc(e) for each tree edge (0, j). When PostProc(0) is called,
rootChildren therefore contains the number of children of vertex 0, which
is the root of the depth-first spanning tree. As we have observed earlier, the
root is an articulation point iff it has more than one child.

Case 2: i > 0. Then artPoints[i] is set to true iff s.highest[j] has a value
equal to s.pre.Num(i) in the call to TreePostProc(e), where e = (i, j)
for some vertex j. Because (i, j) is passed to TreePostProc, j must be
a child of i. From Lemma 13.4, the call to Dfs sets s.highest[j] to the
correct value, as defined in Equation (13.1). Furthermore, an examination
of the proof of Lemma 13.4 reveals that this value is set by the processing of
vertex j. This processing is done prior to the call to TreePostProc(e), so
that s.highest[j] has the correct value by the time it is used. Furthermore,
s.pre.Num(i) is set to its proper value by PreProc(i), which is also called
before TreePostProc(e). As we have already shown, i is an articulation
point iff s.highest[j] = s.pre.Num(i) for some child j of i.

13.5 Topological Sort Revisited

In Section 9.2, we gave an algorithm for finding a topological sort of a
directed acyclic graph. This algorithm ran in Θ(n+a) time for a ListGraph
with n vertices and a edges. In this section, we give an alternative algorithm
that illustrates the use of depth-first search on directed acyclic graphs.
We first note that a shortcoming of the Dfs algorithm is that it only
processes vertices that are reachable from the root. In an arbitrary graph,
there may be no vertex from which every vertex is reachable; hence it may
be impossible to find a depth-first spanning tree (or any spanning tree) for
the graph.
To remedy this shortcoming, we provide the algorithm DfsAll, shown
CHAPTER 13. DEPTH-FIRST SEARCH 435

Figure 13.11 Algorithm for processing an entire graph with depth-first

search

Precondition: G is a Graph and s is a Searcher with size G.Size().

Postcondition: Traverses a depth-first spanning forest on G. Each ver-
tex j in this forest is processed by calling s.PreProc(j) before any of
j’s proper descendants are processed and by calling s.PostProc(j) after
all of j’s descendants are processed. Each edge (j, k) in the forest is pro-
cessed by calling s.TreePreProc((j, k)) before k is processed and by call-
ing s.TreePostProc((j, k)) after k is processed. All other edges e from j
to any node in G are processed by calling s.OtherEdgeProc(e).
DfsAll(G, s)
n ← G.Size(); sel ← new Selector(n)
for i ← 0 to n − 1
if sel.IsSelected(i)
Dfs(G, i, sel, s)

in Figure 13.11. In order to understand its postcondition, it helps compare

its behavior with that of Dfs. In particular, suppose G has n vertices, and
let G′ be the graph obtained by adding a new vertex n to G and edges from n
to every other vertex in the graph. Notice that the behavior of the for loop in
DfsAll(G, s) is exactly the same as the behavior of Dfs(G′ , n, sel, s), except
that DfsAll only processes the vertices and edges in G. In particular, each
time Dfs is called by DfsAll, it traverses a subtree of G′ rooted at a
child of the root n. For this reason, we call the collection of trees traversed
by DfsAll a depth-first spanning forest. In particular, note that either
Theorem 13.2 or Theorem 13.3, depending on whether G is undirected or
directed, can be extended to apply to this forest.
Let G be a ListGraph with n vertices and a edges. A graph G′ con-
structed by adding a new vertex and n − 1 new edges to G then has n + a − 1
edges. Therefore, the running time of DfsAll(G, s) is easily seen to be in
Θ(n + a), provided each of the vertex and edge processing operations in s
runs in Θ(1) time.
Now consider the depth-first spanning forest for a directed acyclic graph.
Because there are no cycles, the spanning forest can have no back edges. This
leaves only tree edges, forward edges and cross edges. Furthermore, for each
CHAPTER 13. DEPTH-FIRST SEARCH 436

Figure 13.12 TopSortSearcher implementation of Searcher

TopSortSearcher(n)
order ← new Array[0..n − 1]; loc ← n

TopSortSearcher.PostProc(i)
loc ← loc − 1; order[loc] ← i

Figure 13.13 Topological sort algorithm using depth-first search

Precondition: G is a directed acyclic graph.

Postcondition: Returns an array listing the vertices of G in topological
order.
DfsTopSort(G)
n ← G.Size(); s ← new TopSortSearcher(n)
DfsAll(G, s)
return s.Order()

of these types of edge (i, j), j is postorder processed before i. This property
suggests a straightforward algorithm for topological sort, namely, to order
the vertices in the reverse of the order in which they are postorder processed
by a depth-first search.
The Searcher for this algorithm needs as representation variables a
readable array order[0..n − 1] for storing the listing of vertices in topological
order and a natural number loc for storing the location in order of the last
vertex to be inserted. Only the constructor and the PostProc operation
are nonempty; these are shown in Figure 13.12. The topological sort algo-
rithm is shown in Figure 13.13. If G is implemented as a ListGraph, the
algorithm’s running time is clearly in Θ(n + a), where n is the number of
vertices and a is the number of edges in G. We leave the proof of correctness
as an exercise.
CHAPTER 13. DEPTH-FIRST SEARCH 437

13.6 Strongly Connected Components

Let G = (V, E) be a directed graph. We say that G is strongly connected
if for each pair of vertices u and v, there is a path from u to v. For an
arbitrary directed graph G, let S ⊆ V . We say that S is a strongly connected
component of G if

• the subgraph of G induced by S is strongly connected; and

• for any subset S ′ ⊆ V , if S ⊆ S ′ and the subgraph of G induced by S ′

is strongly connected, then S ′ = S.

Thus, the strongly connected component containing a given vertex i is the

set of vertices j such that there are paths from i to j and from j to i.
It is easily seen that the strongly connected components of a directed
graph G = (V, E) partition the V into disjoint subsets. We wish to design
an algorithm to find this partition.
Let us consider a depth-first spanning forest of a directed graph G. We
will begin by trying to find one strongly connected component. We first
note that from any vertex i, we can reach all descendants of i. Depending
on which back edges exist, we may be able to reach some ancestors of i.
Depending on which cross edges exist, we may be able to reach some other
vertices as well; however, we cannot reach any vertices in a tree to the right
of the tree containing i (i.e., vertices in a tree that is processed later).
This suggests that we might focus on either the first or the last tree
processed. Consider the last tree processed. Let i be the root, and let j be
any vertex such that there is a path from j to i. Then j must be in the
same tree as i, and hence must be a descendant of i. There is therefore a
path from i to j. As a result, the strongly connected component containing
i is the set of vertices j such that there is a path from j to i. We generalize
this fact with the following theorem.

Theorem 13.6 Let G be a directed graph, and let F be a depth-first span-

ning forest of G. Let S be a strongly connected component of G, and let i
be the vertex in S that is postorder processed last. Let G′ be the subgraph
of G induced by set of vertices postorder processed no later than i. Then S
is the set of vertices j such that there is a path from j to i in G′ .

Proof: Clearly, for every vertex j ∈ S, there is a path from j to i that stays
entirely within S. Because i is postorder processed last of the vertices in S,
this path stays within G′ . Therefore, let j be a vertex such that there is a
CHAPTER 13. DEPTH-FIRST SEARCH 438

path from j to i in G′ . We will show that j ∈ S. Specifically, we will show

that j is a descendant of i, so that there is a path from i to j. The proof is
by induction on n, the length of the shortest path from j to i in G′ .

Base: n = 0. Then j = i, so that j is a descendant of i.

Induction Hypothesis: Let n > 0, and assume that for every m < n, if
there is a path of length m from k to i in G′ , then k is a descendant of i.

Induction Step: Suppose there is a path of length n from j to i in G′ .

Let (j, k) be the first edge in this path. Then there is a path of length n − 1
from k to i in G′ . From the Induction Hypothesis, k is a descendant of i.
We now have three cases.

Case 1: (j, k) is a back edge. Then j is clearly a descendant of i.

Case 2: (j, k) is either a forward edge or a tree edge. Then i and j are both Corrected
5/2/12.
ancestors of k. Because j is in G′ , it can be postorder processed no later
than i. Therefore, j cannot be a proper ancestor of i. j must therefore be a
descendant of i.

Case 3: (j, k) is a cross edge. Then k is postorder processed before j

is. Because j is postorder processed between k and i, and because k is a
descendant of i, j must also be a descendant of i.

The above theorem suggests the following approach to finding the con-
nected components of G. We first do a depth-first search on the entire graph
using a postorder VisitCounter post. We then select all of the vertices.
To see how we might find an arbitrary strongly-connected component, sup-
pose some of the components have been found and unselected. We find the
selected vertex i that has maximum post.Num(i). We then find all vertices
j from among the selected vertices such that there is a path from j to i
containing only selected vertices.
We have to be careful at this point because the set of selected vertices
may not be exactly the set of vertices that are postorder processed no later
than i. Specifically, there may be a vertex j that belongs to one of the
components that have already been found, but which is postorder processed
before i. However, Theorem 13.6 tells us that because j belongs to a different
component than i, there is no path from j to i. Therefore, eliminating such
nodes will not interfere with the correct identification of a strongly connected
CHAPTER 13. DEPTH-FIRST SEARCH 439

component. We conclude that the vertices that we find comprise the strongly
connected component containing i.
In order to be able to implement this algorithm, we need to be able to
find all vertices j from which i is reachable via selected vertices. This is
almost the same as the reachability problem covered in Section 13.2, except
that the edges are now directed, and we must follow the edges in the wrong
direction. It is not hard to see that we can use depth-first search to find all
vertices reachable from a given vertex i in a directed graph. In order to be
able to use this algorithm to find all vertices j from which i is reachable, we
must reverse the direction of the edges.
Because DfsAll processes all of the edges in the graph, we can use it
to build a new graph in which all of the edges have been reversed. In fact,
we can use the same depth-first search to record the order of the postorder
processing of the vertices. We use three representation variables:

• a readable ListMultigraph reverse (recall from Section 9.5 that if

we know we will not attempt to add parallel edges, it is more efficient
to add edges to a ListMultigraph and construct a ListGraph from
it);

• a readable array order[0..n − 1]; and

• a natural number loc.

As we process each edge, we add its reverse to reverse. As we postorder

process each vertex, we add it to order as we did for topological sort (Figure
13.12). The resulting RevSearcher is shown in Figure 13.14.
Once DfsAll is called with a RevSearcher, we need to perform a
second depth-first search on the entire reversed graph. The Searcher we
need for this search uses a readable array components[0..n − 1] in which
it will store values indicating the component to which a given vertex be-
longs, along with a natural number count to keep track of the number of
strongly connected components completely found. It also includes an op-
eration NextComp, used to indicate that a strongly connected component
has been found completely. Its implementation is shown in Figure 13.15.
Because the second depth-first search must start each tree at a particular
vertex, we need to modify DfsAll slightly. The resulting algorithm is shown
in Figure 13.16. We leave it as an exercise to show that this algorithm runs
in Θ(n + a) time, where n is the number of vertices and a is the number of
edges.
CHAPTER 13. DEPTH-FIRST SEARCH 440

Figure 13.14 RevSearcher implementation of Searcher

RevSearcher(n)
reverse ← new ListMultigraph(n)
order ← new Array[0..n − 1]; loc ← n

RevSearcher.TreePreProc(e)
reverse.Put(e.Dest(), e.Source(), e.Data())

RevSearcher.OtherEdgeProc(e)
reverse.Put(e.Dest(), e.Source(), e.Data())

RevSearcher.PostProc(i)
loc ← loc − 1; order[loc] ← i

13.7 Summary
Many graph problems can be reduced to depth-first search. In perform-
ing the reduction, we focus on a depth-first spanning tree or a depth-first
spanning forest. Because a rooted tree is more amenable to the top-down
approach than is a graph, algorithmic design is made easier. Furthermore,
depth-first spanning trees have structural properties that are often useful in
designing graph algorithms.
The implementation of a reduction to depth-first search consists mainly
of defining an implementation of the Searcher ADT. This data structure
defines what processing will occur at the various stages of the traversal of
the depth-first spanning tree. Proofs of correctness can then focus on the
traversal, utilizing induction as necessary.

13.8 Exercises
Exercise 13.1 Analyze the worst-case running time of the algorithm Pre-
PostTraverse, shown in Figure 13.2, assuming the tree T is implemented
as a MatrixGraph.
CHAPTER 13. DEPTH-FIRST SEARCH 441

Figure 13.15 SccSearcher implementation of Searcher

SccSearcher(n)
components ← new Array[0..n − 1]; count ← 0

SccSearcher.PreProc(i)
components[i] ← count

SccSearcher.NextComp()
count ← count + 1

Figure 13.16 An algorithm for finding strongly connected components in

a directed graph

Precondition: G refers to a directed Graph.

Postcondition: Returns an array C[0..n − 1], where n is the number of
vertices in G, such that C[i] = C[j] iff i and j belong to the same strongly
connected component.
StronglyConnComp(G)
n ← G.Size(); rs ← new RevSearcher(n)
DfsAll(G, rs)
order ← rs.Order(); G′ ← new ListGraph(rs.Reverse())
ss ← new SccSearcher(n); sel ← new Selector(n)
for i ← 0 to n − 1
if sel.IsSelected(order[i])
Dfs(G′ , order[i], sel, ss); ss.NextComp()
return ss.Components()
CHAPTER 13. DEPTH-FIRST SEARCH 442

Exercise 13.2 Prove that DfsTopSort, shown in Figures 13.12 and 13.13,
meets its specification.

Exercise 13.3 Show that StronglyConnComp, shown in Figures 13.14,

13.15, and 13.16, runs in Θ(n + a) time, where n is the number of vertices
and a is the number of edges in the given graph, assuming the graph is
implemented as a ListGraph.

Exercise 13.4 Prove that StronglyConnComp, shown in Figures 13.14,

13.15, and 13.16, meets its specification.

Exercise 13.5 Give an algorithm that decides whether a given directed

graph G contains a cycle. Your algorithm should return a boolean value
that is true iff G has a cycle. Assuming G is implemented as a ListGraph,
your algorithm should run in O(n+a) time, where n is the number of vertices
and a is the number of edges in G.

Exercise 13.6 A bridge in a connected undirected graph is an edge whose

removal disconnects the graph. Give an algorithm that returns a ConsList
containing all bridges of a given connected undirected graph. Your algorithm
should run in O(a) time in the worst case, where a is the number of edges
in the graph, assuming the graph is implemented as a ListGraph.

Exercise 13.7 A connected undirected graph is said to be biconnected if it

is impossible to disconnect the graph by removing a single vertex; i.e., it is
biconnected iff it has no articulation points. A biconnected component of a
connected undirected graph G is a maximal biconnected subgraph G′ of G
(by “maximal”, we mean that there is no biconnected subgraph of G that
contains all of G′ plus other vertices and/or edges).

a. Prove that each edge in a connected undirected graph G belongs to

exactly one biconnected component of G.

* b. Give an algorithm to identify the biconnected components of a given

connected undirected graph G. Specifically, your algorithm should
set the data field of each Edge in G to a natural number so that
e.data = f.data iff e and f belong to the same biconnected component.
Your algorithm should run in O(a) time in the worst case, where a is
the number of edges in the graph, assuming the graph is implemented
as a ListGraph.
CHAPTER 13. DEPTH-FIRST SEARCH 443

* Exercise 13.8 A directed graph is semiconnected if for each pair of ver-

tices i and j, there is either a path from i to j or a path from j to i. Give
an algorithm to decide whether a given directed graph G is semiconnected.
Your algorithm should return a boolean that is true iff G is semiconnected.
Your algorithm should run in O(n + a) time, where n is the number of
vertices and a is the number of edges in G.

* Exercise 13.9 An arborescence of a directed graph G = (V, E) is a sub-

set E ′ ⊆ E such that (V, E ′ ) is a rooted tree with edges directed from parents
to children. Give an algorithm to determine whether a given directed graph
G contains an arborescence, and if so, returns one. If G contains an ar-
borescence, your algorithm should return an array parent[0..n − 1] such that
parent[i] gives the parent of vertex i for all vertices other than the root, and
such that parent[i] = −1 if i is the root of the arborescence. If G does not
contain an arborescence, your algorithm should return nil. You algorithm
should operate in O(n + a) time, where n is the number of vertices and a is
the number of edges in G.

* Exercise 13.10 Give an algorithm that takes a connected undirected

graph G = (V, E) as input and produces as output a strongly connected
directed graph G′ = (V, E ′ ) such that

• if {i, j} ∈ E, then exactly one of (i, j) and (j, i) is in E ′ ; and

• if {i, j} 6∈ E, then neither (i, j) nor (j, i) is in E ′ .

Thus, G′ is obtained from G by assigning a direction to each edge of G. If

no such G′ exists, your algorithm should return nil. Your algorithm should
run in O(a) time, where a is the number of edges in the graph, assuming G
is implemented as a ListGraph.

* Exercise 13.11 A directed graph is singly connected if for each pair of

vertices i and j, there is at most one simple path from i to j. Give an
efficient algorithm to determine whether a given directed graph G is singly
connected. Your algorithm should return a boolean that is true iff G is singly
connected. Analyze the worst-case running time of your algorithm assuming
that G is implemented as a ListGraph.

* Exercise 13.12 A coloring of an undirected graph is an assignment of

labels to the vertices such that no two adjacent vertices have the same label.
A k-coloring is a coloring that uses no more than k distinct labels. Give
CHAPTER 13. DEPTH-FIRST SEARCH 444

an efficient algorithm to find a 3-coloring for a given connected undirected

graph G such that no vertex in G is adjacent to more than 3 vertices, and
at least one vertex is adjacent to strictly fewer than 3 vertices (a 3-coloring
always exists for such a graph). Your algorithm should run in O(n) time,
where n is the number of vertices in G, assuming G is implemented as a
ListGraph.

Exercise 13.13 An undirected graph is said to be bipartite if its vertices

can be partitioned into two disjoint sets such that no two vertices belonging
to the same partition are adjacent. (Note that such a partitioning is a 2-
coloring, as defined in Exercise 13.12.) Give an efficient algorithm to find
such a partitioning if one exists. Your algorithm should run in O(n + a)
time, where n is the number of vertices and a is the number of edges in the
graph.

13.9 Chapter Notes

The depth-first search technique was developed in the nineteenth century
by Trémaux, as reported by Lucas [87]. Its properties were studied by
Tarjan [102], who presented an algorithm he credits to Hopcroft for finding
articulation points and biconnected components (Exercise 13.7); see also
Hopcroft and Tarjan [62]. The algorithm given in Section 13.6 for finding
strongly connected components is due to Sharir [97].
Chapter 14

Network Flow and Matching

In this chapter, we examine the network flow problem, a graph problem to

which many problems can be reduced. In fact, some problems that don’t
even appear to be graph problems can be reduced to network flow, yielding
efficient algorithms. We begin by defining the problem.
Let Z denote the set of integers, and let Z>0 denote the set of positive
integers. A flow network is a 4-tuple (G, u, v, C), where

• G = (V, E) is a directed graph;

• u ∈ V is the source vertex ;

• v ∈ V is the sink vertex ; and

• C : E → Z>0 gives a positive integer capacity for each edge.

For example, Figure 14.1 shows a flow network whose source is 0, whose
sink is 5, and whose edges all have capacity 1. Intuitively, the capacities
represent the maximum flow that the associated edges can support. We are
interested in finding the maximum total flow from u to v that the network
can support.
The above definition is more general than what is typical. The standard
definition prohibits incoming edges to the source and outgoing edges from
the sink. However, this more general definition is useful for the development
of our algorithms.
In order to define formally a network flow, we need some additional
notation. For a vertex x in a directed graph, let

• x← = {(w, x) ∈ E}, the set of incoming edges to x; and

445
CHAPTER 14. NETWORK FLOW AND MATCHING 446

Figure 14.1 A flow network

sink 5
1 1

3 4
1
1 1

1 2

1 1
source 0

• x→ = {(x, y) ∈ E}, the set of outgoing edges from x.

A flow for a flow network ((V, E), u, v, C) is function F : E → N such that

• for each e ∈ E, F (e) ≤ C(e); and

• for each vertex x ∈ V \{u, v},

X X
F (e) = F (e).
e∈x← e∈x→

Thus, the flow on each edge is no more than that edge’s capacity, and the
total flow into a vertex other than the source or the sink is the same as the
total flow out of that vertex. An example of a flow on the network shown
in Figure 14.1 would have a flow of 1 on every edge except (4, 1); this edge
would have a flow of 0.
We leave it as an exercise to show that for any flow F of a flow network
(G, u, v, C),
X X X X
F (e) − F (e) = F (e) − F (e). (14.1)
e∈u→ e∈u← e∈v ← e∈v →

We therefore define the value of a flow to be the above difference — the

net flow out of the source, or equivalently, the net flow into the sink. Thus,
CHAPTER 14. NETWORK FLOW AND MATCHING 447

the flow described above for the network in Figure 14.1 has a value of 2.
Given a flow network, the network flow problem is to find a network flow
with maximum value. Clearly, 2 is the maximum value of any flow for the
network in Figure 14.1.
In the next two sections, we will examine algorithms for the network
flow problem. In the remainder of the chapter, we will consider the bipartite
matching problem, and show how to reduce it to network flow.

14.1 The Ford-Fulkerson Algorithm

A flow for a given network can be found by finding a simple path (i.e, one in
which no vertices are repeated) from the source to the sink. We will refer to
such a path as an augmenting path. If no augmenting path exists, then the
maximum flow must be 0. Otherwise, suppose m is the minimum capacity
of any edge in some particular augmenting path. We can clearly place a flow
of m on each edge in that path. We can then solve the smaller network flow
problem obtained by reducing the capacity of each edge on the augmenting
path by m, and removing any edge whose capacity would become 0. If we
measure the size of a problem instance as the value of a maximum flow, then
the resulting problem instance is clearly smaller. We can then combine the
flow obtained from the solution to the smaller problem with the first flow
we obtained.
The above approach clearly finds a flow for a given network. Unfor-
tunately, that flow is not guaranteed to be a maximum flow. To see why,
consider the flow network in Figure 14.1. Suppose the first augmenting path
found is h0, 2, 4, 1, 3, 5i. We can put a flow of 1 on each of these edges. In
the smaller instance, each of these edges would be removed, so that there is
no augmenting path in the resulting network. The flow found on the smaller
instance is therefore empty, so that the final flow has value 1. As we have
already seen, a maximum flow for this network has value 2.
The most obvious approach to repairing this algorithm is to be more
careful in how we choose the augmenting path. However, it turns out that
a more straightforward approach is to be more careful in how we construct
the smaller problem instance. The problem with the reduction described
above is that once we decide to place a flow on an edge, we cannot reverse
that decision. If we are more careful, we can construct a smaller instance
that allows us to reverse these decisions.
Specifically, when we decrease the capacity of an edge by m, we also
increase by m the capacity of the edge going the opposite direction. If there
CHAPTER 14. NETWORK FLOW AND MATCHING 448

is no such edge, we add it to the graph. When we combine the two flows,
we allow flows in opposite directions to cancel each other; i.e., if edge (x, y)
has flow k and edge (y, x) has flow k ′ ≤ k, we set the flow on (x, y) to k − k ′
and the flow on (y, x) to 0. Note that because any edge added to the graph
by the reduction will have a capacity of m, and the initial flow will be m
in the opposite direction, the combination of the two flows will result in no
flow on any edge that was added to the graph. We can therefore remove
these edges from the resulting flow.
Let us now formalize the construction outlined above. Let (G, u, v, C)
be a flow network, and let P be the set of edges in some augmenting path.
Let m be the minimum capacity of any edge in P . We define the residual
network of (G, u, v, C) with respect to P to be the flow network (G′ , u, v, C ′ ),
where G′ and C ′ are defined as follows:
• G′ is constructed from G by removing any edges in P with capacity
m and by adding edges (y, x) such that (x, y) ∈ P and (y, x) is not an
edge in G.

• C ′ ((x, y)) is defined as follows for each edge (x, y) ∈ G′ :

– If (x, y) ∈ P , C ′ ((x, y)) = C((x, y)) − m.

– If (y, x) ∈ P and (x, y) is an edge in G, C ′ ((x, y)) = C((x, y))+m.
– If (y, x) ∈ P and (x, y) is not an edge in G, C ′ ((x, y)) = m.
– Otherwise, C ′ ((x, y)) = C((x, y)).

Thus, Figure 14.2(a) shows the residual network for the flow network in
Figure 14.1 with respect to the augmenting path h0, 2, 4, 1, 3, 5i. This graph
has an augmenting path: h0, 1, 4, 5i. The residual network with respect to
this augmenting path is shown in Figure 14.2(b). There is no augmenting
path in this graph. If we combine the flows obtained by assigning a flow of
1 to each edge in the respective paths, the flows on the edges (4, 1) in the
original graph and (1, 4) in the graph in Figure 14.2(a) cancel each other
out. The resulting flow therefore has a flow of 1 on each edge except (4, 1)
in the original network. This flow has a value of 2, which is maximum.
We now need to prove the correctness of this reduction. We begin by
showing the following lemma.

Lemma 14.1 Let (G, u, v, C) be a flow network, and let P be the set of
edges on some augmenting path. Let F1 be the flow obtained by adding a
flow of m to each edge in P , where m is the minimum capacity of any edge
in P . Let F2 be a maximum flow on the residual graph of (G, u, v, C) with
CHAPTER 14. NETWORK FLOW AND MATCHING 449

Figure 14.2 Examples of residual graphs

(a) (b)
sink 5 sink 5
1 1 1 1

3 4 3 4
1 1
1 1 1 1

1 2 1 2

1 1 1 1
source 0 source 0

respect to P , and suppose F2 has value k. Then the combination of F1 and

F2 is a flow with value k + m on (G, u, v, C).

Proof: We must first show that the combination of the two flows does not
give a flow where there is no edge in G. This can only happen if there is a
flow in F2 on an edge (x, y) that is not in G. Then (y, x) must be an edge
in P . The capacity of (x, y) in the residual graph is therefore m. Because
the flow on (y, x) in F1 is m, the combination of F1 and F2 cannot give a
positive flow on (x, y).
We will now show that in the combination of F1 with F2 , the flow on
each edge (x, y) is no more than C((x, y)). The only way this can happen
is if there is positive flow on (x, y) in F2 . We consider three cases.

Case 1: (x, y) ∈ P . Then C ′ ((x, y)) = C((x, y)) − m. The sum of the two
flows on (x, y) is therefore at most C((x, y)).

Case 2: (y, x) ∈ P . Then C ′ ((x, y)) = C((x, y)) + m. In the combination

of F1 with F2 , the flow on (x, y) is its flow in F2 , minus m. This total flow
can be no more than C((x, y)).

Case 3: (x, y) 6∈ P and (y, x) 6∈ P . Then C ′ ((x, y)) = C((x, y)). In the
CHAPTER 14. NETWORK FLOW AND MATCHING 450

combination of F1 with F2 , the flow on (x, y) is simply its flow in F2 , which

can be no more than C((x, y)).
Finally, it is clear that for each vertex w in G other than u and v, the
total flow into w must equal the total flow out of w, and that the total flow
out of u is k + m.

Using the above Lemma, we can prove the theorem below. Combined
with Lemma 14.1, this theorem ensures that the reduction yields a maximum
flow for the given network.

Theorem 14.2 Let (G, u, v, C) be a flow network with maximum flow k,

and let P be the set of edges in some augmenting path. Let m be the
minimum capacity of any edge in P , and let (G′ , u, v, C ′ ) be the residual
network of (G, u, v, C) with respect to P . Then the maximum flow for
(G′ , u, v, C ′ ) has value k − m.

Proof: Let F1 be a maximum flow for (G, u, v, C). Then F1 has value k.
To simplify our discussion, let us interpret F1 as a function F1 : V × V → Z,
where V is the set of vertices in G, such that F1 (x, y) gives the flow over
(x, y) minus the flow over (y, x); if either of these edges does not exist, we
use 0 for this edge’s flow.
Let us assign flows to the edges of G′ as follows:

1. If (x, y) ∈ P and F1 (x, y) ≥ m, assign a flow of F1 (x, y) − m to (x, y).

Because C ′ ((x, y)) = C((x, y)) − m, the flow on this edge does not
exceed the capacity.

2. If (x, y) ∈ P and F1 (x, y) < m, assign a flow of m − F1 (x, y) to (y, x).

Because C ′ ((y, x)) = C((y, x)) + m if (y, x) ∈ G, or C ′ ((y, x)) = m if
(y, x) 6∈ G, the flow on this edge does not exceed the capacity.

3. If (x, y) 6∈ P , (y, x) 6∈ P , and F1 (x, y) > 0, assign a flow of F1 (x, y) to

(x, y). Because C ′ ((x, y)) = C((x, y)), the flow on this edge does not
exceed its capacity.

4. Assign a flow of 0 to all other edges.

Because the above construction essentially reduces the flow F1 by m

along P , the sum of incoming flows equals the sum of outgoing flows for
all vertices in G′ except u and v. This assignment is therefore a network
flow, which we will denote F2 . Furthermore, P contains exactly one edge
incident on u — the first edge, which we will call (u, w). The effect of Steps
CHAPTER 14. NETWORK FLOW AND MATCHING 451

1 and 2 therefore decreases the net flow from u by m. The value of F2 is

therefore k − m. From Lemma 14.1, the combination of F2 with a flow of
m along P gives a flow of k for (G, u, v, C). Because this is true for any
flow on (G′ , u, v, C ′ ), and because k is the maximum flow for (G, u, v, C), we
conclude that F2 must be a maximum flow for (G′ , u, v, C ′ ).

Though this reduction is not a transformation, we can implement it using

a loop by maintaining a graph in which the flows on augmenting paths are
combined as they are found. The resulting algorithm, known as the Ford-
Fulkerson algorithm, is shown in Figure 14.3. Because the Graph ADT
(see Figure 9.3 on page 312) provides no operation for removing an edge,
we will allow the residual graph to have edges with capacity 0. The proof
of correctness is easily shown using Theorem 14.2; the details are left as an
exercise.
Four auxiliary functions are specified in Figure 14.3. We leave it as
exercises to show that if G is represented as a ListGraph, then

• CopyGraph can be implemented to return a ListGraph and to run

in O(n+a) time, where n is the number of vertices and a is the number
of edges in G; and

• AddFlow can be implemented to run in O(n + a) time, where n is

the number of vertices in F (or equivalently R) and a is the number
of edges in F and R together, assuming F and R are implemented as
ListGraphs.

Furthermore, MinVal can clearly be implemented to run in O(n) time,

where n is the number of Edges in L, and FindPath can be implemented
to run in O(n+a) time using either depth-first search or breadth-first search
(see Exercise 11.6), where n is the number of vertices and a is the number
of edges in its first argument G.
Note that FindPath does not specify which augmenting path will be
chosen. As a result, the Ford-Fulkerson algorithm can perform very poorly.
Consider, for example, the flow network shown in Figure 14.4(a), where k is
some large integer. It is easily seen by inspection that the maximum flow is
2k. Suppose the algorithm first selects the augmenting path h0, 1, 2, 3i. The
minimum capacity on this path is 1, and the resulting residual graph is shown
in Figure 14.4(b). Suppose the algorithm then chooses the augmenting path
h0, 2, 1, 3i. The minimum capacity is again 1, and the resulting residual
graph is shown in Figure 14.4(c). It is easily seen that this process can
continue increasing the flow by 1 until the maximum flow of 2k is achieved.
CHAPTER 14. NETWORK FLOW AND MATCHING 452

Figure 14.3 The Ford-Fulkerson algorithm for network flow

Precondition: G is directed Graph in which each edge contains a Nat

giving its capacity, and source and sink are distinct Nats less than the
number of vertices in G.
Postcondition: Returns a directed graph F in which the contents of the
edges give a maximum flow for G from source to sink.
NetworkFlow(G, source, sink)
F ← CopyGraph(G, true); R ← CopyGraph(G, false)
P ← FindPath(R, source, sink)
// Invariant: F combined with a maximum flow for R gives a maximum
// flow for G. P is a ConsList containing the Edges of a path from
// source to sink in R if there is such a path; otherwise, P = nil.
while P 6= nil
m ← MinVal(P ); AddFlow(F, P, R, m)
P ← FindPath(R, source, sink)
return F

Precondition: G is a Graph, and zeroEdges is a boolean.

Postcondition: Returns a copy of G. If zeroEdges is true, the contents of
all edges are set to 0; otherwise, they are unchanged.
CopyGraph(G, zeroEdges)
Precondition: G is a Graph whose edges contain natural numbers, and i
and j are distinct natural numbers strictly less than the number of vertices
in G.
Postcondition: Returns a ConsList P containing the Edges in a simple
path of non-zero Edges from i to j in G. If no such path exists, returns nil.
FindPath(G, i, j)
Precondition: L is a ConsList of Edges containing positive integers.
Postcondition: Returns the minimum integer stored on any Edge in L.
MinVal(L)
Precondition: F and R are directed Graphs having the same number
of vertices and whose edges contain natural numbers, P is a ConsList of
Edges forming a simple path in R, and m is a positive integer.
Postcondition: Adds a flow of m to each edge in F that appears in P and
sets R to the resulting residual graph. Edges are added to each Graph if
necessary.
AddFlow(F, P, R, m)
CHAPTER 14. NETWORK FLOW AND MATCHING 453

Figure 14.4 A flow network on which NetworkFlow can perform poorly

(a) (b) (c)

sink 3 3 3
k k k k−1 k−1 k−1
1 1 1
1 1 1
1 2 1 2 1 2
1 1 1
k k k−1 k k−1 k−1
source 0 0 0

On the other hand the algorithm could have achieved the same flow with
two augmenting paths: h0, 1, 3i and h0, 2, 3i.
In the next section, we will consider how to make good augmenting path
choices. For now, we note that in the worst case, the loop in NetworkFlow
can iterate M times, where M is the value of the maximum flow. Assuming
the initialization and the body of the loop are implemented to run in Θ(n+a)
time, where n is the number of vertices and a is the number of edges in G,
the algorithm runs in Θ(M (n + a)) time in the worst case. If we assume
that all vertices are reachable from the source, then a ≥ n − 1, and we can
simplify the running time to Θ(M a).
Before we move on to a discussion on finding augmenting paths, we note
one additional property of the Ford-Fulkerson algorithm. As long as each
augmenting path found is simple — and there is no reason a path-finding
algorithm would find a path containing a cycle — the path will not contain
any edges to the source or any edges from the sink. The obvious consequence
is that if the graph contains edges into the source or out of the sink, they
will not be used. A less obvious consequence is that after such edges are
introduced into the residual graph, they will not be used. Thus, once a flow
is added to an edge from the source or to the sink, the flow on that edge
will never be decreased.

14.2 The Edmonds-Karp Algorithm

We have seen that the way augmenting paths are chosen by the Ford-
Fulkerson algorithm can significantly impact its performance. In this sec-
CHAPTER 14. NETWORK FLOW AND MATCHING 454

tion, we consider how to select augmenting paths in order to avoid very

bad performance. One might suppose that we should try to select edges
with high capacity; however, the approach we take does not even consider
the edge capacities. Rather, we instead select a shortest augmenting path,
in terms of the number of edges. The resulting network flow algorithm is
known as the Edmonds-Karp algorithm.
Exercise 11.6 on page 389 outlined the breadth-first search technique for
finding a shortest path in Θ(n + a) time, where n is the number of vertices
and a is the number of edges. We will therefore focus on analyzing the
running time of the Edmonds-Karp algorithm. We first show the following
lemma.

Lemma 14.3 No iteration of the Edmonds-Karp algorithm can bring any

vertex nearer to the source.

Proof: By contradiction. Let R be the residual graph at the beginning of

an iteration, and let R′ be the residual graph at the end of the iteration.
Suppose some vertex x is closer to the source u in R′ than in R. Specifically,
let x be a vertex that is closest to u in R′ of all such vertices. There must
be some edge on the shortest path from u to x in R′ that is not in R, for
otherwise this path would also be a path from u to x in R. Specifically,
the last edge (w, x) on this path must have been added, for otherwise w,
which is closer than x to u in R′ , would also be closer to u in R′ than in R.
This means that (x, w) is on a shortest augmenting path in R, so that x is
closer than w to u in R. Therefore, w is closer to u in R′ than in R — a
contradiction.

Knowing that no vertex ever gets any closer to the source over the course
of the Edmonds-Karp algorithm, we will now prove a lemma that shows when
a vertex must get farther away from the source. If a vertex is farther than
n − 1 edges from the source, where n is the number of vertices, then it must
be unreachable. By Lemma 14.3, once a vertex becomes unreachable, it will
never become reachable. This next lemma will therefore enable us to bound
the number of iterations of the Edmonds-Karp algorithm.

Lemma 14.4 Suppose that when some vertex x is at a distance d from the
source u, the Edmonds-Karp algorithm removes an edge (x, y). Suppose that
later this edge is added again. Then after this edge is added, the distance
from u to x is at least d + 2.
CHAPTER 14. NETWORK FLOW AND MATCHING 455

Proof: If (x, y) is removed, then this edge must be on a shortest augmenting

path. Then before this edge is removed, the distance from u to y is d + 1.
When (x, y) is added again, (y, x) must be on a shortest augmenting path.
From Lemma 14.3, the distance from u to y is still at least d + 1 when this
path is found. The distance to x is therefore at least d + 2. From Lemma
14.3, the distance from u to x must still be at least d + 2 after the edge (x, y)
is added.

Theorem 14.5 In the worst case, the Edmonds-Karp algorithm iterates no

more than na times, where n is the number of vertices and a is the number
of edges in G.

Proof: Because a flow equal to the minimum edge capacity on a shortest

augmenting path is added by each iteration, each iteration removes at least
one edge. By Lemmas 14.3 and 14.4, no edge can be removed more than
n/2 times. When edges are added, they are always added in the opposite
direction of an existing edge; hence, at most 2a distinct edges ever appear
in the residual graph. The loop can therefore iterate at most na times.

If the initialization and the body of the loop are implemented to run in
Θ(n + a) time, we can conclude that the algorithm runs in O(na(n + a))
time in the worst case. Furthermore, the analysis of the last section still
applies, so that the running time is in O(min(M, na)(n + a)), where M is
the value of the maximum flow. If we assume that every vertex is reachable
from the source, we can simplify this to O(min(M a, na2 )).

14.3 Bipartite Matching

A matching in an undirected graph is a subset of the edges such that no
two edges are incident on the same vertex. In this section, we consider the
problem of finding a matching of maximum size in a given bipartite graph,
as defined in Exercise 13.13 on page 444. Specifically, we will show that this
problem can be reduced to the network flow problem.
As an example, consider the bipartite graph shown in Figure 14.5. We
claim that the heavier edges, namely, {0, 4}, {2, 5}, and {3, 7}, form a match-
ing of maximum size. Clearly, these edges form a matching because no two
of them share a common vertex. To see that it is of maximum size, we first
note that any larger matching must contain all of the vertices in {0, 1, 2, 3}
as endpoints. However, the only edges incident on 1 and 3 are {1, 7} and
CHAPTER 14. NETWORK FLOW AND MATCHING 456

Figure 14.5 A maximum-sized matching in a bipartite graph

4 5 6 7 8

0 1 2 3

{3, 7}, respectively, and they share a common vertex. Hence, any matching
must exclude either 1 or 3. Therefore, there is no matching of size larger
than 3.
We will now show how to reduce bipartite matching to network flow.
Given a bipartite graph G, we construct an instance of network flow as
follows. For simplicity, we will assume that the vertices of the bipartite
graph have already been partitioned into the sets V1 and V2 (see Exercise
13.13). We first direct all of the edges from V1 to V2 . We then add a new
source vertex u and edges from u to each vertex in V1 . Next, we add a new
sink vertex v and edges from each vertex in V2 to v. Finally, we assign a
capacity of 1 to each edge. See Figure 14.6 for the result of applying this
reduction to the graph in Figure 14.5.
Consider any matching in the given bipartite graph. We can construct
a flow in the constructed network by adding a flow of 1 to each edge in the
matching, as well as to each edge leading to a matched vertex in V1 and
to each edge leading from a matched vertex in V2 . Clearly, any unmatched
vertex from the bipartite graph will have a flow of 0 on all of its incoming
and outgoing edges. Furthermore, each matched vertex in V1 will have a
flow of 1 on its incoming edge and a flow of 1 on the single outgoing edge
in the matching. Likewise, each matched vertex in V2 will have a flow of 1
on the single incoming edge in the matching and a flow of 1 on its outgoing
edge. Thus, we have constructed a flow whose value is the number of edges
in the matching.
Conversely, consider any flow on the constructed network. Because any
vertex in V1 can have an incoming flow of at most 1, at most one of its
outgoing edges will contain a positive flow. Likewise, because any vertex in
V2 can have an outgoing flow of at most 1, at most one of its incoming edges
will contain a positive flow. The edges from V1 to V2 containing positive flow
CHAPTER 14. NETWORK FLOW AND MATCHING 457

Figure 14.6 The flow network constructed from the bipartite graph shown
in Figure 14.5

sink 10

4 5 6 7 8

0 1 2 3

source 9

All edge capacities are 1.

therefore correspond to a matching in the bipartite graph. Furthermore, the

value of the flow is the number of edges in the matching.
We conclude that the edges from V1 to V2 in a maximum flow for the
constructed network are the edges in a maximum-sized matching for the
given bipartite graph. The resulting flow network has n+2 vertices and n+a
edges, where n and a are the number of vertices and edges, respectively, in
the bipartite graph. The Edmonds-Karp algorithm will therefore solve the
constructed network flow instance in O(M (n + a)) time, where M is the
number of edges in the maximum-sized matching. Because M can be no
more than n/2, if we assume that each vertex is incident on at least one
edge, the running time is in O(na). Furthermore, it is not hard to construct
the flow network in O(n + a) time, so the bipartite matching problem can
be solved in O(na) time.
Rather than presenting the code for the reduction, let us first examine the
reduction more carefully to see if we can optimize the bipartite matching
algorithm. For example, the addition of new vertices and edges is only
needed to form a flow network. We could instead adapt one of the network
CHAPTER 14. NETWORK FLOW AND MATCHING 458

flow algorithms to operate without the source and/or the sink explicitly
represented.
We also note that as flow is added, the edges containing the flow —
which are the edges of a matching — have their direction reversed. Rather
than explicitly reversing the direction of the edges, we could keep track of
which edges have been included in the matching in some other way. For
example, we could use an array matching[0..n − 1] such that matching[i]
gives the vertex to which i is matched, or is −1 if i is unmatched. Because
a matching has at most one edge incident on any vertex, this may end up
being a more efficient way of keeping track of the vertices adjacent (in the
flow network) to vertices in V2 . The maximum-sized matching could also be
returned via this array.
As we observed at the end of Section 14.1, once flow is added to any
edge from the source or to any edge to the sink, that flow is never removed.
To put this in terms of the matching algorithm, once a vertex is matched, it
remains matched, although the vertex to which it is matched may change.
Furthermore, we claim that if we ever attempt to add a vertex w ∈ V1 to
the current matching M and are unable to do so (i.e., there is no path from
w to an unmatched vertex in V2 ), then we will never be able to add w to
the matching.
To see why this is true, notice that if there were a maximum-sized match-
ing containing all currently matched vertices and w, then there is a matching
M ′ containing no other vertices from V1 . If we delete all vertices from V1
that are unmatched in M ′ , then M ′ is clearly a maximum-sized matching
for the resulting graph. The Ford-Fulkerson algorithm must therefore be
able to find a path that yields M ′ from M .
As a result, we only need to do a single search from each vertex in V1 .
The following theorem summarizes this property.

Theorem 14.6 Let G be a bipartite graph, and let S be the set of vertices
in some matching on G. Suppose some maximum-sized matching includes
all of the vertices in S. Let i be some vertex not in S. Then there is a
maximum-sized matching including S ∪ {i} if there is a matching including
S ∪ {i}.
In order to implement the above optimizations, it is helpful to define
a data structure called MatchingGraph, which implements the Graph
ADT. Its purpose is to represent a particular directed graph G′ derived from
a given bipartite graph G and a matching on G. Suppose G has vertices
0, 1, . . . , n − 1. G′ then contains the vertices 0, 1, . . . , n. If {i, j} is an edge in
CHAPTER 14. NETWORK FLOW AND MATCHING 459

Figure 14.7 The MatchingGraph for the bipartite graph shown in Figure
14.5 with matching {{0, 5}, {3, 7}}

4
0
5
1
9 6
2
7
3
8

G and j is unmatched, then G′ will contain the edge (i, n). Every other edge
in G′ will represent two edges in G — an edge not in the matching followed
by an edge in the matching. Thus, for 0 ≤ i < n, 0 ≤ j < n, and i 6= j,
G′ contains the edge (i, j) iff there is a vertex k in G such that {k, j} is in
the matching and {i, k} is an edge in G. For example, Figure 14.7 shows
the MatchingGraph for the bipartite graph of Figure 14.5 with matching
{{0, 5}, {3, 7}}.
Suppose the two partitions of G are V1 and V2 . Then augmenting paths
in the flow network constructed by the reduction correspond to paths from
unmatched vertices in V1 to n in G′ . In order to find an augmenting path in
the flow network, we need to find a path to n in G′ from an unmatched vertex
in G. For example, consider the MatchingGraph shown in Figure 14.7.
We can add vertex 2 to the matching by finding a path from 2 to 9 in G′ .
Taking the path h2, 0, 9i could yield the augmenting path h2, 5, 0, 4i, which
produces the matching shown in Figure 14.5. This path could also yield the
augmenting path h2, 5, 0, 6i because 0 is adjacent to two unmatched vertices,
4 and 6. Alternatively, taking the path h2, 9i would yield the augmenting
path h2, 8i.
Note that G′ actually represents two flow networks. If (i, j) is an edge in
′
G and j 6= n, then i and j must both be in the same partition. Therefore,
the subgraph induced by V1 ∪{n} represents the flow network in which edges
CHAPTER 14. NETWORK FLOW AND MATCHING 460

in the matching lead from V2 to V1 . Symmetrically, the subgraph induced

by V2 ∪ {n} represents the flow network in which edges in the matching
lead from V1 to V2 — i.e., the flow network that would be constructed by
swapping V1 with V2 . Thus, 4 could be added to the matching by finding a
path from 4 to 9 in G′ . The only such path, h4, 5, 9i, would also yield the
matching shown in Figure 14.5.
To implement a MatchingGraph, we need two representation vari-
ables:
• a Graph bipartite representing the bipartite graph; and
• a readable array matching[0..n − 1] representing the matching, so that
matching[i] gives the vertex to which i is matched, or −1 if i is un-
matched.
Its structural invariant will be that for 0 ≤ i < n, if matching[i] 6= −1, then
matching[matching[i]] = i.
A partial implementation is shown in Figure 14.8 — we only include
implementations of those operations we will actually be using. These oper-
ations include an additional operation for adding an edge to the matching,
while removing any edges that might be incident on either endpoint. We
also include a constructor that constructs a MatchingGraph from a given
bipartite graph with an empty matching. We use the data variable of an
Edge to store the intermediate vertex between the two edges of the bipartite
graph represented by that Edge.
Note that this implementation is not secure, because its constructor
allows an outside reference to bipartite, and because matching is readable.
We could easily modify the implementation so that the constructor stores
a copy of its input graph and the Matching operation returns a copy of
matching; however, if we write our matching algorithm so that it doesn’t
change these items except via operations in MatchingGraph, we can avoid
this extra copying.
In order to complete the matching algorithm, we need to be able to
find a path from i to j in a directed graph. Because the maximum flow on
the constructed flow network is no more than n/2, where n is the number
of vertices in the bipartite graph, the Ford-Fulkerson algorithm will not
perform badly. Therefore, we will use depth-first search to find paths. (We
leave it as an exercise to implement the algorithm using breadth-first search.)
We therefore need a Searcher with the following representation variable:
• incoming[0..n − 1]: a readable array of Edges giving the incoming
edge in the depth-first spanning tree for each vertex reached.
CHAPTER 14. NETWORK FLOW AND MATCHING 461

Figure 14.8 MatchingGraph implementation of Graph (partial)

Structural Invariant: For 0 ≤ i < n, if matching[i] 6= −1, then

matching[matching[i]] = i.
Precondition: G refers to a bipartite Graph.
Postcondition: Constructs a MatchingGraph representing G with an
empty matching.
MatchingGraph(G)
n ← G.Size(); bipartite ← G; matching ← new Array[0..n − 1]
for i ← 0 to n − 1
matching[i] ← −1

MatchingGraph.Size()
return bipartite.Size() + 1

MatchingGraph.AllFrom(i)
n ← bipartite.Size(); L ← new ConsList()
if i < n
foundUnmatched ← false; adj ← bipartite.AllFrom(i)
while not adj.IsEmpty()
e ← adj.Head(); adj ← adj.Tail()
k ← e.Dest(); j ← matching[k]
if j = −1 and not foundUnmatched
L ← new ConsList(new Edge(i, n, k), L)
foundUnmatched ← true
else if j 6= −1
L ← new ConsList(new Edge(i, j, k), L)
return L

Precondition: i and j are distinct Nats representing vertices in the

bipartite graph.
Postcondition: Adds {i, j} to the matching, removing from the matching
any edge incident on i or j.
MatchingGraph.Match(i, j)
if matching[i] 6= −1
matching[matching[i]] ← −1
if matching[j] 6= −1
matching[matching[j]] ← −1
matching[i] ← j; matching[j] ← i
CHAPTER 14. NETWORK FLOW AND MATCHING 462

Figure 14.9 PathSearcher implementation of Searcher

PathSearcher(n)
incoming ← new Array[0..n − 1]

PathSearcher.TreePreProc(e)
incoming[e.Dest()] ← e

The implementation of PathSearcher is shown in Figure 14.9.

By Theorem 14.6, a depth-first search of a MatchingGraph can be
used to determine whether a given vertex can be safely matched. Note
that this theorem doesn’t specify which partition the vertex comes from. In
particular, we really don’t need to know the partition to which any vertex
belongs — we can simply test them in any order, and add the ones that
can be safely added. We therefore no longer need to require that the first
k vertices form the first partition. The algorithm is shown in Figure 14.10.
Note that in and M maintain references to the incoming variable in s and the
matching variable of matchGraph, respectively (this would not be possible
if PathSearcher and MatchingGraph were secure). Note also that as
long as an augmenting path is not found, we do not need to select any
unselected nodes because no unselected node leads to n.
Let n be the number of vertices and a be the number of edges in G. To
simplify the analysis of the running time, suppose each vertex has at least
one incident edge, so that n ∈ O(a). Let us first focus on a single iteration
of the for loop. Clearly, the running time of the call to Dfs is in O(a).
The number of iterations of the inner loop is at most the current size of the
matching, so its running time is in O(n) ⊆ O(a). The call to SelectAll
also runs in O(n) ⊆ O(a) time. We therefore conclude that a single iteration
of the for loop runs in O(a) time, so that the entire algorithm runs in O(na)
time.
To show that the running time of the algorithm is in Ω(na), we will first
construct a graph with 4k vertices and 4k − 1 edges for k ∈ N. We will
show that the algorithm runs in Ω(k 2 ) time for these graphs. We will then
generalize the construction to an arbitrary number n of vertices and a edges
such that n − 1 ≤ a < n(n + 20)/32. We will show that the algorithm runs
in Ω(na) time for these graphs.
CHAPTER 14. NETWORK FLOW AND MATCHING 463

Figure 14.10 Bipartite matching algorithm

Precondition: G is a bipartite graph.

Postcondition: Returns an array M [0..n − 1] describing a maximum-sized
matching of G, so that M [i] = j if j and i are matched, and M [i] = −1 if i
is unmatched.
Matching(G)
n ← G.Size(); sel ← new Selector(n)
s ← new PathSearcher(n + 1)
matchGraph ← new MatchingGraph(G)
in ← s.Incoming(); M ← matchGraph.Matching()
// Invariant: M represents a matching, and there is no matching
// containing the matched vertices in M and any unmatched vertex j < i.
for i ← 0 to n − 1
if M [i] = −1
Dfs(matchGraph, i, sel, s)
if not sel.IsSelected(n)
j←n
while j 6= i
e ← in[j]; k ← e.Source()
matchGraph.Match(k, e.Data()); j ← k
sel.SelectAll()
return M

We begin by setting V = {i | 0 ≤ i < 4k} (refer to Figure 14.11 for the

case in which k = 4). We then add the following edges:

• for 0 ≤ i < k, the edges {2i, 2k + i} and {2i + 1, 2k + i};

• for 0 < i < k, the edge {2i, 2k + i − 1}; and

• for 0 ≤ i < k, the edge {2k − 1, 3k + i}.

We arrange the edges so that when we try to add vertex 2i for 0 ≤ i <
k, we first encounter the edge {2i, 2k + i}. Because 2k + i is not in the
matching, it is added. It will then be impossible to add vertex 2i + 1,
but each node 2k + j, for 0 ≤ j < i, will be reached in the search for an
augmenting path. (For example, consider the search when trying to add 5
CHAPTER 14. NETWORK FLOW AND MATCHING 464

Figure 14.11 An illustration of the lower bound for Matching

8 9 10 11 12 13 14 15

0 1 2 3 4 5 6 7

to the matching {{0, 8}, {2, 9}, {4, 10}} in Figure 14.11.) Constructing this
matching therefore uses Ω(k 2 ) time.
We can now generalize the above construction to arbitrary n by adding
or removing a few vertices adjacent to 2k − 1. Furthermore, we can add
edges {2i, 2k + j} for 0 ≤ i < k and 0 ≤ j < i − 1 without increasing the
size of the maximum-sized matching. However, these additional edges must
all be traversed when we try to add vertex 2i + 1 to the matching. This
construction therefore forces the algorithm to use Ω(na) time. Furthermore,
the number of edges added can be as many as

X
k−1
k(k − 1)
(i − 1) = −k
2
i=0
k 2 − 3k
=
2
n2 − 12n
= .
32
Including the n−1 original edges, the total number of edges a is in the range

n(n + 20)
n−1≤a< .
32
The above construction is more general than we really need, but its
generality shows that some simple modifications to the algorithm won’t im-
prove its asymptotic running time. For example, the graph is connected,
so processing connected components separately won’t help. Also, the two
partitions are the same size, so processing the smaller (or larger) partition
first won’t help either. Furthermore, using breadth-first search won’t help
because it will process just as many edges when no augmenting path exists.
CHAPTER 14. NETWORK FLOW AND MATCHING 465

On the other hand, this algorithm is not the most efficient one known for
this problem. In the exercises, we explore how it might be improved.
Although the optimizations we made over a direct reduction to network
flow did not improve the asymptotic running time of the algorithm, the
resulting algorithm may have other advantages. For example, suppose we are
trying to match jobs with job applicants. Each applicant may be qualified
for several jobs. We wish to fill as many jobs as possible, but still assign
jobs so that priority is given to those who applied earlier. If we process the
applicants in the order in which they applied, we will obey this priority.

14.4 Summary
The network flow problem is a general combinatorial optimization problem
to which many other problems can be reduced. Although the Ford-Fulkerson
algorithm can behave poorly when the maximum flow is large in comparison
to the size of the graph, its flexibility makes it useful for those cases in which
the maximum flow is known to be small. For cases in which the maximum
flow may be large, the Edmonds-Karp algorithm, which is simply the Ford-
Fulkerson algorithm using breadth-first search to find augmenting paths,
performs adequately.
The bipartite matching problem is an example of a problem which occurs
quite often in practice and which can be reduced to network flow to yield a
reasonably efficient algorithm. Furthermore, a careful study of the reduction
yields insight into the problem that leads to a more general algorithm.

14.5 Exercises
Exercise 14.1 Prove Equation (14.1) on page 446. [Hint: Show by induc-
tion that the net flow out of any set of vertices including the source but not
the sink is equal to the left-hand side.]

Exercise 14.2 Prove that NetworkFlow, shown in Figure 14.3, meets

its specification.

Exercise 14.3 Implement CopyGraph, specified in Figure 14.3, to return

a ListGraph and run in O(n + a) time, where n is the number of vertices
and a is the number of edges in the given graph.

Exercise 14.4 Implement AddFlow, specified in Figure 14.3, to run in

O(n + a) time, where n is the number of vertices in F and a is the number
CHAPTER 14. NETWORK FLOW AND MATCHING 466

of edges in F and R together. For the purposes of your analysis, you may
assume that F and R are implemented as ListGraphs, and that the Edges
in P form a simple path in R.

* Exercise 14.5 Suppose we generalize the network flow problem to allow

positive rational edge capacities. Prove that the Ford-Fulkerson algorithm
always finds a maximum flow for such a network.

** Exercise 14.6 Suppose we generalize the network flow problem to al-

low positive real edge capacities. Give such a flow network for which the
Ford-Fulkerson algorithm does not terminate and does not converge to a
maximum flow.

Exercise 14.7 Implement the matching algorithm using breadth-first search

(see Exercise 11.6) to find augmenting paths.

** Exercise 14.8 Let G be a bipartite graph whose partitions are {i | 0 ≤

i < k} and {i | k ≤ i < n}, and let M be a matching on G smaller than the
maximum size. Suppose the minimum length of any augmenting path in G
is l. Let S = {P1 , . . . , Pm } be a maximal set of vertex-disjoint augmenting
paths of length l; i.e., any augmenting path of length l shares at least one
vertex with some path in S. We define the symmetric difference of two sets
A and B as
A ⊕ B = (A ∪ B) \(A ∩ B);
thus, the symmetric difference is the set of elements in exactly one of the
two sets.

a. Prove that M ′ = M ⊕ (P1 ∪ · · · ∪ Pm ) is a matching with m more edges

than M .

b. Prove that any augmenting path in M ′ , where M ′ is as defined above,

has more than l edges. [Hint: For the case in which P shares a
vertex with some Pi , define T = (M ⊕ M ′ ) ⊕ P . Prove that T =
(P1 ∪ · · · ∪ Pm ) ⊕ P and that T has at least (m + 1)l edges.]

c. Prove that the size of every matching exceeds the size of M by no

more than n/l.

d. Give an algorithm to find a maximal set of minimum-length augment-

ing paths. Your algorithm should run in O(a) time, where a is the
number of edges in G, assuming G is represented as a ListGraph
CHAPTER 14. NETWORK FLOW AND MATCHING 467

and n ∈ O(a). [Hint: Use both breadth-first search and a modified

depth-first search.]
√
e. Give an O(a n) algorithm to find a maximum-sized matching in G.

Exercise 14.9 Suppose we modify the network flow problem so that the
input includes an array cap[0..n − 1] of integers such that for each vertex i,
cap[i] gives an upper bound on the flow we allow to go to and from vertex
i. Show how to reduce this problem to the ordinary network flow problem.
Your reduction must run in O(n + a) time, where n is the number of vertices
and a is the number of edges in the graph.

Exercise 14.10 We define an n × n grid to be an undirected graph (V, E)

where V = {(i, j) | 1 ≤ i ≤ n, 1 ≤ j ≤ n}, and two vertices (i, j) and (i′ , j ′ )
are adjacent iff either i = i′ and j = j ′ ±1 or j = j ′ and i = i′ ±1. Thus, each
vertex in a grid has at most 4 neighbors. We call the vertices with fewer than
4 neighbors boundary vertices (i.e., these are vertices (1, j), (n, j), (i, 1), or
(i, n)). Give an O(mn2 ) algorithm which takes a value n ∈ N and m ≤ n2
starting vertices (i, j) ∈ [1..n] × [1..n] and determines whether there exists a
set of m vertex-disjoint paths in the n × n grid, each connecting a starting
node with a boundary node. You may assume you have an algorithm for
the problem stated in Exercise 14.9.

* Exercise 14.11 A path cover of a directed graph is a set of paths such

that every vertex is included in exactly one path. The size of a path cover is
the number of paths in the set. Show how to reduce the problem of finding
a minimum-sized path cover in a directed acyclic graph to the problem of
finding a maximum-sized matching in a bipartite graph. The total running
time of the algorithm should be in O(na).

* Exercise 14.12 We are given two arrays of integers, R[1..m] and C[1..n]
such that
Xm X
n
R[i] = C[i] = k.
i=1 i=1

Give an O(kmn) algorithm that returns an m × n matrix of 0s and 1s such

that row i contains exactly R[i] 1s and column j contains exactly C[j] 1s,
for all 1 ≤ i ≤ m and 1 ≤ j ≤ n. If there is no such matrix, your algorithm
should return nil.
CHAPTER 14. NETWORK FLOW AND MATCHING 468

** Exercise 14.13 Given a connected undirected graph G, the edge con-

nectivity of G is the minimum number of edges whose removal would dis-
connect the graph. Give an O(n2 a) algorithm to find the edge connectivity
of a given connected undirected Graph with n vertices and a edges. For
your running time analysis, you may assume the graph is represented as a
ListGraph. Prove the correctness of your algorithm.

14.6 Chapter Notes

The NetworkFlow algorithm is due to Ford and Fulkerson [40]. The
running-time analysis of the use of breadth-first search in the Ford-Fulkerson
algorithm is due to Edmonds and Karp [35] and Dinic [30]. Asymptotically
faster algorithms exist — to date, the fastest known is due to Goldberg and
Rao [55]. Their algorithm has a running time in

O(min(n2/3 , a1/2 )a lg(n2 /a + 2) lg C),

where C is the maximum capacity of any edge.

The technique of finding a maximum-sized matching using augmenting
paths is due to Berge [14]. He showed that in an arbitrary undirected graph,
a matching is of maximum size iff no augmenting path exists. Finding aug-
menting paths in arbitrary undirected graphs is more challenging, however,
because we must avoid returning to the same vertex from which we started.
The first efficient algorithm for finding an augmenting path in an arbitrary
undirected graph is due to Edmonds [33]. The algorithm suggested by Exer-
cise 14.8 is due to Hopcroft and Karp [61], and is the asymptotically fastest
known algorithm for finding a maximum-sized matching in a bipartite graph.
The structure of this exercise is based on a problem in Cormen, et al. [25].
√
An O(a n) algorithm for arbitrary undirected graphs was later given by
Micali and Vazirani [88].
A solution to Exercise 14.6 is given by Ford and Fulkerson [41].
Chapter 15

* The Fast Fourier Transform

In this chapter, we examine an algorithm whose discovery has had a profound

impact on several areas of science and engineering. Although we will not
delve into these application areas, we will show how it has been used to
design an arbitrary-precision natural number multiplication algorithm which
runs in O(n lg n lg lg n) time, where n is the number of bits in the product.
Along the way, we will examine some properties of the complex numbers as
well as the natural numbers. We begin by examining the computation of
a convolution, the fundamental problem that is solved by the Fast Fourier
Transform.

15.1 Convolutions
Let a = ha0 , . . . , am−1 i and b = hb0 , . . . , bn−1 i be two vectors. We define the
convolution of a and b as the vector c = hc0 , . . . , cm+n−2 i, where
min(j,m−1)
X
cj = ai bj−i .
i=max(0,j−n+1)

Many applications require the computation of a convolution. For example,

if a and b represent the coefficients of two polynomials, where ai and bi are
the respective coefficients for xi , then the convolution of a and b gives the
coefficients of their product.
We desire an efficient algorithm for computing a convolution. We can
gain some insight into how this might be done by examining the polynomial
multiplication problem. Clearly, the Θ(nlg 3 ) algorithm of Section 10.1 can
be used to compute a convolution. Furthermore, the solution to Exercise

469
CHAPTER 15. * THE FAST FOURIER TRANSFORM 470

10.4 shows that this can be done in O(n1+ǫ ) time for any ǫ ∈ R>0 (though
in fact the hidden constant becomes quite large as ǫ approaches 0). We wish
to improve on these algorithms.
It is a well-known fact that a polynomial of degree n − 1 is uniquely
determined by its values at any n distinct points. Therefore, one way to
multiply two polynomials p(x) and q(x) whose product has degree n − 1 is
as follows:
1. Evaluate p(xi ) and q(xi ) for n distinct values xi , 0 ≤ i < n.

2. Compute r(xi ) = p(xi )q(xi ) for 0 ≤ i < n.

3. Construct the unique polynomial r(x) of degree n − 1 determined by

the values r(xi ) for 0 ≤ i < n.
Note that step 2 can be done in Θ(n) time, assuming each multiplication
can be done in Θ(1) time. We need to show how steps 1 and 3 can be done
efficiently.
The evaluation of a polynomial of degree less than n at n distinct points
can be viewed as a linear transformation — i.e., a multiplication of a 1 × n
vector by an n × n matrix. Specifically, let p be the 1 × n vector representing
the coefficients of a polynomial p(x) as described above (if the degree is less
than n − 1, we can use coefficients of 0 for the high-order terms). Let A
be the n × n matrix such that Aij = xij for 0 ≤ i < n, 0 ≤ j < n, where
x0 , . . . , xn−1 are distinct values. Then the product pA yields the 1×n vector
v = hv0 , . . . , vn−1 i such that
n−1
X
vj = pi Aij
i=0
n−1
X
= pi xij
i=0
= p(xj ).

Furthermore, if A has an inverse A−1 , then this transformation is invert-

ible:

vA−1 = pAA−1
= p.

Thus, given the values of a polynomial at the n points x0 , . . . , xn−1 , we can

compute the polynomial by multiplying the vector of values by A−1 . The
CHAPTER 15. * THE FAST FOURIER TRANSFORM 471

product of two polynomials p(x) and q(x) is therefore represented by the

vector
(pA · qA)A−1 ,
where “·” denotes the component-wise product of two vectors of the same
size.
The main problem with this approach is that the multiplications of a 1×n
vector with an n×n array would appear to require Ω(n2 ) time. However, this
running time can be improved if we choose the points x0 , . . . , xn−1 cleverly.
In order to do this, we need to allow them to be chosen from the set of
complex numbers, C. We also need to define, for any n ≥ 1, a principal nth
root of unity as any value ω ∈ C such that

• ω n = 1; and

• for 1 ≤ j < n,
n−1
X
ω ij = 0.
i=0

We will show how to find such values in C. First, however, let us consider
why having a principal nth root of unity might be helpful. Given a principal
nth root of unity ω, let A be the n × n matrix such that Aij = ω ij . Given a
1 × n vector p, the product pA is said to be the discrete Fourier transform of
p with respect to ω. Note that if p is the coefficient vector for a polynomial
p(x), then pA gives the values of p(ω j ) for 0 ≤ j < n.
In what follows, we will develop a divide-and-conquer algorithm for com-
puting a DFT. To simplify matters, let’s assume that n is a power of 2. The
following theorem shows an important property of principal nth roots of
unity when n is a power of 2. We will use this property in designing our
divide-and-conquer algorithm.

Theorem 15.1 Let ω be a principal nth root of unity, where n ≥ 2 is a

power of 2. Then ω 2 is a principal (n/2)nd root of unity.

Proof: Because ω n = 1, (ω 2 )n/2 = 1. Let 1 ≤ j < n/2. Then 1 ≤ 2j < n.

Because ω is a principal nth root of unity, we have
n−1
X
ω i(2j) = 0.
i=0
CHAPTER 15. * THE FAST FOURIER TRANSFORM 472

Note that for i ≥ n/2,

ω i(2j) = ω nj ω (2i−n)j
= ω (2i−n)j

because ω n = 1. Therefore, we can write

n
n−1 2
−1 n−1
X X X
i(2j) 2ij
ω = ω + ω 2ij
i=0 i=0 i=n/2
n
2
−1 n−1
X X
2ij
= ω + ω (2i−n)j
i=0 i=n/2
n n
2
−1 2
−1
X X
2ij
= ω + ω (2(i+n/2)−n)j
i=0 i=0
n
2
−1
X
=2 ω 2ij .
i=0

Because the above value is 0, it follows that

n
2
−1
X
(ω 2 )ij = 0.
i=0

Hence, ω 2 is a principal (n/2)nd root of unity.

Knowing that ω 2 is a principal n/2nd root of unity, we can now reduce

the problem of computing a DFT for a 1 × n vector to two smaller instances.
We form these two smaller instances by dividing a given 1 × n vector p
into its odd components p′ = hp1 , p3 , . . . , pn−1 i and its even components
p′′ = hp0 , p2 , . . . , pn−2 i. Thus,
n
n−1 2
−1
X X
pi ω ij = (p2i ω 2ij + p2i+1 ω (2i+1)j )
i=0 i=0
n n
2
−1 2
−1
X X
= p2i ω 2ij + ω j p2i+1 ω 2ij .
i=0 i=0

Note that each sum on the right-hand side is the jth component of the
DFT with respect to ω 2 of a 1 × n/2 vector. Specifically, let d′ and d′′ be the
CHAPTER 15. * THE FAST FOURIER TRANSFORM 473

DFTs of p′ and p′′ , respectively, with respect to ω 2 , and let d be the DFT
of p with respect to ω. Then for 0 ≤ j < n/2, we have

dj = d′′j + ω j d′j . (15.1)

Furthermore,
n n
2
−1 2
−1
X X
dj+n/2 = p′′i ω 2i(j+n/2) + ω j+n/2 p′i ω 2i(j+n/2)
i=0 i=0
n n
2
−1 2
−1
X X
= p′′i ω 2ij + ω j+n/2 p′i ω 2ij
i=0 i=0
= d′′j + ω j+n/2 d′j . (15.2)

The above equation can be simplified somewhat by applying the following

theorem.

Theorem 15.2 Let n > 1 be a power of 2. Then ω is a principal nth root

of unity iff ω n/2 = −1.

Proof:

⇒: Suppose ω is a principal nth root of unity. It follows from Theorem 15.1

by induction on n that ω n/2 is a principal 2nd root of unity. It is therefore
sufficient to show that if ω is a principal 2nd root of unity, then ω = −1.
Suppose ω is a principal 2nd root of unity. Then from the definition, we
have
1
X
0= ωi
i=0
= 1 + ω.

Rearranging terms, we have ω = −1.

⇐: Let n = 2k . We will show by induction on k ≥ 1 that if ω n/2 = −1, then

ω is a principal nth root of unity.

Base: k = 1. Then n = 2, and n/2 = 1. Because −1 is a principal 2nd root

of unity, the result follows.
CHAPTER 15. * THE FAST FOURIER TRANSFORM 474

Induction Hypothesis: Assume for some k > 1 that whenever 1 ≤ k ′ < k

′
and n = 2k , if ω n/2 = −1, then ω is a principal nth root of unity.

Induction Step: Suppose n = 2k and ω n/2 = −1. Clearly, ω n = 1. Let

1 ≤ j < n. We consider two cases.

Case 1: j is odd. We therefore have

n
n−1 2
−1 n−1
X X X
ij ij
ω = ω + ω ij
i=0 i=0 i=n/2
n n
2
−1 2
−1
X X
ij
= ω + ω (i+n/2)j
i=0 i=0
n n
2
−1 2
−1
X X
= ω ij + ω ij (ω n/2 )j
i=0 i=0
n n
2
−1 2
−1
X X
= ω ij + ω ij (−1)j
i=0 i=0
= 0.

Case 2: j is even. Then because 1 ≤ j < n, n must be at least 4. We first

observe that

(ω 2 )n/4 = ω n/2
= −1.

By the Induction Hypothesis, ω 2 is a principal (n/2)nd root of unity. We

then have
n−1
X n−1
X
ij
ω = (ω 2 )ij/2
i=0 i=0
n
2
−1 n−1
X X
= (ω 2 )ij/2 + (ω 2 )ij/2
i=0 i=n/2
n−1
X
= (ω 2 )ij/2
i=n/2
CHAPTER 15. * THE FAST FOURIER TRANSFORM 475

because ω 2 is a principal (n/2)nd root of unity. Re-indexing the above

equality, we have
n−1
X n−1
X
ω ij = (ω 2 )ij/2
i=0 i=n/2
n
2
−1
X
= (ω 2 )(i+n/2)j/2
i=0
n
2
−1
X
= (ω 2 )(n/2)j/2 (ω 2 )ij/2
i=0
n
2
−1
X
= (ω 2 )ij/2
i=0
= 0.

We conclude that ω is a principal nth root of unity.

Using the fact that ω n/2 = −1, we can now rewrite (15.2) for 0 ≤ j < n/2
as
dj+n/2 = d′′j − ω j d′j . (15.3)
We therefore have the divide-and-conquer algorithm, known as the Fast
Fourier Transform, shown in Figure 15.1. Note that we use the type Com-
plex to represent a complex number.
Because Fft should only be called with a vector whose size n is a power
of 2, n is not a good measure of the size of the problem instance for the
purpose of analyzing the algorithm. Instead, we will use k = lg n. Assuming
each arithmetic operation on complex numbers can be performed in Θ(1)
time, it is easily seen that the running time excluding the recursive calls is
in Θ(2k ). The worst-case running time is therefore given by the recurrence

f (k) ∈ 2f (k − 1) + Θ(2k ).

From Theorem 3.31, f (k) ∈ Θ(k2k ).

In order to use Fft to compute a convolution, we need to be able to
compute the inverse transform. Let A be the n × n matrix defining a DFT.
In order to compute the inverse transform, we need to know that A−1 exists,
and we need an efficient way to multiply a given 1 × n vector on the right
by A−1 . The following theorem gives A−1 .
CHAPTER 15. * THE FAST FOURIER TRANSFORM 476

Figure 15.1 The Fast Fourier Transform algorithm

Precondition: p[0..n − 1] is an array of Complexes, n is a Nat containing

a power of 2, and ω is a Complex containing a principal nth root of unity.
Postcondition: Returns the DFT of p with respect to ω.
Fft(p[0..n − 1], ω)
d ← new Array[0..n − 1]; mid ← n/2
if n = 1
d[0] ← p[0]
else
p′ ← new Array[0..mid − 1]; p′′ ← new Array[0..mid − 1]
for i ← 0 to mid − 1
p′′ [i] ← p[2i]; p′ [i] ← p[2i + 1]
d ← Fft(p′ , ω 2 )
′

d′′ ← Fft(p′′ , ω 2 )
υ←1
// Invariant: d[0..i − 1] and d[mid..mid + i − 1] contain the correct
// values for the DFT of p, and υ = ω i .
for i ← 0 to mid − 1
d[i] ← d′′ [i] + υ(d′ [i])
d[i + mid] ← d′′ [i] − υ(d′ [i])
υ ← υω
return d

Theorem 15.3 Let A be the n × n matrix such that for 0 ≤ i < n and
0 ≤ j < n, Aij = ω ij , where ω is a principal nth root of unity. Then A−1 is
the matrix B, where Bij = ω −ij /n.

Proof: We must show that AB = I, where

(
1 if i = j
Iij =
0 otherwise
CHAPTER 15. * THE FAST FOURIER TRANSFORM 477

for 0 ≤ i < n and 0 ≤ j < n. Let C = AB. Then

n−1
X
Cij = Aik Bkj
k=0
n−1
X
= ω ik ω −kj /n
k=0
n−1
1X
= ω k(i−j) .
n
k=0

We now consider three cases.

Case 1: i = j. Then
n−1
1 X k(i−j)
Cij = ω
n
k=0
n−1
1 X
= ω0
n
k=0
= 1.

Case 2: i > j. Then 1 ≤ i − j < n. Because ω is a principal nth root of

unity, we have
n−1
1 X k(i−j)
Cij = ω
n
k=0
= 0.

Case 3: i < j. Then 1 ≤ i−j +n < n. Because ω n = 1, ω k(i−j) = ω k(i−j+n) .

Hence, as in Case 2, Cij = 0.
We conclude that C = I, so that B = A−1 .

Note that the matrix A−1 can be written A′ /n, where A′ij = ω −ij . The
following theorem shows that ω −1 is also a principal nth root of unity, so
that multiplication by A′ is also a DFT. As a result, we can use Fft to
compute the inverse transform.

Theorem 15.4 Let ω be a principal nth root of unity, where n ≥ 2 is a

power of 2. Then ω −1 is a principal nth root of unity.
CHAPTER 15. * THE FAST FOURIER TRANSFORM 478

Proof: From Theorem 15.2, we need only to show that ω −n/2 = −1. Be-
cause ω n = 1, we have

ω −n/2 = ω n−n/2
= ω n/2
= −1

from Theorem 15.2. ω −1 is therefore a principal nth root of unity.

In order to complete the convolution algorithm, we need a principal nth

root of unity for each n that is a power of 2. The following theorem provides
these values. The theorem actually holds for all positive n, but the proof is
simpler when n is a power of 2.

Theorem 15.5 Let n be a power of 2. Then

2π 2π
cos + i sin
n n
is a principal nth root of unity.

Proof: We first observe that if n = 1, then

2π 2π
cos + i sin = 1 + 0i
n n
= 1.

Thus, from Theorem 15.2, it suffices to show that

n/2
2π 2π
cos + i sin = −1
n n

whenever n > 1 is a power of 2. We proceed by induction on n.

Base: n = 2. Then

2π n/2

2π
cos + i sin = cos π + i sin π
n n
= −1 + 0i
= −1.
CHAPTER 15. * THE FAST FOURIER TRANSFORM 479

Induction Hypothesis: Assume for some n > 2, where n is a power of 2,

that for any k such that 1 < k < n, if k is a power of 2, then

2π k/2

2π
cos + i sin = −1.
k k

Induction Step:
!n/4
2π n/2 2π 2

2π 2π
cos + i sin = cos + i sin
n n n n
n/4
2 2π 2 2π 2π 2π
= cos − sin + 2i cos sin .
n n n n

We now apply the following trigonometric identities:

cos2 x − sin2 x = cos 2x

and
2(cos x sin x) = sin 2x.
We therefore have

2π n/2 4π n/4

2π 4π
cos + i sin = cos + i sin
n n n n
n/2
2π 2π 2
= cos + i sin
n/2 n/2
= −1

by the Induction Hypothesis.

Before we give the algorithm for computing a convolution, let us consider

a slight generalization of the problem. We have defined a convolution to be
a vector whose size is the sum of the sizes of two given vectors. In order to
apply Fft, we must pad the two input vectors with enough zeros so that
each has a size n, where n is a power of 2 and is at least as large as the size
of the convolution. It would be somewhat easier to do this if we required
the input vectors to be padded with enough zeros so that they were both
the size of the convolution.
Let us therefore consider the value of the vector (pA · qA)A−1 when p
and q are arbitrary 1 × n vectors over C and A is the DFT matrix with
CHAPTER 15. * THE FAST FOURIER TRANSFORM 480

respect to ω. The jth component of the vector pA is given by

n−1
X n−1
X
pi Aij = pi ω ij .
i=0 i=0

The jth component of the component-wise product (pA · qA) is therefore

n−1
X n−1
X X n−1
n−1 X
ij kj
pi ω qk ω = pi qk ω (i+k)j .
i=0 k=0 i=0 k=0

Finally, multiplying the above summation on the right by A−1 , we obtain a

vector whose jth component is

X n−1
X n−1
n−1 n−1 n−1 n−1
X 1 XXX
pi qk ω (i+k)l ω −lj /n = pi qk ω (i+k−j)l
n
l=0 i=0 k=0 l=0 i=0 k=0
X n−1
n−1 n−1
1 X X
= pi qk ω (i+k−j)l .
n
i=0 k=0 l=0

We now observe that in the exponent for ω above, 1 − n ≤ i + k − j ≤

2n − 2. Because ω n = 1, we can multiply any of the terms by either ω n or
ω −n without changing its value. Hence, because ω is a principal nth root of
unity,
n−1
(
X n if i + k − j = 0 or i + k − j = n
ω (i+k−j)l =
l=0
0 otherwise.

Therefore, the jth component of (pA · qA)A−1 is

j
X n−1
X
pi qj−i + pi qj−i+n .
i=0 i=j+1

We will refer to this vector as the positive wrapped convolution of p and q.

We will denote this operation by p ⊗ q. Note that if either pi or qj is 0
whenever i + j ≥ n, the second summation in the above definition is 0, so
that we can reduce the problem of computing an ordinary convolution to
the problem of computing a positive wrapped convolution.
Figure 15.2 gives an algorithm for computing a positive wrapped convo-
lution using the Fast Fourier Transform. Note that m is the smallest power
of 2 no larger than n, so that n ≤ m < 2n. Assuming each arithmetic
operation can be performed in Θ(1) time, the running time excluding the
CHAPTER 15. * THE FAST FOURIER TRANSFORM 481

Figure 15.2 Algorithm for computing a positive wrapped convolution over

C using the Fast Fourier Transform

Precondition: p[0..n − 1] and q[0..n − 1] are arrays of Complexes, and n

is a positive Nat.
Postcondition: Returns the positive wrapped convolution of p and q.
Convolution(p[0..n − 1], q[0..n − 1])
m ← 2⌈lg n⌉ ; p′ ← new Array[0..m − 1]; q ′ ← new Array[0..m − 1]
Copy(p[0..n − 1], p′ [0..n − 1]); Copy(q[0..n − 1], q ′ [0..n − 1])
for i ← n to m − 1
p′ [i] ← 0; q ′ [i] ← 0
ω ← cos(2π/m) + i sin(2π/m)
ptrans ← Fft(p′ , ω); qtrans ← Fft(q ′ , ω)
rtrans ← new Array[0..m − 1]
for i ← 0 to m − 1
rtrans[i] ← ptrans[i]qtrans[i]
r′ ← Fft(rtrans, 1/ω); r ← new Array[0..n − 1]
for i ← 0 to n − 1
r[i] ← r′ [i]/m
return r

calls to Fft is in Θ(n). If k = lg m, the running time for each call to Fft is
in Θ(k2k ) = Θ(n lg n). The overall running time is therefore in Θ(n lg n). It
is easily seen that this algorithm can be used to multiply two polynomials
over C in Θ(n lg n) time, where n is the degree of the product.
Throughout this discussion, we have been assuming that we can store ar-
bitrary complex numbers and perform arithmetic operations on them in Θ(1)
time. These assumptions are rather dubious. However, for most scientific
and engineering applications, it is sufficient to use floating-point approxima-
tions. The Convolution algorithm is therefore very useful in practice.

15.2 Commutative Rings

In Exercise 10.5 (page 366), we suggested that the polynomial multiplica-
tion algorithm of Section 10.1 could be adapted to multiply two arbitrary-
precision natural numbers in Θ(nlg 3 ) time, where n is the number of bits
CHAPTER 15. * THE FAST FOURIER TRANSFORM 482

in the product. Because we now have a Θ(n lg n) algorithm for multiplying

polynomials, we might conclude that it could be used to multiply arbitrary-
precision natural numbers in Θ(n lg n) time. However, there are two prob-
lems with this conclusion. First, Convolution uses complex numbers hav-
ing infinite binary representations. It turns out that if we are careful, we
can use finite approximations and still obtain correct results for arbitrary-
precision multiplication. The second problem is more serious, though. Note
that both Convolution and Fft perform multiplications involving val-
ues derived from the vectors they are processing. These values can become
much larger than the original elements of the vectors, so that recursive calls
would need to be made to do these multiplications. This has the effect of
increasing the running time.
In view of the above complications, we will take a somewhat different
approach. In this section, we will show that the results of the previous
section can be extended to various other mathematical structures, including
some involving only natural numbers. In the next section, we will develop a
multiplication algorithm that uses the Fast Fourier Transform over certain
of these structures. This algorithm will have a running time in O(n lg2 n).
In the following section, we will show how to improve it to achieve a running
time in O(n lg n lg lg n). This algorithm is asymptotically the fastest known
algorithm for arbitrary-precision multiplication.
In order to show how the results of the previous section extend to other
mathematical structures, we need a few definitions. Let S be a set, and let
+ be any binary operation on S; i.e., for every x, y ∈ S, x + y ∈ S. The pair
hS, +i is said to be a group if the following properties hold:

• Associativity: For every x, y, z ∈ S, (x + y) + z = x + (y + z).

• Identity: There is an element 0 ∈ S such that for every x ∈ S,

0 + x = x + 0 = x.

• Inverse: For every element x ∈ S, there is an element −x ∈ S such

that x + (−x) = −x + x = 0.

If, in addition commutativity holds — for every x, y ∈ S, x + y = y + x —

then we say hS, +i is an abelian group.

Example 15.6 hZ, +i, the set of integers with addition, is an abelian group.

Example 15.7 hN, +i, the set of natural numbers with addition, is not a
group because only 0 has an inverse.
CHAPTER 15. * THE FAST FOURIER TRANSFORM 483

Example 15.8 For a positive integer m, let Zm denote the set of natural
numbers strictly less than m, and let + denote addition mod n. It is not
hard to see that hZm , +i is an abelian group, with 0 being the identity and
n − i being the inverse of i.

Example 15.9 Let S3 be the set of permutations of three distinct elements,

and let ◦ denote composition. We denote a permutation by the result of
applying it to h1, 2, 3i; for example, the permutation that swaps the first and
second elements is denoted by h2, 1, 3i. Then h2, 1, 3i◦h1, 3, 2i = h2, 3, 1i, but
h1, 3, 2i ◦ h2, 1, 3i = h3, 1, 2i. Hence, hS3 , ◦i is not commutative. However, it
is not hard to see that it is a group, with h1, 2, 3i being the identity element,
h2, 3, 1i being the inverse of h3, 1, 2i and vice versa, and every other element
being its own inverse.
Let hS, +i be an abelian group, and let · be a binary operation on S.
Then hS, +, ·i is said to be a ring if the following properties hold for every
x, y, and z in S:

• Associativity: (x · y) · z = x · (y · z).

• Distributivity: x · (y + z) = x · y + x · z and (x + y) · z = x · z + y · z.

If, in addition, commutativity holds for ·, hS, +, ·i is said to be a commutative

ring. If a ring hS, +, ·i has an element 1 ∈ S such that for every x ∈ S,
x · 1 = 1 · x = x, then 1 is said to be a unit element.

Example 15.10 It is not hard to see that hC, +, ·i, where + and · denote
ordinary addition and multiplication, respectively, is a commutative ring
with unit element 1.

Example 15.11 For a positive integer m, consider hZm , +, ·i, where + is

addition mod m and · is multiplication mod m. As we observed in Example
15.8, hZm , +i is an abelian group. It is not hard to see that hZm , +, ·i
is a commutative ring, and that 1 is a unit element. We will be using
commutative rings of this form for the multiplication algorithms of the next
two sections.

Example 15.12 Let S = {0, 2, 4, 6}, and let + and · denote addition and
multiplication, respectively, mod 8. Then it is not hard to see that hS, +, ·i
is a commutative ring. However, it does not have a unit element, because
0 · 2 = 0, 2 · 4 = 0, and 6 · 2 = 4.
CHAPTER 15. * THE FAST FOURIER TRANSFORM 484

Example 15.13 Let S be the set of 2 × 2 matrices over R, and let + and
· denote matrix addition and multiplication, respectively. Then It is not
hard to show that hS, +, ·i is a ring, and that the identity matrix is a unit
element. However, the ring is not commutative; for example,

1 1 1 0 2 1
= ,
0 1 1 1 1 1

but
1 0 1 1 1 1
= .
1 1 0 1 1 2
In what follows, we will show that the results of the previous section
extend to an arbitrary commutative ring R = hS, +, ·i with unit element
1. For convenience, we will typically abbreviate x · y as xy. We will also
abbreviate x + (−y) as x − y.
We first observe that for x ∈ S and n ∈ N, we can define xn as follows:
(
n 1 if n = 0
x = n−1
xx otherwise.

Hence, the definition of a principal nth root of unity makes sense for R.
Furthermore, the definition of a discrete Fourier transform also makes sense
over this ring. The following theorem states that some familiar properties
of exponentiation must hold for any ring with unit element; its proof is left
as an exercise.

Theorem 15.14 Let R be any ring with unit element. Then the following
properties hold for any x in R and any m, n ∈ N:
a. xm xn = xm+n .

b. (xm )n = x(mn) .
Theorem 15.1 can be shown using only the properties given in the def-
inition of a ring, together with Theorem 15.14. It therefore applies to R.
The derivations of equations (15.1) and (15.2) use the properties of a ring,
together with commutativity, so that they also hold for R. The proof of
Theorem 15.2 applies for arbitrary rings with unit elements, so equation
(15.3) holds for R. The algorithm Fft therefore can be used to compute a
DFT over R, provided ω is a principal nth root of unity for that ring, and
that addition and multiplication on elements of the ring are the + and ·
operations from R.
CHAPTER 15. * THE FAST FOURIER TRANSFORM 485

In order to extend Theorem 15.3 to R, we must consider what it would

mean to divide by n in that ring. First of all the ring might not contain
n as an element. However, we can always embed the integers into a ring
with unit element as follows. First, if the ring has a unit element 1, it also
contains −1 (the additive inverse of 1) and 0 (the additive identity). For
n > 1, if n − 1 is in the ring, we can give the element (n − 1) + 1 the name n,
and we can give the element −(n − 1) − 1 the name −n. Thus, each integer
refers to some element of the ring. Note that a particular element of the
ring might not correspond to any integer, or it might correspond to more
than one. If it does correspond to more than one integer, it is not hard to
show that it corresponds to infinitely many integers.
Now that we have identified n with some element in the ring, we can
define division by n as multiplication by n−1 , provided n has a multiplicative
inverse. We note that if ω is a principal nth root of unity, then ωω n−1 = 1,
so that ω −1 = ω n−1 . Then the proof of Theorem 15.3 can easily be seen to
apply to an arbitrary ring with unit element, provided n has a multiplicative
inverse in that ring. Because the proof of Theorem 15.4 also applies to an
arbitrary ring with unit element, Fft can be used to compute an inverse
DFT over R.
In order to compute a convolution over R, we need to be able to find
a principal nth root of unity when n is a power of 2. Unfortunately, not
every commutative ring with unit element has a principal nth root of unity
whenever n is a power of 2. In the next section, we will focus on partic-
ular commutative rings and determine when they have principal nth roots
of unity. We will then show how to multiply arbitrary-precision natural
numbers by using the Fast Fourier Transform over these rings.

15.3 Integer Multiplication

Suppose we wish to multiply two BigNums, u and v, as specified in Exercise
4.14 on page 145. If uv contains n bits, and m ≥ 2n , then the product uv
in the ring hZm , +, ·i is the same as the ordinary product over N. We have
therefore reduced arbitrary-precision multiplication to multiplication in a
ring of the form hZm , +, ·i. In this section and the next, we show how to
use the Fast Fourier Transform to compute a product over such a ring for
specific values of m.
We first need to choose m in such a way that we can find principal nth
roots of unity in the ring, when n is a power of 2. Because Theorem 15.2
holds for any ring with unit element, we need to find m and ω such that
CHAPTER 15. * THE FAST FOURIER TRANSFORM 486

ω n/2 mod m = m − 1, the inverse of 1 in hZm , +i. One way of satisfying this
constraint is to select m = 2k + 1 for some positive integer k. Then 22k/n is
a principal nth root of unity, provided 2k is divisible by n. Because n is a
power of 2, we should require k to be a power of 2 such that 2k ≥ n.
We also need to be able to find the multiplicative inverse of n in this
ring. Because 22k/n is a principal nth root of unity, 22k = 1 in this ring.
Therefore, n−1 = 22k /n. We therefore have the following theorem.

Theorem 15.15 Let k and n be powers of 2 such that 1 ≤ n ≤ 2k, and let
m = 2k + 1. In the ring hZm , +, ·i:

a. 22k/n is a principal nth root of unity; and

b. n−1 = 22k /n.

Note that if k and n are both powers of 2 such that 1 ≤ n ≤ 2k, both
22k/n and 22k /n are also powers of 2. This fact is advantageous because
multiplying a BigNum by a power of 2 can be done very efficiently via the
Shift operation.
In order to complete the reduction of arbitrary-precision multiplication
to multiplication in a ring hZm , +, ·i, we must select a specific m. Suppose
the natural numbers u and v together have a total of n bits. Then uv will
have at most n bits. We can therefore set k to the smallest power of 2 no
smaller than n, and let m = 2k + 1. The resulting algorithm is shown in
Figure 15.3 (see Figure 10.6 on page 350 for its specification).
Let us consider how to multiply two k-bit numbers, u and v, mod 2k + 1,
where k is a power of 2. Suppose we break u and v into b blocks of l bits
each. Let these blocks be u0 , . . . , ub−1 and v0 , . . . , vb−1 , so that
b−1
X
u= ui 2il
i=0

and
b−1
X
v= vi 2il .
i=0

The product uv mod (2k + 1) is then given by

 
X min(j,b−1)
2b−1 X
uv mod (2k + 1) =  ui vj−i 2jl  mod (2k + 1).
j=0 i=max(0,j−b+1)
CHAPTER 15. * THE FAST FOURIER TRANSFORM 487

Figure 15.3 Implementation of Multiply (specified in Figure 10.6, p. 350)

using modular multiplication

MultFft(u, v)
n ← Max(1, u.NumBits() + v.NumBits()); k ← 2⌈lg n⌉
return ModMult(u, v, k)

Precondition: u and v are BigNums each having at most k bits, and k is

a Nat containing a power of 2.
Postcondition: Returns a BigNum representing

uv mod (2k + 1).

ModMult(u, v, k)

Note that the last term in the above sum (i.e., for j = 2b − 1) is 0. We
include it in order to simplify the derivation that follows.
Because k = bl, 2bl = −1 in the ring hZm , +, ·i, where m = 2k + 1. We
can therefore write the product uv in this ring as
   
b−1 X
X j 2b−1
X X b−1
uv =  ui vj−i 2jl  −  ui vj−i 2(j−b)l 
j=0 i=0 j=b i=j−b+1
   
X j
b−1 X b−1 X
X b−1
= ui vj−i 2jl  −  ui vj−i+b 2jl 
j=0 i=0 j=0 i=j+1
 
b−1
X j
X b−1
X
= 2jl  ui vj−i − ui vj−i+b  .
j=0 i=0 i=j+1

Let p = hp0 , . . . , pb−1 i, where

j
X b−1
X
pj = ui vj−i − ui vj−i+b .
i=0 i=j+1

Thus, in hZm , +, ·i,

b−1
X
uv = pj 2jl .
j=0
CHAPTER 15. * THE FAST FOURIER TRANSFORM 488

Furthermore, the vector p closely resembles the positive wrapped convolu-

tion hu0 , . . . , ub−1 i ⊗ hv0 , . . . , vb−1 i. The only difference is that the two sums
are subtracted, rather than added. For this reason, we define p to be the
negative wrapped convolution of the two vectors. The following theorem
shows how computing a negative wrapped convolution can be reduced to
computing a positive wrapped convolution.

Theorem 15.16 Let R be a commutative ring with unit element, and sup-
pose ψ is a principal (2n)th root of unity in R. Let p and q be 1 × n vectors
over R, and let Ψ and Ψ′ be 1×n vectors such that Ψj = ψ j and Ψ′j = ψ 2n−j
for 0 ≤ j < n. Then the negative wrapped convolution of p and q is given
by
Ψ′ · ((Ψ · p) ⊗ (Ψ · q)), (15.4)
where · denotes the component-wise product of two vectors over R.

Proof: Let 0 ≤ j < n. Then the jth component (15.4) is

 
j
X n−1
X
ψ 2n−j  ψ i pi ψ j−i qj−i + ψ i pi ψ j−i+n qj−i+n 
i=0 i=j+1
j
X n−1
X
2n 3n
=ψ pi qj−i + ψ pi qj−i+n
i=0 i=j+1
j
X n−1
X
n
= pi qj−1 + ψ pi qj−i+n
i=0 i=j+1
j
X n−1
X
= pi qj−1 − pi qj−i+n .
i=0 i=j+1

Let us therefore reduce multiplication in the ring hZm , +, ·i to computing

a negative wrapped convolution. In order to do this, the negative wrapped
convolution must be computed over a commutative ring with a principal
′
(2b)th root of unity. If we are to use a ring hZm′ , +, ·i, where m′ = 2k + 1
and k ′ is a power of 2, then from Theorem 15.15, we must have k ′ ≥ b.
Furthermore, m′ must be large enough so that
 
Xj b−1
X
 ui vj−i − ui vj−i+b  mod m′
i=0 i=j+1
CHAPTER 15. * THE FAST FOURIER TRANSFORM 489

uniquely determines
j
X b−1
X
ui vj−i − ui vj−i+b .
i=0 i=j+1

Because each component of u and v is strictly less than 2l , the above expres-
sion is strictly less than b22l and strictly greater than −b22l . We therefore
need
′
2k + 1 ≥ 2b22l
k ′ ≥ lg(b22l+1 − 1).

The above inequality is satisfied if k ′ ≥ lg b + 2l + 1. Because our convo-

lution algorithm works on vectors whose size is a power of 2, it makes sense
to choose b as a power of 2. Because k is a power of 2, this implies that
l = k/b is also a power of 2. Because k ′ must also be a power of 2, we can
satisfy this inequality by taking k ′ ≥ 4l, provided lg b + 1 ≤ 2l.
In order to make k ′ as small as possible and still at least max(b, 4l), we
should choose
√ b and 4l to√be roughly equal. If k is an even power of 2, we can
set l = k/2 and b = 2 k. In this case, both p will be powers √of 2, and we
can set k ′ = b = 4l. Otherwise, we can set l = k/2 and b = 2k. Again,
both are powers of 2, and in this case we can set k ′ = 4l. Furthermore, it is
easily seen that for these choices, lg b + 1 ≤ 2l whenever k ≥ 16.
Finally, the computation of the negative wrapped convolution will in-
volve arithmetic, including multiplication, over the ring we choose. Thus,
this computation will be reduced to modular multiplication. In order to
avoid a circular reduction, we must make sure we choose k ′ < k. It is easily
seen that when k ≥ 16, this constraint is satisfied.
The reduction of modular multiplication to a negative wrapped con-
volution is shown in Figure 15.4. For the base case, it uses some other
multiplication algorithm satisfying the spec given in Figure 10.6. It uses a
function ToRing to apply the mod 2k + 1 operation. It also uses a func-
tion Eval, which coverts the negative wrapped convolution into the product
mod 2k + 1. We will consider the design of these two functions shortly.
The NegConv function can now be implemented by directly applying
Theorem 15.16. Its implementation is shown in Figure 15.5. In order to
multiply a BigNum x by ψ j , which is a power of 2, we shift x to the right
by j lg ψ bits. We use the variable lgPsi to store the value lg ψ.
NegConv uses the function PosConv, whose implementation is shown
in Figure 15.6. This algorithm is simply a modification of Convolution
CHAPTER 15. * THE FAST FOURIER TRANSFORM 490

Figure 15.4 Implementation of ModMult, specified in Figure 15.3

ModMultFft(u, v, k)
if k < 16
return ToRing(MultiplyAdHoc(u, v), k)
else
if (lg k) mod
√ 2=√ 0
b ← 2 k; l ← k/2
else √ p
b ← 2k; l ← k/2
uarray ← new Array[0..b − 1]; varray ← new Array[0..b − 1]
for j ← 0 to b − 1
uarray[j] ← new BigNum(u.GetBits(jl, l))
varray[j] ← new BigNum(v.GetBits(jl, l))
conv ← NegConv(uarray, varray, 4l)
return Eval(conv, k, l)

Precondition: x is a BigNum and k is a positive Nat.

Postcondition: Returns a BigNum representing x mod (2k + 1).
ToRing(x, k)
Precondition: p[0..n − 1] and q[0..n − 1] are arrays of BigNums each
having at most k bits, k is a Nat containing a power of 2, and n is a Nat
containing a power of 2 such that n ≤ k.
Postcondition: Returns a BigNum representing the negative wrapped
convolution of p and q over the ring hZm , +, ·i, where m = 2k + 1.
NegConv(p[0..n − 1], q[0..n − 1], k)
Precondition: v[0..n − 1] is an array of BigNums no larger than 24l , n, k,
and l are Nats containing powers of 2.
Postcondition: Returns a BigNum representing
 
n−1
X
 uj 2jl  mod (2k + 1),
j=0

where uj = v[j] if v[j] ≤ 24l−1 , or uj = v[j] − (24l + 1) otherwise.

Eval(v[0..n − 1], k, l)
CHAPTER 15. * THE FAST FOURIER TRANSFORM 491

Figure 15.5 Implementation of NegConv, as specified in Figure 15.4

NegConv(p[0..n − 1], q[0..n − 1], k)

lgPsi ← k/n; p′ ← new Array[0..n − 1]; q ′ ← new Array[0..n − 1]
for j ← 0 to n − 1
p′ [j] ← ToRing(p[j].Shift(j · lgPsi), k)
q ′ [j] ← ToRing(q[j].Shift(j · lgPsi), k)
r′ ← PosConv(p′ , q ′ , k)
r ← new Array[0..n − 1]
for j ← 0 to n − 1
r[j] ← ToRing(r′ [j].Shift((2n − j)lgPsi), k)
return r

Precondition: p[0..n − 1] and q[0..n − 1] are arrays of BigNums each no

larger than 2k , k is a Nat containing a power of 2, and n is a Nat containing
a power of 2 such that n ≤ k.
Postcondition: Returns a BigNum representing the positive wrapped con-
volution of p and q over the ring hZm , +, ·i, where m = 2k + 1.
PosConv(p[0..n − 1], q[0..n − 1], k)

for the modular ring; however, because the precondition requires that n is a
power of 2, we don’t need to copy the elements to arrays of such a size. In
order to facilitate multiplication by n−1 , we use the variable lgInv to store
lg(n−1 ). Also, recall that the precondition for ModMult (Figure 15.3)
requires that each argument is at most k bits. However, the discrete Fourier
transforms may contains elements equal to 2k , which has k + 1 bits. We
must therefore handle this case separately.
The principal nth root of unity used for computing the DFT will be 22k/n .
For computing the inverse DFT, we therefore must use the multiplicative
inverse of 22k/n . Because 22k mod (2k + 1) = 1, (22k/n )−1 = 22k−2k/n . For
reasons of efficiency and ease of analysis, we use a boolean to indicate which
of these roots the function ModFft is to use.
The implementation of ModFft is shown in Figure 15.7. It is a fairly
straightforward adaptation of Fft (Figure 15.1) to the ring hZm , +, ·i. We
must be careful, however, when subtracting ω i d′ [i] from d′′ [i] in order to
obtain d[i + mid], because ω i d′ [i] may be greater than d′′ [i]. In order to
satisfy the precondition of BigNum.Subtract (Figure 4.18 on page 146),
CHAPTER 15. * THE FAST FOURIER TRANSFORM 492

Figure 15.6 Implementation of PosConv, specified in Figure 15.5

PosConv(p[0..n − 1], q[0..n − 1], k)

lgInv ← 2k − lg n
ptrans ← ModFft(p, k, false)
qtrans ← ModFft(q, k, false)
rtrans ← new Array[0..m − 1]
for i ← 0 to n − 1
if ptrans[i].NumBits() > k
rtrans[i] ← ToRing(qtrans[i].Shift(k), k)
else if qtrans[i].NumBits() > k
rtrans[i] ← ToRing[i](ptrans[i].Shift(k), k)
else
rtrans[i] ← ModMult(ptrans[i], qtrans[i], k)
r ← ModFft(rtrans, k, true)
for i ← 0 to n − 1
r[i] ← ToRing(r[i].Shift(lgInv), k)
return r

Precondition: p[0..n − 1] is an array of BigNums, each no larger than 2k ,

n and k are Nats containing powers of 2 such that n ≤ 2k, and inv is a
Bool.
Postcondition: Returns the DFT of p over hZm , +, ·i with respect to 2−2k/n
if inv = true, or with respect to 22k/n otherwise.
ModFft(p[0..n − 1], k, inv)
CHAPTER 15. * THE FAST FOURIER TRANSFORM 493

Figure 15.7 The Fast Fourier Transform algorithm over a modular ring

ModFft(p[0..n − 1], k, inv)

m ← one.Shift(k).Add(one); d ← new Array[0..n − 1]; mid ← n/2
if n = 1
d[0] ← p[0]
else
p′ ← new Array[0..mid − 1]; p′′ ← new Array[0..mid − 1]
for i ← 0 to mid − 1
p′′ [i] ← p[2i]; p′ [i] ← p[2i + 1]
d′ ← ModFft(p′ , k, inv)
d′′ ← ModFft(p′′ , k, inv)
for i ← 0 to mid − 1
if inv
oddVal ← ToRing(d′ [i].Shift(2k − 2ki/n), k)
else
oddVal ← ToRing(d′ [i].Shift(2ki/n), k)
d[i] ← ToRing(d′′ [i].Add(oddVal), k)
d[i + mid] ← ToRing(d′′ [i].Add(m.Subtract(oddVal)), k)
return d

we first subtract ω i d′ [i] from m, then add the result, mod m, to d′′ [i]. In
order to compute m, we assume the existence of a constant one, which refers
to a BigNum representing 1.
Let us now turn to the implementation of ToRing, specified in Figure
15.4. A straightforward way of computing x mod m is to divide x by m
using long division, and return the remainder. Fortunately, the form of m
makes this long division easy. Suppose we break m and x into k-bit digits.
Then the representation of m in this radix is 11.
In order to see how each step of the long division can proceed, suppose
x = a2k + b, where b < 2k and a < m. We first approximate the quotient as
a. If a ≤ b, the quotient is, in fact a, and the remainder is b − a. If a > b,
CHAPTER 15. * THE FAST FOURIER TRANSFORM 494

Figure 15.8 Algorithm for computing x mod (2k + 1)

ToRing(x, k)
numDig ← ⌈x.NumBits()/k⌉; m ← one.Shift(k).Add(one)
rem ← x.GetBits(k(numDig − 1), k)
// Invariant:
// rem = x.GetBits((i + 1)k, x.NumBits() − (i + 1)k) mod m
for i ← numDig − 2 to 0 by −1
next ← x.GetBits(ik, k)
if rem.CompareTo(next) > 0
next ← next.Add(m)
rem ← next.Subtract(rem)
return rem

we try a − 1 as the quotient. Then because a < m, a ≤ 2k , so that

(a − 1)(2k + 1) = a2k + a − 2k − 1
≤ a2k − 1
≤ a2k + b.

Then a − 1 is the quotient, and the remainder is

a2k + b − (a − 1)(2k + 1) = b + 2k − a + 1
= b + m − a.

We can therefore compute x mod m using only addition and subtraction

of BigNums, as shown in Figure 15.8. We assume the existence of a constant
one referring to a BigNum representing 1.
Finally, we need to implement Eval, specified in Figure 15.4. A straight-
forward implementation by adding and shifting would be too inefficient, be-
cause numbers with up to k bits would need to be copied each iteration.
Instead, we should try to generate the result one l-bit block at a time. We
can store the resulting bits in an array, then convert the result to a BigNum.
A difficulty with this approach is that some elements of the input array may
represent negative values. It therefore makes sense to accumulate the pos-
itive terms in one array and the negative terms in another. We can then
CHAPTER 15. * THE FAST FOURIER TRANSFORM 495

Figure 15.9 Implementation of Eval, specified in Figure 15.4

Eval(v[0..n − 1], k, l)
m ← one.Shift(k).Add(one); m′ ← one.Shift(4l).Add(one)
half ← m′ .Shift(−1)
pos ← new Array[0..nl − 1]; neg ← new Array[0..nl − 1]
posCarry ← zero; negCarry ← zero
for j ← 0 to n − 1
if v[j].CompareTo(half ) > 0
negCarry ← negCarry.Add(m′ .Subtract(v[j]))
else
posCarry ← posCarry.Add(v[j])
negBits ← negCarry.GetBits(0, l); negCarry ← negCarry.Shift(l)
posBits ← posCarry.GetBits(0, l); posCarry ← posCarry.Shift(l)
Copy(negBits[0..l − 1], neg[jl..j(l + 1) − 1])
Copy(posBits[0..l − 1], pos[jl..j(l + 1) − 1]);
posNum ← posCarry.Shift(nl).Add(new BigNum(pos))
negNum ← negCarry.Shift(nl).Add(new BigNum(neg))
return ToRing(posNum.Add(m.Subtract(ToRing(negNum, k))), k)

combine the two arrays into a single BigNum. The algorithm is shown in
Figure 15.9.
To analyze the running time of our multiplication algorithm, we begin by
analyzing ToRing. From the loop invariant, the value of rem never exceeds
2k ; hence, the value of next never exceeds 2k+1 . Thus, the body of the loop
clearly runs in Θ(k) time. The number of iterations is ⌈n/k⌉ − 1, where n
is the number of bits in x. The loop therefore runs in Θ(n) time, provided
n > k. Because the initialization runs in Θ(k) time, the entire algorithm
runs in Θ(max(n, k)) time.
In order to analyze ModFft, let us first ignore the computations whose
running times depend on k, namely, the calculation of m and the calls to
ToRing, Add, and Subtract. Thus, if we let n = 2N , the running time
of the remaining code is in Θ(N 2N ). Specifically, we can conclude that the
total number of iterations of each of the for loops is in Θ(N 2N ).
Now let k = 2K , and let us analyze the running time of a single iteration
of the second for loop, including the calls to ToRing, Add, and Subtract.
CHAPTER 15. * THE FAST FOURIER TRANSFORM 496

We first observe that for all i, d′ [i] ≤ 2k and d′′ [i] ≤ 2k . Because the number
of bits added by the Shift is at most 2k, the Shift therefore runs in O(2K )
time. Because the result of the Shift has O(2K ) bits, the call to ToRing
runs in Θ(2K ) time. Likewise, it is easily seen that the remaining operations
run in O(2K ) time as well. A single iteration of the second for loop therefore
runs in Θ(2K ) time. We conclude that ModFft runs in Θ(N 2N +K ) time.
It is easily seen that the running time of PosConv, excluding the calls
to ModFft and ModMult, is in Θ(2N +K ). Because the first two argu-
ments to ModMult must have at most 2K bits, we can describe the running
time of ModMult in terms of K. In particular, let f (K) denote the worst-
case running time of ModMult, assuming it is implemented using Mod-
MultFft. Because ModMult is called no more than 2K times, the running
time of PosConv is bounded above by a function in O(N 2N +K ) + 2N f (K).
Likewise, it is easily seen that NegConv has the same asymptotic running
time.
In order to analyze Eval, let l = 2L . It is easily seen that exclud-
ing the return statement, this function runs in Θ(max(2K , 2N +L )) time.
Furthermore, it is not hard to see that when the return statement is ex-
ecuted, posNum and negNum each contain at most n(3 + l) bits; hence,
ToRing(negNum, k) runs in Θ(max(2K , 2N +L )) time. Likewise, it is not
hard to see that the entire return statement runs in Θ(max(2K , 2N +L ))
time.
We can now obtain an asymptotic recurrence for f (K), the worst-case
running time of ModMult. In what follows, we assume K ≥ 4. We first
observe that if K is even, then b = 2(K/2)+1 and l = 2(K/2)−1 . Likewise, if
K is odd, b = 2(K+1)/2 , and l = 2(K−1)/2 . We can combine these two cases
by saying that b = 2⌈(K+1)/2⌉ and l = 2⌊(K−1)/2⌋ . The running time of the
for loop is therefore in

Θ(bl) = Θ(2⌈(K+1)/2⌉+⌊(K−1)/2⌋ )
= Θ(2K ).

In the call to NegConv, the number of elements in the arrays is b =

2⌈(K+1)/2⌉ ,and the third parameter is 4l = 2⌊(K−1)/2⌋+2 . Applying the
our analysis of NegConv, we see that the running time of this call is in
O(K2K ) + 2⌈(K+1)/2⌉ f (⌊(K − 1)/2⌋ + 2). Likewise, the call to Eval runs in
Θ(2K ) time. We therefore have

f (K) ∈ O(K2K ) + 2⌈(K+1)/2⌉ f (⌊(K − 1)/2⌋ + 2). (15.5)

In order to simplify the above recurrence, let g(K) = f (K + 3)/2K for

CHAPTER 15. * THE FAST FOURIER TRANSFORM 497

K ≥ 1. Then
O((K + 3)2K+3 ) + 2⌈(K+4)/2⌉ f (⌊(K + 2)/2⌋ + 2)
g(K) ∈
2K
4f (⌊K/2⌋ + 3)
= O(K) +
2⌊K/2⌋
= O(K) + 4g(⌊K/2⌋). (15.6)
Applying Theorem 3.32, we have g(K) ∈ O(K 2 ). Thus, for K ≥ 4,
f (K) = 2K−4 g(K − 3)
∈ 2K−4 O(K 2 )
⊆ O(2K K 2 ).
The running time of ModMult is therefore in O(2K K 2 ). We can therefore
conclude that the running time of MultFft is in
O(2⌈lg n⌉ ⌈lg n⌉2 ) = O(n lg2 n),
where n is the number of bits in the product.
The above analysis is almost sufficient to show that the running time
of MultFft is in Θ(n lg2 n). Specifically, we only need to show that there
are inputs for each sufficiently large n such that the call to ModMult is
made on each iteration of the first loop in PosConv. Unfortunately, such a
proof would be quite difficult. On the other hand, it seems unlikely that our
upper bound on this algorithm’s worst-case running time can be improved.

15.4 The Schönhage-Strassen Algorithm

In this section, we will show how to improve the multiplication algorithm of
the preceding section to achieve a running time in O(n lg n lg lg n). In order
to see what we need to improve, consider recurrence (15.5). Specifically,
consider the 2⌈(K+1)/2⌉ f (⌊(K − 1)/2⌋ + 2) term. The coefficient is b, the
number of calls made to ModMult in PosConv, and the argument to f
is lg(4l), the size of each recursive call. If we add lg b to the size of the
recursive calls, we get K + 2, where the 2 is the lg of the multiplier for l
that we use to define the ring in which the FFT will be computed.
Let us suppose, more generally, that b = 2⌈(K+c)/2⌉ and that the size of
the recursive call is ⌊(K − c)/2⌋ + d, where c and d are natural numbers.
Then the recurrence becomes
f (K) ∈ O(K2K ) + 2⌈(K+c)/2⌉ f (⌊(K − c)/2⌋ + d). (15.7)
CHAPTER 15. * THE FAST FOURIER TRANSFORM 498

Now letting g(K) = f (K − c + 2d)/2K , we have

f (K − c + 2d)
g(K) =
2K
O((K − c + 2d)2K−c+2d ) + 2⌈(K+2d)/2⌉ f (⌊(K − 2c + 2d)/2⌋ + d)
∈
2K
d
2 f (⌊K/2⌋ − c + 2d)
= O(K) +
2⌊K/2⌋
d
= O(K) + 2 g(⌊K/2⌋).

By Theorem 3.32, if d > 1, g(K) ∈ O(K d ), as for recurrence (15.6);

however, if d = 1, then g(K) ∈ O(K lg K). It then follows that f (K) ∈
O(2K K lg K), and the running time of the resulting multiplication algorithm
would be in O(n lg n lg lg n). Thus, in order to improve the running time of
ModMult, it suffices to reduce the size of the ring we use from 24l + 1 =
2
22 l + 1 to 22l + 1.
The difficulty with such an approach is that we have already shown that
⌈lg(b22l+1 −1)⌉ bits are required so that the elements of the negative wrapped
convolution over the given ring uniquely determine the negative wrapped
convolution over the integers. We need an additional result result that will
allow us to extract the elements of the negative wrapped convolution over
the integers from their values over a modular ring. This result is the Chinese
Remainder Theorem.

Theorem 15.17 (Chinese Remainder Theorem) Let a1 , a2 , m1 , and m2

be natural numbers such that a1 < m1 , a2 < m2 , where m1 and m2 are
relatively prime. Then there is a unique natural number i < m1 m2 such
that i mod m1 = a1 and i mod m2 = a2 .
Before we prove this theorem, let’s see why it might useful. We need
to compute the negative wrapped convolution of two vectors u and v, each
of size b and consisting of natural numbers less than 2l . Let wj denote the
jth component of the negative wrapped convolution. As we have already
shown, −b22l < wj < b22l . Suppose we were to compute the negative
wrapped convolution over two separate rings hZmi , +, ·i, where m1 = 22l + 1
and m2 = 2b, as shown in Figure 15.10. (As we will see, it is possible to
compute the second convolution with relatively little overhead.) Then the
results of these convolutions give us

conv[j] = wj mod (22l + 1) (15.8)

CHAPTER 15. * THE FAST FOURIER TRANSFORM 499

Figure 15.10 Implementation of ModMult, specified in Figure 15.3, using

two negative wrapped convolutions

ModMultSS(u, v, k)
if k < 8
return ToRing(MultiplyAdHoc(u, v), k)
else
if (lg k)√mod 2 =√0
b ← k; l ← k
else √ p
b ← 2k; l ← k/2
uarray ← new Array[0..b − 1]; varray ← new Array[0..b − 1]
uarray ′ ← new Array[0..b − 1]; varray ′ ← new Array[0..b − 1]
for j ← 0 to b − 1
uarray[j] ← new BigNum(u.GetBits(jl, l))
varray[j] ← new BigNum(v.GetBits(jl, l))
uarray ′ [j] ← new BigNum(u.GetBits(jl, lg b + 1))
varray ′ [j] ← new BigNum(v.GetBits(jl, lg b + 1))
conv ← NegConv(uarray, varray, 2l)
conv ′ ← NegConvSS(uarray ′ , varray ′ , lg b + 1)
return EvalSS(conv, conv ′ , k, l)

Precondition: p[0..n−1] and q[0..n−1] are arrays of BigNums each having

at most k bits, and k and n are Nats such that n is a power of 2.
Postcondition: Returns a BigNum representing the negative wrapped
convolution of p and q over the ring hZm , +, ·i, where m = 2k .
NegConvSS(p[0..n − 1], q[0..n − 1], k)
Precondition: u[0..n − 1] is an array of BigNums no larger than 22l ,
v[0..n − 1] is an array of BigNums less than 2n, and n, k, and l are Nats
containing powers of 2.
Postcondition: Returns a BigNum representing
 
n−1
X
 wj 2jl  mod (2k + 1),
j=0

where wj mod (22l + 1) = u[j], wj mod 2n = v[j], and −n(22l + 1) ≤ wj <

n(22l + 1).
EvalSS(u[0..n − 1], v[0..n − 1], k, l)
CHAPTER 15. * THE FAST FOURIER TRANSFORM 500

and
conv ′ [j] = wj mod (2b) (15.9)
for 0 ≤ j < b.
Because 2b is a power of 2 and 22l + 1 is odd, they are relatively prime.
Theorem 15.17 therefore guarantees that if wj ≥ 0, then it is the only natural
number less than 2b(22l + 1) that satisfies (15.8) and (15.9). Furthermore,
it is not hard to see that wj + 2b(22l + 1) also satisfies these constraints.
Thus, Theorem 15.17 guarantees that if wj < 0, then wj + 2b(22l + 1) is the
only natural number less than 2b(22l + 1) that satisfies these constraints.
The proof of Theorem 15.17 will be constructive, so that we will be able to
compute the value that it guarantees. Finally, because wj < b(22l + 1) <
wj +2b(22l +1), we can determine whether the value guaranteed by Theorem
15.17 is wj or wj + 2b(22l + 1).
In order to prove Theorem 15.17, we need the following lemma.

Lemma 15.18 Let a be an integer, and let b and m be positive integers.

Then (a mod bm) mod m = a mod m.

Proof: Let r1 = a mod bm, so that for some integer p, bmp + r1 = a. Let
r2 = a mod m, so that for some integer q,

mq + r2 = a
= r1 + bmp
m(q − bp) = r1 − r2 .

Because r1 and r2 differ by a multiple of m, and because 0 ≤ r2 < m,

r1 mod m = r2 . Thus, (a mod bm) mod m = a mod m.

Proof of Theorem 15.17: Let f : Zm1 m2 → Zm1 × Zm2 be defined so

that f (i) = (i mod m1 , i mod m2 ). We will show that f is a one-to-one and
onto function.
In order to show that f is onto, let a1 ∈ Zm1 and a2 ∈ Zm2 . From
Theorem 7.4,
m1 x mod m2 = 1
has a natural number solution x = c. Let

i = (m1 c((a2 − a1 ) mod m2 ) + a1 ) mod m1 m2 . (15.10)

CHAPTER 15. * THE FAST FOURIER TRANSFORM 501

Clearly, 0 ≤ i < m1 m2 . Because m1 c((a2 − a1 ) mod m2 ) is a multiple of m1 ,

from Lemma 15.18, i mod m1 = a1 . Also, from Lemma 15.18, we have

i mod m2 = (m1 c((a2 − a1 ) mod m2 ) + a1 ) mod m2

= ((m1 c mod m2 )((a2 − a1 ) mod m2 ) + a1 ) mod m2
= a2 mod m2
= a2 .

Therefore, f (i) = (a1 , a2 ). Because the choice of a1 and a2 was arbitrary,

we conclude that f is onto.
Because Zm1 m2 and Zm1 × Zm2 are finite sets with the same number of
elements, and f is a mapping from Zm1 m2 onto Zm1 × Zm2 , it follows that
f is one-to-one.

Let us now consider how to implement EvalSS. Let m1 = 22l + 1 and

m2 = 2b. In order to apply (15.10), we need to have a value c such that
m1 c mod m2 = 1. Because b ≤ 2l is a power of 2, (22l + 1) mod 2b = 1.
We can therefore use c = 1. Furthermore, because 0 ≤ a1 < m1 , 0 ≤
m1 ((a2 − a1 ) mod m2 ) + a1 < m1 m2 . The value guaranteed by Theorem
15.17 is therefore

(22l + 1)((v[j] − u[j]) mod 2b) + u[j].

We can multiply by 22l + 1 using a bit shift and an addition. We can then
determine wj by comparing the above value with b(22l+1 ) and subtracting
2b(22l + 1) if necessary. The algorithm is shown in Figure 15.11.
In order to implement NegConvSS, we must be able to compute a
negative wrapped convolution over a ring hZm , +, ·i, where m is a power
of 2. However, because the values of the vectors are much smaller than
those used in the other convolution, we don’t need to be quite as careful
regarding the efficiency of this algorithm. Specifically, we don’t need to use
the FFT. Instead, we can first compute a non-wrapped convolution mod 2k .
Let us refer to this convolution as conv[0..2b − 1]. Element j of the negative
wrapped convolution is then (conv[j] − conv[n + j]) mod 2k . The algorithm
is shown in Figure 15.12.
Recall that PolyMult (Figure 10.1 on page 335) computes a non-
wrapped convolution of two vectors over hZ, +, ·i. We can therefore modify
this algorithm to operate on BigNums such that all operations are mod
2k . In order for the resulting algorithm to satisfy the specification of Non-
WrappedConv, we would also need to modify it to return an array whose
CHAPTER 15. * THE FAST FOURIER TRANSFORM 502

Figure 15.11 Implementation of EvalSS, specified in Figure 15.10

EvalSS(u[0..n − 1], v[0..n − 1], k, l)

m ← one.Shift(2l).Add(one)
m′ ← one.Shift(lg n + 1)
half ← m.Shift(lg n); full ← half .Shift(1)
pos ← new Array[0..nl − 1]; neg ← new Array[0..nl − 1]
posCarry ← zero; negCarry ← zero
for j ← 0 to n − 1
if v[j] ≥ u[j]
diff ← v[j].Subtract(u[j])
else
t ← u[j].Subtract(v[j]).GetBits(0, lg n + 1)
diff ← m′ .Subtract(new BigNum(t))
w ← diff .Shift(2l).Add(diff ).Add(u[j])
if w.CompareTo(half ) > 0
negCarry ← negCarry.Add(full.Subtract(w))
else
posCarry ← posCarry.Add(w)
negBits ← negCarry.GetBits(0, l); negCarry ← negCarry.Shift(l)
posBits ← posCarry.GetBits(0, l); posCarry ← posCarry.Shift(l)
Copy(negBits[0..l − 1], neg[jl..j(l + 1) − 1])
Copy(posBits[0..l − 1], pos[jl..j(l + 1) − 1])
posNum ← posCarry.Shift(nl).Add(new BigNum(pos))
negNum ← negCarry.Shift(nl).Add(new BigNum(neg))
M ← one.Shift(k).Add(1)
return ToRing(posNum.Add(M.Subtract(ToRing(negNum, k))), k)
CHAPTER 15. * THE FAST FOURIER TRANSFORM 503

Figure 15.12 Implementation of NegConvSS, specified in Figure 15.10

NegConvSS(p[0..n − 1], q[0..n − 1], k)

negConv ← new Array[0..n − 1]; m ← one.Shift(k)
conv ← NonWrappedConv(p, q, k)
for j ← 0 to n − 1
if conv[j].CompareTo(conv[n + j])
negConv[j] ← conv[j].Subtract(conv[n + j])
else
negConv[j] ← conv[j].Add(m).Subtract(conv[n + j])
return negConv

Precondition: p[0..n−1] and q[0..n−1] are arrays of BigNums each having

at most k bits, and k is a Nat.
Postcondition: Returns an array r[0..2n − 1] of BigNums giving the non-
wrapped convolution of p and q over the ring hZm , +, ·i, where m = 2k .
NonWrappedConv(p[0..n − 1], q[0..n − 1], k)

size is larger by one element, whose value will be 0. We leave the details as
an exercise.
In order to analyze the Schönhage-Strassen algorithm, which is simply
the Multiply algorithm of Figure 15.3 with ModMult implemented using
ModMultSS, we first observe that the analysis of EvalSS is similar to
the analysis of Eval in the previous section. Hence, its running time is in
Θ(max(2K , 2N +L )), where k = 2K , n = 2N , and l = 2L . Because Poly-
Mult runs in Θ(nlg 3 ) time, where n is the degree of the product, Non-
WrappedConv can be implemented to run in O(nlg 3 M (k)) time, where
M (k) is the time needed to multiply two k-bit BigNums mod 2k . Because
M (k) must be in Ω(k), NegConvSS then runs in O(nlg 3 M (k)) time.
To analyze ModMultSS, we first recall that the running time of Neg-
Conv is in O(N 2N +K ) + 2N f (K), where f (K) denotes the worst-case run-
ning time of ModMult; here, we will assume that ModMult is imple-
mented with ModMultSS. If we now let 2K be the value of k in the call to
ModMultSS, then the call to NegConv runs in O(K2K )+2⌈K/2⌉ f (⌊K/2⌋+
1), and NegConvSS runs in O(2⌈K/2⌉ lg 3 M (K)) time. Hence, even if M (K)
is in Θ(K 2 ), the running time for these two calls together is in O(K2K ) +
2⌈K/2⌉ f (⌊K/2⌋ + 1). Because the call to EvalSS runs in Θ(2K ) time, the
CHAPTER 15. * THE FAST FOURIER TRANSFORM 504

total running time of ModMultSS is easily seen to be given by the recur-

rence
f (K) ∈ O(K2K ) + 2⌈K/2⌉ f (⌊K/2⌋ + 1),
when K ≥ 3.
The above recurrence fits the form of (15.7) with d = 1; hence, as we
showed at the beginning of this section, the running time of the Schönhage-
Strassen algorithm is in O(n lg n lg lg n), where n is the number of bits in
the product.

15.5 Summary
The Fast Fourier Transform is an efficient algorithm for computing a con-
volution, a problem which arises in a variety of applications. For numerical
applications, applying the FFT over hC, +, ·i is appropriate; however, for
number-theoretic applications like arbitrary-precision integer multiplication,
other algebraic structures are more appropriate. The algorithm extends to
any commutative ring containing a principal nth root of unity, and over
which n has a multiplicative inverse, where n is a power of 2 giving the
number of elements in the vectors.
Some rings that are particularly useful for number-theoretic applications
are rings of the form hZm , +, ·i, where m is of the form 2k +1. The properties
of these rings contribute in several ways to the efficiency of the Schönhage-
Strassen integer multiplication algorithm. First, we can compute n mod
(2k +1) efficiently. Second, the principal nth roots of unity in these rings are
powers of 2, so that we can use bit shifting to multiply by these roots. Third,
when n is a power of 2, it has a multiplicative inverse that is also a power of
2. Fourth, we can compute a product in this ring with a negative wrapped
convolution of vectors with half as many elements as would be needed to
compute a non-wrapped convolution. Finally, because any power of 2 is
relatively prime to 2k + 1, we can reduce by half the number of bits we use
in computing the negative wrapped convolution if we instead perform some
computation on a few bits of each value and apply the Chinese Remainder
Theorem.

15.6 Exercises
Exercise 15.1 Prove Theorem 15.14. [Hint: Use induction on either m or
n.]
CHAPTER 15. * THE FAST FOURIER TRANSFORM 505

Exercise 15.2 Suppose that in multiplying two BigNums mod 2k − 1,

where k is a power of 2, instead of making b and 4l as nearly equal as
possible (as in Section 15.3), we were to make b as small as possible. Ana-
lyze the running time of the algorithm that results if we set b to 8 and l to
k/8.

Exercise 15.3

a. Prove Theorem 15.17 by showing that for any a1 ∈ Zm1 and any a2 ∈
Zm2 , if i = (m2 c2 a1 + m1 c1 a2 ) mod m1 m2 , where (m1 c1 ) mod m2 = 1
and (m2 c2 ) mod m1 = 1, then i mod m1 = a1 and i mod m2 = a2 .

* b. Extend the above idea to prove the following. Let m1 , . . . , mn be

positive integers that are all relatively prime to each other, and let
n
Y
M= mj .
j=1

Then for natural numbers a1 , . . . , an such that each aj < mj , there is a

unique natural number i < M such that for 1 ≤ j ≤ n, i mod mj = aj .

Exercise 15.4 Modify PolyMult (Figure 10.1 on page 335) to implement

NonWrappedConv, specified in Figure 15.12. Show that the algorithm
runs in O(nlg 3 M (k)) time, where M (k) is the time required to multiply to
k-bit BigNums.

* Exercise 15.5 For c ∈ C and p = hp0 , . . . , pn−1 i ∈ Cn , the chirp trans-

form of p with respect to c is the vector q ∈ Cn such that for 0 ≤ i < n,
n−1
X
qj = pi cij .
i=0

Thus, if c is a principal nth root of unity, then the chirp transform with
respect to c is a DFT. Show how to reduce the problem of computing a
chirp transform for arbitrary c ∈ C to the problem of computing a convo-
lution. Using this reduction, give an O(n lg n) algorithm for evaluating a
chirp transform.

Exercise 15.6 A Toeplitz matrix is an n×n array A such that for 1 ≤ i < n
and 1 ≤ j < n, Aij = Ai−1,j−1 . Thus, we can describe a Toeplitz matrix
by giving only its first row and its first column. Give an algorithm for
CHAPTER 15. * THE FAST FOURIER TRANSFORM 506

multiplying an n × n Toeplitz matrix over C by an n-element vector over

C. You may choose an appropriate representation for the Toeplitz matrix.
Your algorithm should run in O(n lg n) time, assuming each operation on
complex numbers can be performed in O(1) time.

* Exercise 15.7 Let

n−1
X
p(x) = ai xi
i=0

be a polynomial of degree strictly less than n, where each ai ∈ R, and let

x0 ∈ R. Give an algorithm for computing all of the derivatives of p(x) at
x0 (i.e., your algorithm should find, for 0 ≤ j < n, the jth derivative of
p(x) at x0 ). Your algorithm should run in O(n lg n) time, assuming that all
operations on complex numbers run in O(1) time. [Hint: Define q(x) =
p(x0 + x), and find all of the derivatives of q(x) at 0. You will probably find
the Binomial Theorem ((6.15 on page 241) helpful.]

15.7 Chapter Notes

Heideman, Johnson, and Burrus [58] credit Gauss with the discovery of the
Fast Fourier Transform in 1805. Its importance to computation was shown
by Cooley and Tukey [24]. The multiplication algorithm of Section 15.4 is
due to Schönhage and Strassen [96].
Though we have referred to Theorem 15.17 as the Chinese Remainder
Theorem, it is usually stated in the more general form suggested by Ex-
ercise 15.3. The process of solving so-called simultaneous congruences in
this way dates back to the third or fourth century AD, when the Chinese
mathematician Sun Zi (or Sun Tsŭ) showed how to solve a specific instance
of simultaneous congruences. The technique was published as a general
theorem by Qin Jiushao (or Chhin Chiu-Shao) in 1247.
Part V

Intractable Problems

507
Chapter 16

N P-Completeness

Up to now, we have focused on developing efficient algorithms for solving

problems. The word “efficient” is somewhat subjective, and the degree of
efficiency has varied depending on the problem. Still, in each case, we have
shown a running time that was no worse than a low-order polynomial in
some natural description of the problem size.
It is possible to prove, however, that some problems cannot be solved
by any algorithm with polynomial running time. In fact, it is possible to
prove that some problems cannot be solved by any algorithm at all. We
will not be examining any of these problems, but in this chapter and the
next, we will take a look at a very interesting class of problems for which no
efficient algorithms are known. Part of the reason that this class of problems
is interesting is that if a polynomial-time algorithm were to be found for any
one of these problems, then we could derive polynomial-time algorithms for
all of the problems in this class. Furthermore, no one to date has given a
convincing proof that there are no such algorithms. At the heart of these
issues is the most famous open question in computational complexity theory.

16.1 Boolean Satisfiability

Suppose we are given an expression F containing boolean variables and the
following operators:

• ¬: logical negation;

• ∨: logical or; and

• ∧: logical and.

508
CHAPTER 16. N P-COMPLETENESS 509

There are two questions we might ask regarding F:

• Is F valid ? That is, does F evaluate to true for every possible assign-
ment of true or false to the variables in F?

• Is F satisfiable? That is, does there exist some assignment of true or

false to the variables in F so that F evaluates to true?

For example, let F = ¬x ∨ (y ∧ x). This expression is not valid, because

setting x to true and y to false yields

¬true ∨ (false ∧ true) = false ∨ false

= false.

However, F is satisfiable — in fact, any other assignment of values to x and

y makes F true.
Note that it follows immediately from the definitions of validity and satis-
fiability that for any expression F, F is valid iff ¬F is unsatisfiable. Because
of this duality, we will focus on only one of these problems, the satisfiability
problem. We would like to find a satisfiability algorithm whose worst-case
running time is bounded by some polynomial in the size of the given ex-
pression, where the size is defined to be the total number of occurrences of
variables and operators. However, as of this writing, no such algorithm has
been found. Indeed, as we will see shortly, there is good reason to believe
that no such algorithm is possible. On the other hand, there currently exists
no proof that such an algorithm is impossible.
Before we look at the satisfiability problem in more detail, let us first
consider a simpler problem, that of evaluating a boolean expression F, given
boolean values for its variables. We first observe that F must be of one of
the following forms:

• a single variable;

• the negation of an expression, i.e., ¬F1 ;

• the or of two expressions, i.e., F1 ∨ F2 ; or

• the and of two expressions, i.e., F1 ∧ F2 .

It is therefore convenient to represent the formula using a binary tree in

which the leaves represent variables and the internal nodes represent oper-
ators. For the ¬ operator, the right-hand child will always be empty. To
make this representation more concrete, we can use special constants not,
CHAPTER 16. N P-COMPLETENESS 510

Figure 16.1 Binary trees reresenting the formula ¬x ∨ (y ∧ x)

∨ or

¬ ∧ not and

x y x 1 2 1

and, and or to represent the three operators, and we can represent the vari-
ables using positive integers. In order to avoid unnecessary complications,
we will assume that if j represents a variable in the expression, then for
1 ≤ i ≤ j, i represents some variable in the expression (note, however, that
we cannot apply this assumption to arbitrary subtrees). For example, Fig-
ure 16.1 shows the tree representing the formula ¬x ∨ (y ∧ x), first in a more
abstract form, then in a more concrete form using 1 to represent x and 2
to represent y. Finally, we can represent an assignment of truth values to
the variables using an array A[1..n] of boolean values, where A[i] gives the
value assigned to the variable represented by i.
Given an expression tree for F and an array representing an assignment
of truth values, we can then evaluate F using BoolEval, shown in Figure
16.2. It is not hard to see that this algorithm runs in Θ(m) time, where
m is the number of operators in F (note that the number of leaves in the
expression tree can be no more than m + 1).
Returning to the satisfiability problem, we now point out some charac-
teristics that this problem has in common with the other problems we will
study in this chapter. First, it is a decision problem — the output is either
“yes” or “no”. Second, when the answer is “yes”, there is a relatively short
proof of this fact — an assignment of truth values that satisfies the expres-
sion. Third, given a proposed proof of satisfiability (i.e., some assignment
of truth values to the variables), we can efficiently verify whether it does,
in fact, prove the satisfiability of the expression. However, finding such a
proof, or proving that none exists, appears to be an expensive task in the
worst case (note that there are 2n possible assignments of truth values to n
variables).
CHAPTER 16. N P-COMPLETENESS 511

Figure 16.2 Algorithm for evaluating a boolean expression.

Precondition: F is a BinaryTreeNode referring to the root of a

nonempty boolean expression tree, and A[1..n] is an array of Bools.
Postcondition: Returns the value of F assuming that the variable repre-
sented by i has value A[i] if i ≤ n, or false if i > n.
BoolEval(F, A[1..n])
if F.LeftChild() = nil
if F.Root() ≤ n
return A[F.Root()]
else
return false
else if F.Data() = not
return not BoolEval(F.LeftChild(), A)
else
l ← BoolEval(F.LeftChild(), A)
r ← BoolEval(F.RightChild(), A)
if F.Root() = and
return l and r
else
return l or r

16.2 The Set P

We have suggested that the boolean satisfiability problem may not be effi-
ciently solvable. However, we have not yet formalized what this means. In
this section we will define a set of decision problems that we will consider
to be those that are efficiently decidable.
We will begin by adopting a couple of conventions. First, we will assume
that each problem instance is a bit string. Certainly, we can encode other
types of input data as bit strings, provided the set of all instances is count-
able. We do need to ensure, however, that this encoding is done in such a
way that the length of the encoding is not unnecessarily long. With this An artificially
long input would
convention, we can now express the running time of an algorithm in terms result in an
of the length of its input string. This gives us a uniform way of expressing artificially low
the running times of all algorithms for all problems. Second, we will view running time
when expressed
a decision problem as a subset of its instances. Specifically, let I denote in terms of the
length of the
input.
CHAPTER 16. N P-COMPLETENESS 512

the set of all bit strings that encode boolean expressions. We can then let
Sat denote the set of all expressions in I that are satisfiable. It will also be
convenient to use Sat to denote the problem itself. In general, given a set
of instances I, we will refer to any subset X ⊆ I as a decision problem over
I.
We must now address the question of how efficient is “efficient”. We
will somewhat arbitrarily delineate the “efficient” algorithms as those that
operate within a running time bounded by some polynomial in the length
of the input. This delineation, however, is not entirely satisfactory. On the
one hand, one could make a persuasive argument that an algorithm with a
running time in Θ(n1000 ) is not “efficient”. On the other hand, suppose some
algorithm has a running time in Θ(n⌈α(n)/4⌉ ), where α is as defined in Section
8.4. Because n⌈α(n)/4⌉ ≤ n for every positive n that can be coded in binary
within our universe, such an algorithm might reasonably be considered to
be “efficient”. However, n⌈α(n)/4⌉ is not bounded above by any polynomial.
The main reason we equate polynomial running time with efficiency
is that polynomials have several useful closure properties. Specifically, if
p1 (n) and p2 (n) are polynomials, then so are p1 (n) + p2 (n), p1 (n)p2 (n), and
p1 (p2 (n)). As we will see, these closure properties make the theory much
cleaner. Furthermore, if we can say that a particular decision problem can-
not be solved by any polynomial-time algorithm, then we can be fairly safe
in concluding that there is no algorithm that will terminate in a reasonable
amount of time on large inputs in the worst case.
Before we formalize this idea, we need to be careful about one aspect of
our running-time measures. Specifically, we have assumed in this text that
arithmetic operations can be performed in a single step. This assumption is
valid if we can reasonably expect the numbers to fit in a single machine word.
For larger values, we should use the BigNum type in order to get an ap-
propriate measure of the running time. Also note that all of the algorithms
in this text except those using real or complex numbers can be expressed
using only booleans and natural numbers as primitive types (i.e., those not
defined in terms of other variables). Furthermore, only the algorithms of
Section 15.1 require real or complex numbers — all other algorithms can be
restricted to rational numbers, which can be expressed as a pair of natural
numbers and a sign bit. Thus, it will make sense to stipulate that an “effi-
cient” algorithm contains only boolean or natural number variables, along
with other types built from these, and that each natural number variable
will contain only a polynomial number of bits.
We can now formalize our notion of “efficient”. We say that an algorithm
A is a polynomial-time algorithm if there is a polynomial p(n) such that
CHAPTER 16. N P-COMPLETENESS 513

• A always completes within p(n) steps, where n is the number of bits

in the input string; and

• all primitive variables used by A are either booleans or natural num-

bers whose values remain strictly less than 2p(n) .

We then define P to be the set of all decision problems X such that there
exists a deterministic (i.e., not randomized) polynomial-time algorithm A
deciding X. It is this set P that we will consider to be the set of efficiently
solvable decision problems.
The decision to define P using only deterministic algorithms is rather ar-
bitrary. Indeed, there is a branch of computational complexity theory that
focuses on efficient randomized algorithms. However, the study of deter-
ministic algorithms is more fundamental, and therefore is a more reasonable
starting point for us.

16.3 The Set N P

Sat is clearly a decision problem, but as we have suggested, it is currently
not known whether or not it belongs to P. In this section, we will define a
related set called N P that includes all of P, but also includes Sat, as well
as many other problems not known to be in P. Furthermore, N P will have
the property that if Sat is, in fact, in P, then P = N P.
In order to extend the definition of P to include problems like Sat, we
need to formalize the idea that each element of a decision problem X has a
short proof which can be efficiently checked. For the sake of concreteness,
let us assume that all proofs will be encoded as bit strings. We then denote
the set of all bit strings by B.
We now define N P to be the set of all decision problems X for which
there exist:

• a polynomial p(n); and

• a decision problem Y ⊆ I × B, where I is the set of instances for X;

such that

• Y ∈ P;

• for each x ∈ I, x ∈ X iff there is a proof φ ∈ B such that (x, φ) ∈ Y ;

and
CHAPTER 16. N P-COMPLETENESS 514

• for each x ∈ X, there is a proof φ ∈ B such that (x, φ) ∈ Y and

|φ| ≤ p(|x|). We use |x| to
denote the
From our earlier discussion, it follows that Sat ∈ N P. We can clearly length of the
encoding of x.
consider any array A[1..n] of boolean values to be a bit string, and vice versa.
We then let the decision problem Y be the problem solved by BoolEval.
Hence, Y ∈ P. We can then let p(n) = n. Then a given expression F
is satisfiable iff there is a proof φ such that (x, φ) ∈ Y . Because we have
assumed that whenever an integer j represents a variable in F, all positive
integers less than j also represent variables in F, it follows that if a proof φ
exists, there will always be one with length no more than |F|.
We therefore have an example of a problem in N P that may or may not
be in P. The following theorem gives the relationship between P and N P.

Theorem 16.1 P ⊆ N P.

Proof: Let X ∈ P, and let I be the set of instances of X. We then define

Y = X × B. Thus, Y is comprised of all pairs (x, φ) such that x ∈ X and
φ ∈ B. We can therefore decide whether (x, φ) ∈ I × B belongs to Y by
simply deciding whether x ∈ X. Because X ∈ P, it follows that Y ∈ P. Let
p(n) = 1. Then x ∈ X iff (x, 0) ∈ Y , and the length of 0 is 1. Therefore
X ∈ N P.

It is currently unknown whether the above containment is proper, or

whether P = N P. In fact the “P = N P” question is the most famous open
question in computational complexity theory. Most complexity theorists
believe that these sets are not equal. Though many of the reasons for this
belief are beyond the scope of this book, we will soon see one compelling
reason. For now, let us simply state that we take as a working hypothesis
that P =6 N P. Thus, if we can show that some particular statement implies
that P = N P, we take this as strong evidence that the statement is false.
In order to focus on the relationship between P and N P, it is useful to
identify problems that seem more likely to be in N P \ P. In some sense, we
want to identify the hardest problems in N P. We can do this using a refine-
ment of the notion of problem reduction. Specifically, let X and Y be two
decision problems whose instances comprise the sets I and J, respectively.
We say that X is polynomially many-one reducible to Y , written X ≤pm Y ,
if there is a function f : I → J such that

• for all x ∈ I, x ∈ X iff f (x) ∈ Y ; and

CHAPTER 16. N P-COMPLETENESS 515

• there is a deterministic polynomial-time algorithm for computing f .

Note that polynomial many-one reductions are transformations — given

an instance x of problem X, we can decide whether x ∈ X simply by comput-
ing f (x) and deciding whether f (x) ∈ Y . The notation may seem confusing
at first, because when we use the word “reduce”, we usually think of decreas-
ing the size. As a result, denoting a reduction from X to Y by X ≤pm Y
seems backwards. The proper way to understand the notation is to realize
that when there is a polynomial many-one reduction from X to Y , then in
some sense, X is no harder than Y . This idea is formalized by the following
theorem.

Theorem 16.2 If X ≤pm Y and Y ∈ P, then X ∈ P.

Proof: Let I and J be the sets of instances of X and Y , respectively, and

let f : I → J be the function computing the polynomial many-one reduction
from X to Y . Let p1 (n) be a polynomial bounding the running time and
the values of variables for some algorithm to compute f , and let p2 (n) be a
polynomial bounding the running time and the values of variables for some
algorithm to decide Y . We can then decide whether a given x ∈ I belongs to
X by first computing f (x), then deciding whether f (x) ∈ Y . Let A denote
this algorithm.
The time required to compute f (x) is no more than p1 (|x|). The time
required to decide whether f (x) ∈ Y is no more than p2 (|f (x)|). Because
f (x) is computed using at most p1 (|x|) steps, |f (x)| ≤ cp1 (|x|), where c
is some positive integer constant bounding the number of bits that can be
written in a single step. The total time required by A is therefore no more
than
p(|x|) = p1 (|x|) + p2 (cp1 (|x|)),
which is a polynomial in |x|. Furthermore, the values of the variables in A
do not exceed max(p1 (|x|), p2 (|x|)). Because p(n), p1 (n), and p2 (n) must
be nonnegative for all n ∈ N, all are bounded above by the polynomial
p(n) + p1 (n) + p2 (n). Therefore, X ∈ P.

Note that Theorem 16.2 does not say that if Y can be decided in O(f (n))
time, then X can be decided in O(f (n)) time. Indeed, in the proof of the
theorem, the bound on the time to decide X can be much larger than the
time to decide Y . Thus, if we interpret X ≤pm Y as indicating that X is no
harder than Y , we must understand “no harder than” in a very loose sense
— simply that if Y ∈ P, then X ∈ P.
CHAPTER 16. N P-COMPLETENESS 516

We will often utilize Theorem 16.2 in the following equivalent form.

Corollary 16.3 If X ≤pm Y and X 6∈ P, then Y 6∈ P.

Suppose we have some problem Y ∈ N P such that for every X ∈ N P,
X ≤pm Y . If P = 6 N P, then there is some X ∈ N P \ P. Because X ≤pm Y
and X 6∈ P, Corollary 16.3 tells us that Y 6∈ P. This fact motivates the
following definitions.

Definition 16.4 If Y is a decision problem such that for every X ∈ N P,

X ≤pm Y , then we say that Y is N P-hard.

Definition 16.5 If Y ∈ N P is N P-hard, then we say that Y is N P-

complete.
Suppose we have some N P-hard problem Y , and also suppose that P =6
N P. Then there is some X ∈ N P \ P. Because X ∈ N P and Y is N P-
hard, X ≤pm Y . Hence, from Corollary 16.3, Y 6∈ P. We therefore have the
following theorem and its corollary.

Theorem 16.6 If Y is N P-hard and P =

6 N P, then Y 6∈ P.

Corollary 16.7 If Y is N P-complete and P =

6 N P, then Y ∈ N P \ P.
It turns out that thousands of N P-complete problems in a wide variety
of problem domains have been identified. If we could find a polynomial-time
algorithm for any one of these problems, Corollary 16.7 would then imply
that P = N P. The fact that this has not been accomplished is one reason
to suspect that P =6 N P. Let us now identify an N P-complete problem.

Theorem 16.8 (Cook’s Theorem) Sat is N P-complete.

The idea of the proof of Cook’s Theorem is to give a method for con-
structing, from an arbitrary X ∈ N P, a polynomial-time algorithm that
takes as input an instance x of X and produces as output a boolean expres-
sion F such that F is satisfiable iff x ∈ X. In constructing this algorithm,
we can use the polynomial p(n) bounding the size of a proof φ and the al-
gorithm for deciding whether φ proves that x ∈ X. In order to complete
the construction, we must carefully define the computational model so that
the boolean formula can encode the algorithm. Due to the large amount of
work involved, we will delay the proof of Cook’s Theorem until Section 16.8.
CHAPTER 16. N P-COMPLETENESS 517

Fortunately, once we have one N P-complete problem, the task of show-

ing other problems to be N P-complete becomes much easier. The reason
for this is that polynomial many-one reducibility is transitive, as we show in
the following theorem. Its proof is similar to the proof of Theorem 16.2, and
is therefore left as an exercise. Its corollary then gives us a proof technique
for showing N P-hardness, provided we already have at least one N P-hard
problem.

Theorem 16.9 If X ≤pm Y and Y ≤pm Z, then X ≤pm Z.

Corollary 16.10 If X is N P-hard and X ≤pm Y , then Y is N P-hard.

Thus, to show a decision problem Y to be N P-hard, we need only to
show that X ≤pm Y for some N P-hard problem X. This is the technique
that we will use for all subsequent N P-hardness proofs. Note also that the
more problems we have shown to be N P-complete, the more tools we have
for showing additional problems to be N P-complete. For this reason, we
will devote the next few sections to identifying a variety of N P-complete
problems.

16.4 Restricted Satisfiability Problems

In order to illustrate the technique of proving N P-hardness using a poly-
nomial many-one reduction from a known N P-hard problem, we consider
in this section two special cases of boolean satisfiability. In the first special
case, the expression is in conjunctive normal form (CNF); i.e., the expression
is of the form
^n _ki
αij ,
i=1 j=1

where each αij is a literal — either a variable or the negation of a variable.

Let us refer to this problem as CSat. We will now show that CSat is N P-
complete. The proof of N P-hardness will consist of showing that Sat ≤pm
CSat.
It makes sense to represent the input for CSat using a data structure
that follows the form of a CNF formula more closely than does an expression
tree. Specifically, we represent a CNF formula as a ConsList of clauses,
each of which is a disjunction of literals. We then represent a clause as
a ConsList of literals. Finally, we represent each literal as an integer as
follows. For a non-negated variable, we simply use a positive integer, as in an
CHAPTER 16. N P-COMPLETENESS 518

expression tree. For a negated variable ¬x, we use −i, where i is the integer
representing the variable x. We will again assume that for an input formula
F, if j represents a variable in F, then for 1 ≤ i ≤ j, i represents a variable
in F. Again, this assumption will not apply to arbitrary sub-formulas.
One obvious way of reducing Sat to CSat is to convert a given boolean
expression to an equivalent expression in CNF. However, there are boolean
expressions for which the shortest equivalent CNF expression has size expo-
nential in the size of the original expression. As a result, any such conversion
algorithm must require at least exponential time in the worst case.
Fortunately, our reduction doesn’t need to construct an equivalent ex-
pression, but only one that is satisfiable iff the given expression is satisfiable.
In fact, the constructed expression isn’t even required to contain the same
variables. We will use this flexibility in designing our reduction.
For the first step of our reduction, we will construct an equivalent formula
in which negations are applied only to variables. Because of this restriction,
we can simplify our representation for this kind of expression by allowing
leaves to contain either positive or negative integers, as in our representation
of CNF formulas. Using this representation, we no longer need nodes repre-
senting the ¬ operation. We will refer to this representation as a normalized
expression tree.
Fortunately, there is a polynomial-time algorithm for normalizing a bool-
ean expression tree. The algorithm uses DeMorgan’s laws:

• ¬(x ∨ y) = ¬x ∧ ¬y; and

• ¬(x ∧ y) = ¬x ∨ ¬y.

The algorithm is shown in Figure 16.3. This algorithm solves a slightly more
general problem for which the input includes a boolean neg, which indicates
whether the normalized expression should be equivalent to F or ¬F. It is
easily seen that its running time is proportional to the number of nodes in
the tree, which is in O(m), where m is the number of operators in F.
As the second step in our reduction, we need to find the largest integer
used to represent a variable in a normalized expression tree. We need this
value in order to be able to introduce new variables. Such an algorithm is
shown in Figure 16.4. Clearly, its running time is in O(|F|).
As the third step in our reduction, we will construct from a normalized
expression tree F and a value larger than any integer representing a variable
in F, a CNF expression F ′ having the following properties:

P1 : F ′ contains all of the variables in F;

CHAPTER 16. N P-COMPLETENESS 519

Figure 16.3 Algorithm to normalize a boolean expression tree

Precondition: F is a BinaryTreeNode referring to the root of a boolean

expression tree, and neg is a Bool.
Postcondition: Returns a normalized expression tree F ′ such that if neg
is false, F ′ ≡ F, and if neg is true, F ′ ≡ ¬F.
Normalize(F, neg)
F ′ ← new BinaryTreeNode(); op ← F.Root()
if F.LeftChild() = nil
if neg
F ′ .SetRoot(−op)
else
F ′ .SetRoot(op)
else if op = not
F ′ ← Normalize(F.LeftChild(), not neg)
else
F ′ .SetLeftChild(Normalize(F.LeftChild(), neg))
F ′ .SetRightChild(Normalize(F.RightChild(), neg))
if (op = and and neg) or (op = or and not neg)
F ′ .SetRoot(or)
else
F ′ .SetRoot(and)
return F ′

P2 : for any satisfying assignment A for F, there is a satisfying assignment

A′ for F ′ in which all the variables in F have the same values as in A;
and

P3 : for any satisfying assignment A′ for F ′ , the assignment A for F in which

each variable in F is assigned its value from A′ satisfies F.

Thus, F ′ will be satisfiable iff F is satisfiable. We consider three cases.

Case 1: F is a literal. Then because F is in CNF, we let F ′ = F.

Case 2: F = F1 ∧ F2 . We then construct CNF formulas F1′ and F2′ from

F1 and F2 , respectively, such that properties P1 -P3 are satisfied. Then
CHAPTER 16. N P-COMPLETENESS 520

Figure 16.4 Algorithm to find the largest integer representing a variable

in a normalized expression tree

Precondition: F is a BinaryTreeNode representing a normalized ex-

pression tree.
Postcondition: Returns the largest absolute value of any integer in F.
MaxVar(F)
if F.Root() = and or F.Root() = or
l ← MaxVar(F.LeftChild())
r ← MaxVar(F.RightChild())
return Max(l, r)
else
return |F.Root()|

F ′ = F1′ ∧ F2′ is in CNF and clearly satisfies properties P1 -P3 with respect
to F.

Case 3: F = F1 ∨ F2 . We then construct CNF formulas F1′ and F2′ from

F1 and F2 , respectively, such that properties P1 -P3 are satisfied. Let u
be a variable that is contained in neither F1′ nor F2′ . We construct F1′′ by
including u in each clause of F ′ , and we construct F2′′ by including ¬u in
each clause of F2′ . We then let F ′ = F1′′ ∧ F2′′ . Clearly, F ′ is in CNF.
Furthermore, it is not hard to show that F ′ satisfies properties P1 -P3 with
respect to F.

The algorithm for constructing F ′ is shown in Figure 16.5. It uses a

data type MutableNat, which contains a single readable and writable
representation variable data, which is a Nat. It also uses the function
Append specified in Figure 4.15.
It is not hard to see that AddToClauses operates in O(n) time, where
n is the number of clauses in F. Furthermore, NormalizedToCnf only
constructs a new clause when processing a literal; hence, the number of
clauses in the CNF formula is no more than |F|. As suggested in Exercise
4.3, Append can be implemented to run in O(n) time, where n is the num-
ber of elements in its first argument. It follows that the time for a single
call to NormalizedToCnf, excluding recursive calls, runs in O(|F|) time.
CHAPTER 16. N P-COMPLETENESS 521

Figure 16.5 Algorithm for constructing a CNF formula from a normalized

expression tree

Precondition: F is a BinaryTreeNode referring to normalized boolean

expression tree, and m refers to a MutableNat larger than the absolute
value of any variable in F.
Postcondition: Returns a ConsList representing a CNF formula F ′ that
satisfies properties P1 -P3 with respect to F. m has a value larger than the
absolute value of any variable in the returned formula.
NormalizedToCnf(F, m)
root ← F.Root()
if root = and or root = or
l ← NormalizedToCnf(F.LeftChild(), m)
r ← NormalizedToCnf(F.RightChild(), m)
if root = or
x ← m.Data(); l ← AddToClauses(l, x)
r ← AddToClauses(r, −x); m.SetData(x + 1)
return Append(l, r)
else
c ← new ConsList(root, new ConsList())
return new ConsList(c, new ConsList())

Precondition: F is a (possibly empty) ConsList representing a CNF

formula, and α is a nonzero Int.
Postcondition: Returns a ConsList obtained by adding the literal α to
each clause in F.
AddToClauses(F, α)
if F.IsEmpty()
return F
else
h ← new ConsList(α, F.Head())
t ← AddToClauses(F.Tail(), α)
return new ConsList(h, t)
CHAPTER 16. N P-COMPLETENESS 522

Figure 16.6 The reduction from Sat to CSat

Precondition: F is BinaryTreeNode referring to a boolean expression

tree.
Postcondition: Returns a ConsList representing a CNF formula that is
satisfiable iff F is satisfiable.
SatToCSat(F)
F ′ ← Normalize(F, false)
m ← new MutableInt(); m.SetData(MaxVar(F ′ ) + 1)
return NormalizedToCnf(F ′ , m)

Because NormalizedToCnf is called once for every node in the expression

tree F, its overall running time is in O(|F|2 ).
The reduction is implemented in Figure 16.6. It clearly runs in O(|F|2 )
time, so that Sat ≤pm CSat. We can therefore show the following theorem.

Theorem 16.11 CSat is N P-complete.

Proof: By the above discussion, CSat is N P-hard. In order to show CSat

to be in N P, we use essentially the same reasoning as we did in showing Sat
to be in N P. The only difference is that we need an algorithm to evaluate
a CNF expression, rather than an expression tree. It is a straightforward
matter to adapt BoolEval (Figure 16.2) to evaluate a CNF expression
F in O(|F|) time — the details are left as an exercise. It follows that
CSat ∈ N P. CSat is therefore N P-complete.

As a second example, let us further restrict our inputs by limiting the

number of literals in each clause. We say that a CNF formula is in k-
conjunctive normal form (or k-CNF) if no clause contains more than k
literals. We then define k-Sat to be the problem of determining satisfiability
for a given k-CNF formula. Though we won’t show it here, it turns out that
2-Sat ∈ P. In what follows, we will show that 3-Sat is N P-complete.
The fact that 3-Sat ∈ N P follows immediately from the fact that
CSat ∈ N P, as 3-Sat is the same problem as CSat, only with more
restrictions placed on the input. Thus, the proof that CSat ∈ N P also
proves that 3-Sat ∈ N P.
In order to show that 3-Sat is N P-hard, we have two choices: we can
CHAPTER 16. N P-COMPLETENESS 523

reduce either Sat or CSat to 3-Sat. Reducing CSat to 3-Sat would

appear to be less work, as instances of CSat are already in CNF. All that
remains is to ensure that the number of literals in each clause is no more
than 3. We will therefore show that CSat ≤pm 3-Sat.
As in the previous reduction, we will not produce an equivalent formula.
Instead, we will again introduce new variables. In addition, we will break
up clauses that are too long into clauses containing only 3 literals.
Suppose our formula contains a clause C = α1 ∨ · · · ∨ αm , where m > 3.
We first introduce m − 3 new variables, u1 , . . . , um−3 . We then construct
the following clauses to replace C:

• α1 ∨ α2 ∨ u1 ;

• ¬ui ∨ αi+2 ∨ ui+1 for 1 ≤ i ≤ m − 4; and

• ¬um−3 ∨ αm−1 ∨ αm .

We first claim that any assignment of boolean values that satisfies C can
be extended to an assignment that satisfies each of the new clauses. To see
why, first observe that if C is satisfied, then αi must be true for some i. We
can then set u1 , . . . , ui−2 to true and ui−1 , . . . , um−3 to false. Then each of
the first i − 2 clauses is satisfied because u1 , . . . , ui−2 are true. The (i − 1)st
clause, ¬ui−2 ∨αi ∨ui−1 is satisfied because αi is true. Finally, the remaining
clauses are satisfied because ¬ui−1 , . . . , ¬um−3 are true.
We now claim that any assignment that satisfies the new clauses will also
satisfy C. Suppose to the contrary that all the new clauses are satisfied, but
that C is not satisfied — i.e., that α1 , . . . , αm are all false. Then in order for
the first clause to be satisfied, u1 must be true. Likewise, it is easily shown
by induction on i that each ui must be true. Then the last clause is not
satisfied — a contradiction.
If we apply the above transformation to each clause having more than
3 literals in a CNF formula F and retain those clauses with no more than
3 literals, then the resulting 3-CNF formula is satisfiable iff F is satisfiable.
Furthermore, it is not hard to implement this reduction in O(|F|) time —
the details are left as an exercise. Hence, CSat ≤pm 3-Sat. We therefore
conclude that 3-Sat is N P-complete.

16.5 Vertex Cover and Independent Set

So far, all of the problems that we have shown to be N P-complete are
satisfiability problems for various kinds of boolean formulas. As we have
CHAPTER 16. N P-COMPLETENESS 524

seen in earlier chapters, it is sometimes possible to reduce a problem A to

another problem B that at first looks nothing like problem A. By applying
this technique to polynomial many-one reducibility, we can identify N P-
complete problems in other domains.
For example, let us consider the vertex cover problem, which we will
denote VC. Let G = (V, E) be an undirected graph. A vertex cover for G is
a subset C ⊆ V such that for each edge {u, v} ∈ E, C ∩ {u, v} = 6 ∅; i.e., at
least one endpoint of each edge is contained in the vertex cover. The vertex
cover problem is to decide whether a given undirected graph has a vertex
cover of size k, where k is a given positive integer.
To show that VC ∈ N P, we will treat bit strings as arrays A[0..m − 1]
of boolean values. We can interpret an array A[0..m − 1] as describing a
subset S of the vertices {0, 1, . . . , n − 1} such that for 0 ≤ i < n, i ∈ S iff
i < m and A[i] is true. It is then an easy matter to check, in time linear in
the size of a graph G, whether A[0..m − 1] denotes a vertex cover of G with
size k — the details are left as an exercise. Therefore, VC ∈ N P.
In order to show that VC is N P-hard, we need to reduce one of the three
satisfiability problems to it. We will use 3-Sat because 3-CNF formulas have
a simpler structure than either CNF or arbitrary boolean formulas. Still, it
is not immediately clear how we can construct, from a given 3-CNF formula
F, an undirected graph G and a positive integer k such that G has a vertex
cover of size k iff F is satisfiable.
One rather simplistic approach is first to decide whether F is satisfiable,
then to construct one of two fixed graphs — one that has a vertex cover
of size 1, or one that does not. However, because 3-Sat is N P-hard, we
cannot decide in polynomial time whether F is satisfiable unless P = N P.
As a result, such an approach will probably never work.
Instead, we need to construct an instance of VC whose solution will give
us a solution to our original instance of 3-Sat. In order to do this, we should
try to see what the two problems have in common. A particularly useful
technique is to compare the proofs of membership in N P. Often we can
find a reduction that has the side-effect of transforming each proof φ ∈ B
for one problem to a proof φ′ for the other.
Let F be a given 3-CNF formula with n clauses, C1 , . . . , Cn . For 1 ≤ i ≤
n, let αi1 , αi2 , and αi3 be the three literals in clause Ci (if there are fewer
than three literals in Ci , we set αi3 and, if necessary, αi2 to equal αi1 ). A
proof for this instance of 3-Sat represents an assignment of boolean values
to the variables. A proof for an instance of VC represents a set of vertices.
Perhaps we can associate the selection of a boolean value to the selection of
one of two possible vertices. In particular, let us construct, for each variable
CHAPTER 16. N P-COMPLETENESS 525

Figure 16.7 The graph constructed from (x1 ∨ ¬x2 ∨ x3 ) ∧ (¬x1 ∨ x3 ) in

the reduction from 3-Sat to VC

c11 c13 c21 c23

c12 c22

x1 x2 x3

¬x1 ¬x2 ¬x3

xi in F, two vertices xi and ¬xi , together with an edge {xi , ¬xi }. Then
any vertex cover must include either xi or ¬xi . Furthermore, by choosing
an appropriate size for the vertex cover, we might be able to prohibit the
simultaneous inclusion of both xi and ¬xi .
In order to complete the reduction, we need to ensure that any vertex
cover of size k describes a satisfying assignment for F, and that for any
satisfying assignment for F, there is a vertex cover of size k that describes
it. To this end, we will add more structure to the graph we are constructing.
We know that for a satisfying assignment, each clause contains at least one
true literal. In order to model this constraint with a graph, let us construct,
for each clause Ci , the vertices ci1 , ci2 , and ci3 , along with the edges {ci1 , ci2 },
{ci2 , ci3 }, and {ci3 , ci1 }. Then any vertex cover must contain at least two of
these three vertices.
Finally, for 1 ≤ i ≤ n and 1 ≤ j ≤ 3, we construct an additional edge
{cij , αij }. For example, Figure 16.7 shows the graph constructed from the
3-CNF formula (x1 ∨ ¬x2 ∨ x3 ) ∧ (¬x1 ∨ x3 ). By setting k = m + 2n, where m
CHAPTER 16. N P-COMPLETENESS 526

is the number of variables and n is the number of clauses in F, we force any

vertex cover of size k to contain exactly one of the two vertices constructed
for each variable and exactly two of the three vertices constructed for each
clause. In order to cover all of the edges {cij , αij }, the vertex cover must be
such that in each clause Ci , there is at least one literal αij that belongs to
the vertex cover. Thus, we can represent an assignment of true to a literal
by including it in the vertex cover. We can now show the following lemma.

Lemma 16.12 Let F be a 3-CNF formula with m variables and n clauses,

and let G and k be the graph and positive integer resulting from the above
construction. Then G has a vertex cover of size k iff F is satisfiable.

Proof: We must show the implication in both directions.

⇒: Suppose G has a vertex cover S of size k = m+2n. Then S must contain

at least one of the two vertices xi and ¬xi for 1 ≤ i ≤ m, plus at least two
of the three vertices ci1 , ci2 , and ci3 for 1 ≤ i ≤ n. Because this gives a
total of at least k vertices, we conclude that S must contain exactly one
of xi and ¬xi and exactly two of ci1 , ci2 , and ci3 . Let us set xi to true iff
xi ∈ S. Consider any clause Ci . Let j be such that cij 6∈ S. Because the
edge {cij , αij } must be covered, αij must be in S. Therefore, at least one
literal in Ci is true. We conclude that F is satisfiable.

⇐: Suppose F is satisfiable. Let A be a satisfying assignment. We can then

construct a vertex cover S as follows. First, for 1 ≤ i ≤ m, if xi is true in
A, we include xi in S; otherwise, we include ¬xi . Thus, each of the edges
{xi , ¬xi } is covered. Then for 1 ≤ i ≤ n, we include in S two of ci1 , ci2 ,
and ci3 , so that the vertex that is not included is adjacent to an αij ∈ S
(note that because A is a satisfying assignment, such a vertex exists for each
clause). Thus, S is of size m + 2n = k and covers all edges in G.

It is easily seen that the above construction can be implemented to run

in O(m + n) time, or linear in the size of the formula — the details are
left as an exercise. From Lemma 16.12, 3-Sat ≤pm VC. Because 3-Sat
is N P-hard, it follows that VC is N P-hard. Because we have shown that
VC ∈ N P, we have the following theorem.

Theorem 16.13 VC is N P-complete.

A problem closely related to VC is the independent set problem, which
we denote IS. An independent set in an undirected graph G is a subset S of
CHAPTER 16. N P-COMPLETENESS 527

the vertices such that no pair of vertices in S is adjacent in G. The indepen-

dent set problem is to decide, for a given undirected graph G and natural
number k, whether G has an independent set of size k. The relationship
between IS and VC is shown by the following theorem.

Theorem 16.14 Let G = (V, E) be an undirected graph and S ⊂ V . Then

S is an independent set iff V \ S is a vertex cover.

Proof: We must show the implication in both directions.

⇐: Suppose S is an independent set. Let {u, v} ∈ E. Because u and v

cannot both be in S, at least one of them is in V \ S. It follows that V \ S
is a vertex cover.

⇒: Suppose V \ S is a vertex cover. Let u and v be two vertices in S.

Because V \ S is a vertex cover containing neither u nor v, u cannot be
adjacent to v. It follows that S is an independent set.

Given this close relationship between the two problems, it is an easy

matter to modify the proof that VC ∈ N P to show that IS ∈ N P. Further-
more, it follows from Theorem 16.14 that for an undirected graph G with n
vertices and a positive integer k, G has an independent set of size n − k iff G
has a vertex cover of size k. Clearly, we can construct n − k in polynomial If k > n, we can
time; hence, VC ≤pm IS. We therefore have the following theorem. use n + 1
instead.

Theorem 16.15 IS is N P-complete.

16.6 3-Dimensional Matching

In this section, we will study a problem closely related to the bipartite
matching problem of Section 14.3. The input will consist of three nonempty
disjoint sets, X, Y , and Z, each having the same number of elements, and a
set of triples W ⊆ X × Y × Z. We wish to decide if there is a subset M ⊆ W
such that each element of X ∪ Y ∪ Z occurs in exactly one triple in M . We
call this problem the 3-dimensional matching problem (3DM).
Note that if we were to use two disjoint sets instead of three, we could
think of the two sets as the two vertex sets of a bipartite graph. The set
of pairs (instead of triples) would then be directed edges. Our problem
would then be that of deciding whether there is a matching including all
CHAPTER 16. N P-COMPLETENESS 528

the vertices of this directed graph. 3DM is then the natural extension of
this problem to 3-dimensional hypergraphs. Using the algorithm Matching
(Figure 14.10 on page 463), we can decide the 2-dimensional version (2DM)
in O(na) time, where n is the number of vertices and a is the number of
edges in the graph; thus, 2DM ∈ P. However, we will now show that 3DM
is N P-complete.
In order to show that 3DM ∈ N P, let us first denote an instance by
X = {x1 , . . . , xm }
Y = {y1 , . . . , ym }
Z = {z1 , . . . , zm }
W = {w1 , . . . , wn }.
We interpret a bit string φ as encoding an array A[1..k] such that each block
of b bits encodes an element of A, where b is the number of bits needed to
encode n. Any bit string that does not have length exactly bm will be
considered to be invalid. To verify that the array A encoded by φ is a proof,
we can check that
• φ is valid;
• 1 ≤ A[i] ≤ n for 1 ≤ i ≤ m; and
• each element of X∪Y ∪Z belongs to some triple wA[i] , where 1 ≤ i ≤ m.
This can easily be done in O(bm2 ) time — the details are left as an exercise.
Hence, 3DM ∈ N P.
In order to show that 3DM is N P-hard, we need to reduce some N P-
complete problem to it. So far, we have identified five N P-complete prob-
lems: three satisfiability problems and two graph problems. However, none
of these bears much resemblance to 3DM. We therefore make use of a prin-
ciple that has proven to be quite effective over the years: when in doubt,
try 3-Sat.
As we did in showing 3-Sat ≤pm VC, we will begin by focusing on the
proofs of membership in N P for two problems. Specifically, we want to
relate the choice of a subset of W to the choice of truth values for boolean
variables. Let’s start by considering two triples, hx, ax , bx i and h¬x, ax , bx i,
where x is some boolean variable. If these are the only two triples containing
ax or bx , any matching must include exactly one of these triples. This choice
could be used to set the value of x.
If we were to construct two such triples for each variable, we would then
need to construct triples to represent the clauses. Using a similar idea,
CHAPTER 16. N P-COMPLETENESS 529

we could introduce, for a given clause αi1 ∨ αi2 ∨ αi3 , the triples hαi1 , ci , di i,
hαi2 , ci , di i, and hαi3 , ci , di i — one triple for each literal in the clause. Again,
any matching must contain exactly one of these triples. If we let x be false
when hx, ax , bx i is chosen, then the triple chosen for the clause must contain
a true literal.
This construction has a couple of shortcomings, however. First, because
each literal must occur exactly once in a matching, we can use a given
variable to satisfy only one clause. Furthermore, if more than one literal is
true in a given clause, there may remain literals that are unmatched. These
shortcomings should not be too surprising, as we could do essentially the
same construction producing pairs instead of triples — the third components
are redundant. Thus, if this construction had worked, we could have used
the same technique to reduce 3-Sat to 2DM, which belongs to P. We would
have therefore proved that P = N P.
In order to overcome the first shortcoming, we need to enrich our con-
struction so that we have several copies of each literal. To keep it simple, we
will make one copy for each clause, regardless of whether the literal appears
in the clause. We must be careful, however, so that when we choose the
triples to set the boolean value, we must either take all triples containing x
or all triples containing ¬x. Because we are constructing triples rather than
pairs, we can indeed accomplish these goals.
Let x1 , . . . , xn denote all the copies of the literal x, and let ¬x1 , . . . , ¬xn
denote all the copies of the literal ¬x. We then introduce the following
triples (see Figure 16.8):
• hxi , axi , bxi i for 1 ≤ i ≤ n;
• h¬xi , axi , bx,i+1 i for 1 ≤ i ≤ n − 1; and
• h¬xn , axn , bx1 i.
It is not too hard to see that in order to match all of the axi s and bxi s, a
matching must include either those triples containing the xi s or those triples
containing the ¬xi s.
We can now use the construction described earlier for building triples
from clauses, except that for clause i, we include the ith copy of each literal
in its triple. Thus, in any matching, there must be for each clause at least one
triple containing a copy of a literal. However, there still may be unmatched
copies of literals. We need to introduce more triples in order to match the
remaining copies.
Suppose our 3-CNF formula F has n clauses and m variables. Then our
construction so far contains:
CHAPTER 16. N P-COMPLETENESS 530

Figure 16.8 Triples for setting boolean values in the reduction from 3-Sat
to 3DM, with n = 4

x1

¬x1 ax1 bx1 ¬x4

bx2 ax4

x2 x4
ax2 bx4

¬x2 bx3 ax3 ¬x3

x3

• 2mn copies of literals;

• mn as and n cs; and

• mn bs and n ds.

In order to make the above three sets of equal size, we add ei to the second
set and fi to the third set, for 1 ≤ i ≤ (m−1)n. We then include all possible
triples hxi , ej , fj i and h¬xi , ej , fj i for 1 ≤ i ≤ n and 1 ≤ j ≤ (m−1)n. Using
this construction, we can now show the following theorem.

Theorem 16.16 3DM is N P-complete.

Proof: We have shown that 3DM ∈ N P. We will show that it is N P-hard

by showing that 3-Sat ≤pm 3DM. Specifically, we will show that the above
construction is a polynomial-time many-one reduction.
We will first show that the construction can be computed in polynomial
time. It is easily seen that the time required for the construction is pro-
CHAPTER 16. N P-COMPLETENESS 531

portional to the number of triples produced. Suppose the CNF formula F

contains m variables and n clauses. The triples produced include
• 2mn triples for setting truth values;

• one triple for each literal in each clause, or at most 3n triples; and

• triples hxi , ej , fj i and h¬xi , ej , fj i for each variable x, each i such that
1 ≤ i ≤ n, and each j such that 1 ≤ j ≤ (m − 1)n, or 2(m − 1)mn2
triples.
Thus, the total number of triples produced is at most

2mn + 3n + 2(m − 1)mn2 .

Because this value is polynomial in the size of F, the construction can be

done in polynomial time.
Let W be the set of triples constructed. In order to complete the proof,
we must show that W contains a matching iff F is satisfiable.

⇒: Suppose W contains a matching M . As we have argued above, for each

variable x, M must contain either those triples hxi , axi , bxi i or those triples
h¬xi , axi , bxi i, for 1 ≤ i ≤ n. Let us set x to false iff the triples hxi , axi , bxi i
belong to M . M must also contain some triple hαij , ci , di i for 1 ≤ i ≤ n.
Because αij cannot also be in another triple in M , αij must be true. Thus,
each clause contains at least one true literal, so that F is satisfiable.

⇐: Suppose F is satisfiable, and let A denote a satisfying assignment of

boolean values to the variables in F. We construct a matching M as follows.
First, if x is false in A, we include hxi , axi , bxi i for 1 ≤ i ≤ n; otherwise, we
include h¬xi , axi , bxi i for 1 ≤ i ≤ n. Thus, each axi and bxi is included
exactly once. Then for clause i, because A is a satisfying assignment there
is at least one literal αij that is true in A. Because αij has not yet been
included in M , we can include the triple hαij , ci , di i in M . Thus, M includes
each ci and di exactly once.
At this point M includes no item more than once, but does not include
any of the ei s or fi s. Furthermore, because exactly mn + n of the xi s
and ¬xi s have been included, (m − 1)n have not yet been included. Let
β1 , . . . , β(m−1)n denote the xi s and ¬xi s that have not yet been included.
We complete M by including hβi , ei , fi i for 1 ≤ i ≤ (m − 1)n. It is now
easily seen that M is a matching.
CHAPTER 16. N P-COMPLETENESS 532

16.7 Partitioning and Strong N P-Completeness

In this section, we will look at partitioning problems related to the 0-1 knap-
sack problem of Section 12.4. These are the first N P-complete problems we
will have seen in which numbers play a significant role. As we will see, the
N P-completeness of a number problem does not always imply intractability.
For this reason, we will introduce a stronger notion of N P-completeness.
The most basic partitioning problem consists of a set of items, each
having a positive integer weight. The problem is to decide whether the items
can be partitioned into two disjoint subsets having identical total weight.
More formally, let w1 , . . . , wn denote the weights of the items. We wish to
decide whether there is a subset S ⊆ {1, . . . , n} such that
X X
wi = wi .
i∈S i6∈S

The problem is known as the partition problem, or Part. We leave it as an

exercise to show that Part ∈ N P. We will show that Part is N P-hard,
and therefore N P-complete.
Before showing the N P-hardness of Part, however, we first observe
that this problem is a special case of the 0-1 knapsack problem in which the
values are equal to the weights and the weight bound W is half the total
weight. In Section 12.4, we sketched an algorithm to solve this problem in
O(nW ) time. Clearly, this same algorithm can be applied to Part. This
would seem to imply that Part ∈ P, so that showing Part to be N P-hard
would amount to showing P = N P.
However, the O(nW ) algorithm does not prove that Part ∈ P. The
reason is that we have defined P to be the set of decision problems that
can be decided in a time polynomial in the length of their inputs. We claim
that nW is not necessarily polynomial in the length of the input to Part.
To see why, note that the number of bits required to encode an integer is
logarithmic in the value of the integer; hence, the value is exponential in
the length of the encoding. Because W is one of the integers given as input,
nW is not bounded by a polynomial in the length of the input.
The relationship between the value of an integer and the length of its
binary encoding is essential to the N P-hardness of Part, as its proof will
illustrate. We will now present that proof, which is a reduction from 3DM.
Let W , X, Y , and Z represent an instance of 3DM, where
• X = {x0 , . . . , xm−1 };
• Y = {y0 , . . . , ym−1 };
CHAPTER 16. N P-COMPLETENESS 533

• Z = {z0 , . . . , zm−1 }; and

• W = {w0 , . . . , wn−1 } such that each wi ∈ X × Y × Z.

We will construct a weight for each triple, plus two additional weights.
Suppose hxi , yj , zk i ∈ W . The weight we construct for this triple will be

(n + 1)2m+i + (n + 1)m+j + (n + 1)k .

If we were to express this weight in radix n + 1, it would consist entirely of

1s and 0s and have exactly three 1s. The positions of the three 1s in this
encoding determine the three components of the triple as follows:
i digits j digits k digits
z }| { z }| { z }| {
1 00 · · · 0 |00 · · · 01 | · · · 01
{z00 · · · 0} 00 {z00 · · · 0} .
m digits m digits

Consider any subset S ⊆ W . Clearly, no element of X ∪ Y ∪ Z can occur

in more than n triples in S. Thus, when viewed in radix n + 1, the sum
of the weights corresponding to the elements of S describes the number of
occurrences of each element of X ∪ Y ∪ Z in S. Specifically, if we number
the digits beginning with 0 for the least significant digit, then

• digit 2m + i gives the number of occurrences of xi in S, for 0 ≤ i < m;

• digit m + j gives the number of occurrences of yj in S, for 0 ≤ j < m;

and

• digit k gives the number of occurrences of zk is S, for 0 ≤ k < m.

It follows that S is a matching iff the sum of its corresponding weights is

3m−1
X
M= (n + 1)i ,
i=0

which in radix n + 1 is simply 3m 1s.

In order to complete the construction, we need two more weights. Let
C denote the sum of the n weights constructed so far. We construct the
following two weights:
A = 2C − M
and
B = C + M.
CHAPTER 16. N P-COMPLETENESS 534

Thus, the sum of all n + 2 weights is 4C. Because A + B = 3C > 2C,

A and B cannot belong to the same subset in a partition. Furthermore, the
subset containing A must also contain items corresponding to elements of W
having total weight M . Because these elements must form a matching, the
weights we have constructed contain a partition iff W contains a matching.
To see that the time needed to construct the instance of Part is poly-
nomial in the size of the instance of 3DM, we first observe that
3m−1
X
2C ≤ 2 n(n + 1)i
i=0
< 2(n + 1)3m .

Therefore each weight constructed has a binary encoding with no more than
1 + ⌈3m lg(n + 1)⌉ bits. Because addition, subtraction, multiplication, and
exponentiation can all be performed in a time polynomial in the number
of bits in their operands (see Exercise 4.14, Section 10.1, Exercise 10.24,
and Sections 15.3-15.4), the construction can clearly be performed in time
polynomial in the size of the instance of 3DM. We therefore have the
following theorem.

Theorem 16.17 Part is N P-hard.

Note that in the above construction, the weights can become extremely
large, though their lengths are all polynomial in the size of the instance of
3DM. It is not too hard to imagine that we might want to solve the partition
problem for a large number of weights — thousands or perhaps even millions.
However, when numbers represent physical quantities — including time —
we don’t expect them to be very long. For example, about 300 bits are
sufficient to encode in binary the estimated number of elementary particles
in the universe. Thus, because there is an algorithm for Part whose running
time is a low-order polynomial in the length of the input and the values
encoded in the input, it seems unreasonable to consider this problem to be
intractable.
In order to accommodate numbers in the input, we say that an algorithm
is pseudopolynomial if its running time is bounded by some polynomial in the
length of the input and the largest integer encoded in the input. Thus, the
O(nW ) algorithm for 0-1 knapsack (and hence partition) is pseudopolyno-
mial. Whenever the numbers in a decision problem’s input refer to physical
quantities, we consider the problem to be tractable if it has a pseudopolyno-
mial algorithm. However, if the numbers are purely mathematical entities
CHAPTER 16. N P-COMPLETENESS 535

(as, for example, in cryptographic applications), we consider the problem to

be tractable only if it belongs to P.
We would also like to extend the notion of N P-hardness to account
for numbers in the input. To this end, we first define a way to restrict a
decision problem so that no integer in an instance is too large. Specifically,
for a decision problem X and a function f : N → R≥0 , we define Xf to be
the restriction of X to instances x in which no integer has a value larger
than f (|x|). We then say that X is N P-hard in the strong sense if there is
a polynomial p such that Xp is N P-hard. If, in addition, X ∈ N P, we say
that X is N P-complete in the strong sense.
Suppose we were to find a pseudopolynomial algorithm for a strongly
N P-hard problem. When we restrict the problem so that its instances
have integers bounded by some polynomial, the pseudopolynomial algorithm
becomes truly polynomial, so that the restricted problem would be in P.
Furthermore, this restricted problem is still N P-hard in the ordinary sense.
Thus, by Theorem 16.6, we would have shown that P = N P. It therefore
seems highly unlikely that there is a pseudopolynomial algorithm for any
strongly N P-hard problem.
In order to show a problem to be N P-hard in the strong sense, we must
ensure that the reduction produces numbers whose values are bounded above
by some polynomial in the length of the instance we construct. The proof of
Cook’s Theorem in Section 16.8 does not construct large integers; hence, Sat
is N P-complete in the strong sense. Furthermore of all of the N P-hardness
proofs we have presented so far, only the proof that Part is N P-hard con-
structs integers whose values are not bounded by some polynomial in the
length of the input. As a result, CSat, 3Sat, VC, IS, and 3DM are all
N P-complete in the strong sense. However, these results are rather unin-
teresting because none of their instances contain numbers that can become
large in comparison to the length of the input without rendering the problem
trivial.
In what follows, we will show a problem with potentially large numbers
to be N P-complete in the strong sense. We will use a restricted form of
polynomial many-one reduction motivated by the following theorem.

Theorem 16.18 Let f be a polynomial many-one reduction from problem

X to problem Y , where X is N P-hard in the strong sense. Suppose that f
satisfies the following properties:

1. there is a polynomial p1 such that p1 (|f (x)|) ≥ |x| for every instance
x of X; and
CHAPTER 16. N P-COMPLETENESS 536

2. there is a two-variable polynomial p2 such that each integer con-

structed has a value no greater than p2 (|x|, µ(x)), where µ(x) denotes
the maximum value of any integer in x.
Then Y is N P-hard in the strong sense.

Proof: Because X is N P-hard in the strong sense, there is some poly-

nomial p such that Xp is N P-hard. If the reduction is then applied to
Xp , all numbers constructed will have values bounded by p2 (|x|, p(|x|)), by
Property 2. Furthermore, by Property 1, these values are no more than
p2 (p1 (|f (x)|), p(p1 (|f (x)|))), which is a polynomial in the length of the in-
stance constructed. The reduction from Xp to Y therefore shows that Y is
N P-hard in the strong sense.

We say that a polynomial many-one reduction is a pseudopolynomial

reduction, denoted by ≤pp m , if it satisfies the properties given in Theorem
16.18.
Let us now consider other partitioning problems. For a fixed natural
number k > 1, the k-partition problem (k-Part) is defined as follows. The
input consists of kn items, each having a positive integer weight, such that
the sum of the weights is Bn for some positive integer B. Furthermore, each
weight w must satisfy
B B
<w< .
k+1 k−1
The question we ask is whether the kn items can be partitioned into n dis-
joint subsets, each having total weight exactly B. Note that the constraints
on the weights imply that each subset will contain exactly k items.
In what follows, we will show that 4-Part is N P-hard in the strong
sense. We leave it as exercises to show that
• k-Part ∈ N P for all k > 1;

• 3-Part is N P-complete in the strong sense; and

• 2-Part ∈ P.
We will show that 3DM ≤pp m 4-Part. The reduction will be somewhat
similar to the reduction from 3DM to Part, but we must be careful that
the weights we construct are not too large. Let us describe an instance of
3DM using the same notation as we did for the earlier reduction. We will
assume that each element occurs in at least one triple. Otherwise, there is
no matching, and we can create an instance with 7 items having weight 6
CHAPTER 16. N P-COMPLETENESS 537

and one item having weight 8, so that the total weight is 50, and B = 25.
Clearly, 25/5 < 6 < 8 < 25/3; hence, this is a valid instance, but there is
clearly no way to form a subset with weight 25.
We will construct, for each triple hxi , yj , zk i ∈ W , four weights: one
weight for each of xi , yj , and zk , plus one weight for the triple itself. Because
each element of X ∪ Y ∪ Z can occur in several triples, we may construct
several items for each element. Exactly one of these will be a matching
item. All non-matching items constructed from the same element will have
the same weight, which will be different from that of the matching item
constructed from that element. We will construct the weights so that in any
4-partition, the item constructed from a triple must be grouped with either
the matching items constructed from the elements of the triple, or three
non-matching items — one corresponding to each element of the triple. In
this way, a 4-partition will exist iff W contains a matching.
As in the previous reduction, it will be convenient to view the weights
in a particular radix r, which we will specify later. In this case, however,
the weights will contain only a few radix-r digits. We will choose r to be
large enough that when we add any four of the weights we construct, each
column of digits will have a sum strictly less than r; hence, we will be able
to deal with each digit position independently in order to satisfy the various
constraints. Note that if we construct the weights so that for every triple,
the sum of the four weights constructed is the same, then this sum will be
B.
We will use the three low-order digits to enforce the constraint that the
four items within any partition must be derived from some triple and its
three components. To this end, we make the following assignments:
• For any weight constructed from xi ∈ X, we assign i + 1 to the first
digit and 0 to the second and third digits.

• For any weight constructed from yj ∈ Y , we assign j + 1 to the second

digit and 0 to the first and third digits.

• For any weight constructed from zk ∈ Z, we assign k + 1 to the third

digit and 0 to the first two digits.

• For a triple hxi , yj , zk i, we assign the first three digits the values 2m−i,
2m − j, and 2m − k, respectively.
B will therefore have 2m + 1 as each of its three low-order digits.
Note that because each weight constructed from a triple has a value of
at least m + 1 in each of its three low-order digits, no two of these weights
CHAPTER 16. N P-COMPLETENESS 538

can be grouped together. Furthermore, because all three low-order digits of

any weight constructed from an element of X ∪ Y ∪ Z have values no more
than m, and at least two of these values are 0, in order to reach a sum of B
with only four weights, at least one must correspond to a triple. Thus, in
any 4-partition, each group must contain exactly one weight corresponding
to a triple, and the other three weights must correspond to the elements of
that triple.
We will use the fourth digit to enforce the constraint that for any four
items grouped together, the items corresponding to elements from X ∪Y ∪Z
are either all matching items or all non-matching items. We therefore assign
the fourth digits as follows:
• for each matching item: 1;
• for each non-matching item from X ∪ Y : 0;
• for each non-matching item from Z: 3; and
• for each item from W : 0.
The fourth digit of B will therefore be 3. Furthermore, because any group
must have one item from each of W , X, Y , and Z, it must contain either
three matching items or three non-matching items. Setting the digits in this
way therefore ensures that the set of weights has a 4-partition iff W contains
a matching.
Finally, in order to ensure that each weight is greater than B/5 and less
than B/3, we set the fifth digit of all weights to 1. As a result, the fifth digit
of B is 4. We will see that by choosing r to be sufficiently large, all weights
will be within the proper range.
To summarize, for a triple hxi , yj , zk i, we have the weight
r4 + (2m − k)r2 + (2m − j)r + 2m − i.
For the elements xi ∈ X, yj ∈ Y , and zk ∈ Z, we have for the matching
items the weights
r4 + r3 + i+1
r4 + r3 + (j + 1)r
r4 + r3 + (k + 1)r2
and for any non-matching item the weights
r4 + + i+1
r4 + + (j + 1)r
r4 + 3r3 + (k + 1)r2
CHAPTER 16. N P-COMPLETENESS 539

Furthermore,

B = 4r4 + 3r3 + (2m + 1)r2 + (2m + 1)r + 2m + 1.

To complete the reduction, we must assign a value to r. As we have

observed, r must be larger than the sum of any four digits occurring in the
same column. Thus, r must be strictly larger than both 8m and 12. Because
m ≥ 1, we can satisfy these constraints by setting r = 13m. We then have

B/5 < (4r4 + 4r3 )/5

< (4r4 + r4 )/5
= r4 ,

so that every weight is larger than B/5. Furthermore, each weight is less
than

r4 + 4r3 < r4 + r4 /3
= 4r4 /3
< B/3.

We must now show that the above reduction is pseudopolynomial. We

first observe that no weight is larger than 2r4 = 2(13m)4 , which is polyno-
mial in the size of the instance of 3DM. Furthermore, it is easily seen that
all the weights can be constructed in a time linear in the size of the instance
of 3DM. We therefore have the following theorem.

Theorem 16.19 4-Part is N P-hard in the strong sense.

16.8 Proof of Cook’s Theorem

In this section, we will present a proof of Cook’s Theorem, namely, that Sat
is N P-complete. As we have already observed in Section 16.1, Sat ∈ N P. It
therefore remains to be shown that Sat is N P-hard. Furthermore, because
we have used Cook’s Theorem either directly or indirectly to show all of our
N P-hardness results, we cannot use any of these results to prove Cook’s
Theorem. Because we cannot show the N P-hardness of Sat by reducing
any known N P-hard problem to it, we must directly use the definition of
N P-hardness — that is, we must directly show that for every problem
X ∈ N P, X ≤pm Sat.
CHAPTER 16. N P-COMPLETENESS 540

Our proof will therefore involve a generic reduction. This kind of re-
duction is more abstract in that we begin with an arbitrary X ∈ N P.
Specifically, the same reduction should work for any problem X that we
might choose from N P. Thus, the only assumption we can make about X
is that it satisfies the definition of N P: there exist a polynomial p(n) and a
decision problem Y ⊆ I × B, where I is the set of instances of X, such that
• Y ∈ P — that is, there exist a polynomial p′ (n) and an algorithm A
that takes an element x ∈ I and an element φ ∈ B as its inputs and
decides within p′ (|x| + |φ|) steps whether (x, φ) ∈ Y ;

• for each x ∈ I, x ∈ X iff there is a proof φ ∈ B such that (x, φ) ∈ Y ;

and

• for each x ∈ X, there is a proof φ ∈ B such that (x, φ) ∈ Y and

|φ| ≤ p(|x|).
For our reduction, we need to construct, for a given instance x ∈ I, a
boolean formula F that is satisfiable iff x ∈ X. Equivalently, F must be
satisfiable iff there is a φ ∈ B such that |φ| ≤ p(|x|) and (x, φ) ∈ Y . Our
reduction therefore will construct from x a formula F that in some sense
simulates the algorithm A on x and some unknown φ, where |φ| ≤ p(|x|).
The input for our reduction is the instance x. However, because our
reduction is generic, it must work for any algorithm A and any polynomials
p(n) and p′ (n) satisfying the above constraints. Therefore, A, p, and p′ are
in a sense additional inputs to our reduction. What makes constructing such
a reduction rather difficult is the fact that one of these additional inputs is
an algorithm. In order to be able to handle an algorithm as input to an
algorithm, we need to define more precisely what we mean by an algorithm.
Rather than formalizing all of the constructs we have been using in our
algorithms, we will instead simplify matters by defining a lower-level model
of computation, which we will call a random access machine, or RAM . Thus,
in constructing a boolean formula to simulate a RAM, we will in essence be
defining an interpreter for a simple machine language. Such a task is much
simpler than defining an interpreter for a high-level language, such as the
notation we have been using to present our algorithms.
In order to maintain some consistency between this computational model
and the algorithms we have designed, we will assume that a RAM consists
of the following:
• a fixed program consisting of a sequence of P > 0 instructions num-
bered 0 through P − 1;
CHAPTER 16. N P-COMPLETENESS 541

• a program counter, which is initially 0 and can store any natural num-
ber less than P ;

• countably infinitely many memory locations, each of which is initially

0 and can store any natural number;

• two input streams from which values may be read one bit at a time;
and

• a single output bit, which is produced when the program terminates.

Because we will be using this model only for representing an algorithm for
deciding whether (x, φ) ∈ Y , we need exactly two input streams, one for x
and one for φ. Furthermore, we can represent a “yes” output by setting the
output bit to 1.
We will assume that each memory location is addressed by a unique
natural number. Each machine will then have the following instruction set:

• Input(i, l): Stores the next bit from input stream i, where i is either
0 or 1, in memory location l. If all of the input has already been read,
the value 2 is stored.

• Load(n, l): Stores the natural number n at memory location l.

• Copy(l1 , l2 ): Copies the value stored at location l1 into location l2 .

• Goto(p): Changes the value of the program counter to p.

• IfLeq(l1 , l2 , p): If the value at location l1 is less than or equal to the

value at location l2 , changes the value of the program counter to p.

• Add(l1 , l2 ): Adds the value in location l1 to the value in location l2 ,

saving the result in location l2 .

• Subtract(l1 , l2 ): Subtracts the value in location l1 from the value in

location l2 , saving the result in location l2 .

• Shift(l): Replaces the value n stored in location l with ⌊n/2⌋.

• Halt(b): Terminates the program with output b, which must be either

0 or 1.

In addition, wherever a memory location is used, an indirection operator

“∗” may be added. The expression “∗l” indicates that the location referenced
CHAPTER 16. N P-COMPLETENESS 542

by the value stored in location l should be used. Thus, for example, if 21 is

stored at 0, 2 is stored at 1, and 52 is stored at 2, the instruction

Add(0, ∗1)

will add 21 (stored at 0) to 52 (stored at 2, the value referenced by 1), and

store the sum 73 in location 2. Indirection operators cannot be nested.
Each instruction that does not explicitly change the program counter will
increment it by 1. We will assume that any time an instruction cannot be
executed (e.g., because a larger number is subtracted from a smaller number
or the program counter would index beyond the program), the program will
immediately terminate with output 0.
We will now argue somewhat informally that for any decision problem
X, there is a deterministic polynomial-time algorithm deciding X iff there
is a polynomial-time RAM deciding X. First, if we have a deterministic
polynomial-time algorithm A, we can build a RAM to execute A using stan-
dard compiling techniques. Some statements in an algorithm may require
much more time when compiled to a RAM. For example, to implement a
multiplication, we can use the following top-down formulation:

0
 if b = 0
ab = (a + a) 2 b
if b is positive and even

 b−1
(a + a) 2 + a if b is odd.

To determine whether b = 0, we can simply check whether b ≤ 0. To

determine whether b is even, we can copy b to c, shift c yielding d = ⌊b/2⌋,
and subtract d + d from b. If the result is less than or equal to 0, then b is
even; otherwise, b is odd. This technique can be implemented with a loop
that runs in time linear in the number of bits in b.
In order to implement data structures, we need a memory manager for
a RAM. Because we have infinitely many memory locations, only finitely
many of which may be in use at any given time, we can use one memory
location l1 to store a value avail such that for every l ≥ avail, l is unused.
If we need to allocate n locations, we can copy l1 to some memory location
l2 and add n to l1 . To access the ith of these n locations, we add i to l2
and use the indirection operator on the result. Thus, it is a straightforward
matter to implement the data structures in this text. Specifically, because
we can implement a Stack, we can implement function and operation calls,
including recursion.
It is important to realize that even though some operations are less
efficient on a RAM, there is some polynomial p(n) that bounds the running
CHAPTER 16. N P-COMPLETENESS 543

time of any simulation of an algorithmic step. Thus, if we have an algorithm

that runs in p′ (n) steps, the RAM simulation will run in no more than
p′ (p(n)) steps, which is still polynomial in n. Furthermore, if the values of
the natural numbers in the algorithm are bounded by p′ (n), then so are the
values in the memory locations of the RAM.
Conversely, it is not hard to write an algorithm to simulate a polynomial-
time RAM. We can use variables to store the value of the program counter
and the indices of the next available bits of the two input streams. In addi-
tion, we can use a VArray (see Section 9.5) to store the memory locations
used by the RAM. Note that because the values of all memory locations must
always be less than 2p(n) , where p(n) is some polynomial in the number of
bits in the two input streams, all memory locations that can be accessed We must also
have addresses strictly less than 2p(n) . We can therefore use a VArray of make sure that
2p(n) bounds
size 2p(n) to keep track of the RAM’s memory. It is then a straightforward each constant
matter to simulate the RAM using constant time for each instruction, plus appearing in the
RAM program.
some constant time for initialization.
Using RAMs, we now have a slightly different characterization of N P.
Specifically, N P is the set of all decision problems X such that there exist

• polynomials p(n) and p′ (n); and

• a RAM M deciding a problem Y ⊆ I ×B, where I is the set of instances

for X;

such that

• M terminates within p′ (|x| + |φ|) steps on input (x, φ);

′
• all memory locations of M maintain values strictly less than 2p (|x|+|φ|)
given input (x, φ);

• for each x ∈ I, x ∈ X iff there is a proof φ ∈ B such that (x, φ) ∈ Y ;

and

• for each x ∈ X, there is a φ ∈ B such that (x, φ) ∈ Y and |φ| ≤ p(|x|).

Thus, to reduce an arbitrary problem X ∈ N P to Sat, we need to

construct from a given x ∈ I, where I is the set of instances of X, a boolean
formula F such that F is satisfiable iff there is a φ ∈ B with |φ| ≤ p(|x|)
such that M outputs 1 on input (x, φ). Furthermore, the running time of the
construction of F must be bounded by some polynomial in |x|. In designing
the construction, we may utilize p(n), p′ (n), and M , which depend only on
CHAPTER 16. N P-COMPLETENESS 544

the problem X; however, the instance x is the input for the construction, so
that we cannot know it in advance.
Because we can use any polynomial-time M that decides Y , we can sim-
plify matters further by making a couple of assumptions about M . First,
we can assume that M contains at least one Halt(1) instruction — if there
is no input yielding a “yes” answer, we can always include such an instruc-
tion at an unreachable location. Second, because statements that cannot
be executed due to error conditions have the same effect as Halt(0), and
because the instruction set is powerful enough to check any run-time er-
ror conditions, we can assume that all statements can be executed without
error.
In addition, we make some simplifying assumptions regarding the poly-
nomials p and p′ . First, we note that by removing any negative terms from
p′ , we obtain a polynomial that is nondecreasing and never less than the
original polynomial. Thus, we can assume that p′ is nondecreasing, so that
p′ (|x| + p(|x|)) will give an upper bound on the number of steps executed by
M on an input (x, φ) with |φ| ≤ p(|x|). Furthermore, because p′ (n + p(n))
is a polynomial, we can assume that p′ (n) is an upper bound on the number
of steps taken by M on any input (x, φ) such that |x| = n and |φ| ≤ p(n).
Note that with these assumptions, p′ (n) ≥ p(n) for all n. We can therefore
choose p(n) = p′ (n), so that we can use a single polynomial p to bound both
|φ| and the number of steps executed by M . Finally, we can choose p so
that p(n) ≥ n for all n ∈ N.
As the first step in our construction, we need boolean variables to rep-
resent the various components of the state of M at various times in its
execution. First, we need variables describing the input sequences x and φ.
For x, we will use the variables x[k] for 1 ≤ k ≤ n, where n is the length
of x. Because φ is unknown, even during the execution of the construction,
we cannot know its exact length; however, we do know that its length is no
more than p(n). We therefore will use the variables φ[k] for 1 ≤ k ≤ p(n) to
represent φ.
We also need variables to keep track of which bits are unread at each
step of the execution of M . For this purpose, we will use the variables x bi [k]
for 0 ≤ i ≤ p(n) and 0 ≤ k ≤ n, plus the variables φbi [k] for 0 ≤ i ≤ p(n)
and 0 ≤ k ≤ p(n). We want x bi [k] to be true iff the kth bit of x has not been
read after i execution steps. Likewise, we want φbi [k] to be true iff the kth
bit of φ exists and has not been read after i execution steps.
We then need to record the value of the program counter at each execu-
tion step. We will use the variables pij for 0 ≤ i ≤ p(n) and 0 ≤ j < P for
this purpose, where P denotes the number of instructions in the program of
CHAPTER 16. N P-COMPLETENESS 545

M . We want pij to be true iff the program counter has a value of j after i
execution steps.
Recording the values of the memory locations at each execution step
presents more of a challenge. Because a memory location can contain any
value less than 2p(n) , M can access any memory location with an address less
than 2p(n) . If we were to construct variables for each of these locations, we
would end up with exponentially many variables. We cannot hope to con-
struct a formula containing this many variables in a polynomial amount of
time. However, the number of memory locations accessed by any instruction
is at most four — the number accessed by Copy(∗l1 , ∗l2 ) in the worst case.
As a result, M can access a total of no more than 4p(n) different memory
locations. We can therefore use a technique similar to the implementation of
a VArray (see Section 9.5) in order to keep track of the memory locations
actually used.
Specifically, we will let the variables aj [k], for 1 ≤ j ≤ 4p(n) and 1 ≤ k ≤
p(n), denote the value of the kth bit of the address of some location lj , where
the first bit is the least significant bit. Here, we will let true represent 1 and
false represent 0. Then the variables vij [k], for 0 ≤ i ≤ p(n), 1 ≤ j ≤ 4p(n),
and 1 ≤ k ≤ p(n), will record the value of the kth bit of the value stored
at location lj after i execution steps. We will make no requirement that
location lj actually be used by M , nor do we require that lj be a different
location from lj ′ when j 6= j ′ .
Finally, we will use the additional variables ci [0..p(n)] and di [1..p(n)] for
1 ≤ i ≤ p(n). We will explain their purposes later.
Before we describe the formula F that we will construct, let us first
define some abbreviations, or “macros”, that will make the description of F
simpler. First, we will define the following:

If(y, z) = ¬y ∨ z.

This abbreviation specifies that if y is true, then z must also be true. How-
ever, if y is false, then no constraint is placed upon z. Note that such an
expression can be constructed in O(1) time.
We can extend the above abbreviation to specify an if-then-else con-
struct:
IfElse(y, z1 , z2 ) = If(y, z1 ) ∧ If(¬y, z2 ).
This specifies that if y is true, then z1 is true, but if not, then z2 is true.
Clearly it can be constructed in O(1) time.
CHAPTER 16. N P-COMPLETENESS 546

We will now define an abbreviation for the specification that two vari-
ables y and z are equal:

Eq(y, z) = IfElse(y, z, ¬z).

Again, such an expression can be constructed in O(1) time.

We can extend this abbreviation in a couple of ways. First, we can
use one of the constants true or false in place of one of the variables. This
will be useful, for example, if we want to specify that some variable has the
same value as some specific bit of the input x, say x[k]. Because x[k] is not a
variable in the formula, it cannot appear in the formula; however, when we’re
designing the construction, we don’t know this value. It would therefore be
convenient to be able to write Eq(y, x[k]), and to let the construction fill in
the appropriate value for x[k]. We then define

Eq(y, true) = y

and
Eq(y, false) = ¬y.
We can also extend the abbreviation to arrays of variables as follows:
n
^
Eq(y[1..n], z[1..n]) = Eq(y[k], z[k]).
k=1

Such an expression can be constructed in O(n) time. For different sizes of

arrays, the running time will be proportional to the number of elements. To
aid in readability, we will typically drop the range of subscripts when the
entire array is used.
When we specify the behavior of a program step, we need a way of check-
ing to see if aj records some particular memory location l. Let l[1..p(n)] be
the bits comprising l. We can check whether a given aj [1..p(n)] = l[1..p(n)]
using the Eq abbreviation. However, we may in many cases need to check
whether aj records the memory location indirectly addressed by l. For this
test, we use the following abbreviation:
4p(n)
_
Ind(i, j, l) = (Eq(aj ′ , l) ∧ Eq(aj , vij ′ )).
j ′ =1

Because aj , a′j , l, and vij ′ are arrays of p(n) elements, this expression can
be constructed in O(p2 (n)) time.
CHAPTER 16. N P-COMPLETENESS 547

Finally, we will need some abbreviations for specifying the behavior of

arithmetic and comparison instructions. We can express all of these behav-
iors using the specification of a sum. Thus, we need to express that the sum
of y[1..p(n)] and z[1..p(n)] is s[1..p(n)]. In order to express this constraint,
we will need to represent the “carry” bits used to compute the sum. For
this purpose, we will use the variables ci [0..p(n)] for some i. Specifically, for
1 ≤ k ≤ p(n), ci [k] will be the carry from the sum of y[k], z[k], and ci [k − 1],
and the value of ci [0] will be false, denoting 0.
We first observe that the low-order bit of the sum of y[k], z[k], and
ci [k − 1] is the exclusive-or of the three bits. We therefore first define an
abbreviation for specifying exclusive-or:

Xor(y, z) = (y ∨ z) ∧ ¬(y ∧ z).

We then observe that the carry bit of this sum is 1 iff at least two of the
three bits are 1. Stated another way, the carry bit is 1 iff either

• both y[k] and z[k] are 1; or

• ci [k − 1] is 1 and either y[k] or z[k] is 1.

We therefore define the following abbreviation:

Sum(y, z, s, i) =¬ci [0] ∧ ¬ci [p(n)] ∧

p(n)
^
(Eq(s[k], Xor(Xor(y[k], z[k]), ci [k − 1])) ∧
k=1
Eq(ci [k], (y[k] ∧ z[k]) ∨ (ci [k − 1] ∧ (y[k] ∨ z[k])))).

This expression states that y[1..p(n)] + z[1..p(n)] = s[1..p(n)] and that

ci [0..p(n)] give the carry bits of this sum. This expression can be constructed
in O(p(n)) time.
Let us now begin to construct our formula F. This formula will be the
conjunction of a number of sub-formulas, each of which will specify some
constraint on the values of the boolean variables. These constraints together
will encode the requirement that M gives a “yes” answer on input (x, φ),
where x is the given input for problem X. Because we will not specify the
value of φ, the formula will be satisfiable iff there is some φ with length at
most p(n) such that M yields an output of 1 on (x, φ). Furthermore, we
will show that we can construct this formula within a polynomial amount
of time.
CHAPTER 16. N P-COMPLETENESS 548

For the first constraint, we will specify that for 0 ≤ i ≤ p(n), there is
at most one j such that pij is true. Thus, there will be no ambiguity as to
the value of the program counter at each execution step. We specify this
constraint with the sub-formula,
p(n) P −2 P −1
^ ^ ^
F1 = If(pij , ¬pij ′ ).
i=0 j=0 j ′ =j+1

Because P is a constant depending only on the problem X, this sub-formula

can be constructed in O(p(n)) time.
We now need to specify some initialization constraints. The first is sim-
ply that the program counter has an initial value of 0. We specify this
constraint with the sub-formula,

F2 = p0,0 .

Clearly, this sub-formula can be constructed in O(1) time.

We also need to specify that the variables x[1..n] encode the input string
x. Let the kth bit of x be denoted by x[k] for 1 ≤ k ≤ n. We then construct

F3 = Eq(x, x).

Because x and x are arrays of n elements, this sub-formula can be con-

structed in O(n) time.
In addition, we need to specify that all bits in both input streams are
initially unread. Notice that for each i, we have defined a variable x bi [0], but
there is no x[0]. The purpose of these variables is so that whenever x[k] is
the next bit to be read, x bi [k − 1] is false and x
bi [k] is true. In order to enforce
this constraint initially, we construct
n
^
F4 = ¬b
x0 [0] ∧ x
b0 [k].
k=1

Clearly, this sub-formula can be constructed in O(n) time.

We need a similar specification for φ; however, we don’t know the exact
length of φ. We want φb0 [k] to be true iff 1 ≤ k ≤ |φ|. Thus, we need φb0 [0] to
be false. Then as k increases, φb0 [k] may be true for a while, but as soon as
we encounter a false φb0 [k], these variables must be false for all greater values
of k. We can enforce this constraint with the following sub-formula:
p(n)−1
^
F5 = ¬φb0 [0] ∧ If(¬φb0 [k], ¬φb0 [k + 1]).
k=1
CHAPTER 16. N P-COMPLETENESS 549

We can clearly construct this sub-formula in O(p(n)) time.

As the final initialization specification, we need to specify that all mem-
ory locations are initially 0. We therefore construct
4p(n) p(n)
^ ^
F6 = ¬v0,j [k].
j=1 k=1

Clearly, we can construct this sub-formula in O(p2 (n)) time.

We now need a constraint specifying that at some point, a Halt(1)
instruction is executed. Let A be the set of program locations at which a
Halt(1) instruction appears. We then construct
p(n)−1
_ _
F7 = pij .
i=0 j∈A

Because the size of A depends only on the problem X, this sub-formula can
be constructed in O(p(n)) time.
To complete the formula, we need constraints specifying the correct be-
havior of M . To this end, we will construct one sub-formula for each instruc-
tion in the program of M . These sub-formulas will depend on the particular
instruction. Let 0 ≤ q < P , where P is the number of instructions in
the program. In what follows, we will describe how the sub-formula Fq′ is
constructed depending on the instruction at program location q.
Regardless of the specific instruction, the sub-formula will have the same
general form. In each case, Fq′ must specify that some particular behavior
occurs whenever the program counter has a value of q. Fq′ will therefore
have the following form:
p(n)
^
Fp′ = If(pi−1,q , ψq (i)), (16.1)
i=1

where ψq (i) is a predicate specifying the result of executing the ith instruc-
tion.
Each ψq (i) will be a conjunction of predicates, each specifying some
aspect of the result of executing the ith instruction. In particular, ψq (i) will
be the conjunction of the following predicates:
• Uq (i), which specifies how the memory locations are updated;
• Eq (i), which specifies what memory locations must be represented in
F in order for this instruction to be simulated (this specification is
needed to prevent Uq (i) from being vacuously satisfied);
CHAPTER 16. N P-COMPLETENESS 550

• Iq (i), which specifies which input bits remain unread; and

• Pq (i), which specifies the new value of the program counter.

ψq (i) is then defined as follows:

ψq (i) = Uq (i) ∧ Eq (i) ∧ Iq (i) ∧ Pq (i). (16.2)

There are some instances of the above predicates that occur for more
than one type of instruction.
• If the instruction at location q is not an Input instruction, then

Iq (i) = Eq(b bi−1 ) ∧ Eq(φbi , φbi−1 ).

xi , x (16.3)

• If this instruction is neither a Goto, an IfLeq, nor a Halt, then

Pq (i) = pi,q+1 . (16.4)

• If this instruction is either a Goto or a Halt, then

4p(n)
^
Uq (i) = Eq(vij , vi−1,j ), (16.5)
j=1

and
Eq (i) = true. (16.6)

In what follows, we will define the remaining predicates for several of the
possible instructions. We leave the remaining cases as exercises.
Let us first consider an instruction Load(n, l). Because l is the only
memory location that is accessed, we can define
4p(n)
_
Eq (i) = Eq(aj , l).
j=1

Because its value changes to n, we can define

4p(n)
^
Uq (i) = IfElse(Eq(aj , l), Eq(vij , n), Eq(vij , vi−1,j )).
j=1

Note that the above expression specifies that every vij such that aj = l has
its value changed to n.
CHAPTER 16. N P-COMPLETENESS 551

Let us now compute the time needed to construct the resulting sub-
formula Fq′ . Because the arrays aj and l each contain p(n) elements, Eq (i, j)
can be constructed in O(p2 (n)) time. It is not hard to verify that Uq (i) can
be constructed in O(p2 (n)) time as well. Clearly, Pq (i) as defined in (16.4)
can be constructed in O(1) time. Finally, Iq (i) as defined in (16.3) can be
constructed in O(p(n)) time. Thus, ψq (i) can be constructed in O(p2 (n))
time. The sub-formula Fq′ can therefore be constructed in O(p3 (n)) time.
We can handle an instruction Load(n, ∗l) in a similar way, but using
the Ind abbreviation. Thus, we define
4p(n)
_
Eq (i) = Ind(i − 1, j, l)
j=1

and
4p(n)
^
Uq (i) = If(Ind(i − 1, j, l), Eq(vij , n), Eq(vij , vi−1,j )).
j=1

In this case, Eq (i) and Uq (i) can be constructed in O(p3 (n)) time, so that
Fq′ can be constructed in O(p4 (n)) time.
Let us now consider an instruction IfLeq(l1 , l2 , q ′ ). Because the memory
locations l1 and l2 are referenced, we define
4p(n) 4p(n)
_ _
Eq = Eq(aj , l1 ) ∧ Eq(aj , l2 ).
j=1 j=1

This statement will cause the program counter to be set to q ′ if the

value stored at l1 is less than or equal to the value stored at l2 ; otherwise,
the program counter will be set to q + 1. We first observe that for natural
numbers v1 and v2 , v1 ≤ v2 iff v2 − v1 ≥ 0. We can therefore use the variable
di to record |v2 − v1 | using the Sum abbreviation as follows:
4p(n)
^
Uq (i) = Eq(vij , vi−1,j ) ∧
j=1
4p(n) 4p(n)
^ ^
If(Eq(aj , l1 ) ∧ Eq(aj ′ , l2 ),
j=1 j ′ =1

Sum(vij , di , vij ′ ) ∨ Sum(vij ′ , di , vij )).

CHAPTER 16. N P-COMPLETENESS 552

We can now define Pq (i) as follows:


4p(n) 4p(n)
_ _
Pq (i) = IfElse  Eq(aj , l1 ) ∧ Eq(aj ′ , l2 ) ∧ Sum(vij , di , vij ′ ),
j=1 j ′ =1


piq′ , pi,q+1  .

Eq (i) can be constructed in O(p2 (n)) time, and both Uq (i) and Pq (i) can
be constructed in O(p3 (n)) time. Furthermore, Iq (i) as given in (16.3) can
be constructed in O(p(n)) time. The total time needed to construct Fq′ is
therefore in O(p4 (n)).
Finally, let us consider a Halt instruction. For a Halt instruction, we
have already defined Iq (i) (16.3), Uq (i) (16.5), and Eq (i) (16.6). To define
Pq (i), we need to specify that for all i′ > i, each pi′ j is false:
p(n) P −1
^ ^
Pq (i) = ¬pi′ j .
i′ =i+1 j=0

Because P is a constant depending only on X, Pq (i) can be constructed

in O(p(n)) time. Furthermore, Iq (i) can be constructed in O(p(n)) time,
Uq (i) can be constructed in O(p2 (n)) time, and Eq (i) can be constructed in
O(1) time. The sub-formula Fq′ can therefore be constructed in O(p3 (n))
time.
We leave it as exercises to show that the sub-formula Fq′ can be con-
structed for each of the other cases in a time in O(p5 (n)). We now define
the formula F as the conjunction of all of the sub-formulas:
7
^ P^
−1
F= Fq ∧ Fq′ .
q=1 q=0

Because P is a constant depending only on X, F can be constructed in

O(p5 (n)) time.
We must now show that F is satisfiable iff there is some φ ∈ B such that
M executes a Halt(1) instruction on input (x, φ). Suppose F is satisfiable.
Let us fix some satisfying assignment to the variables of F. Because F5
must be true by this assignment, there must be some k, 1 ≤ k ≤ p(n), such
b ′ ] is true for 1 ≤ k ′ ≤ k and φ[k
that φ[k b ′ ] is false for k < k ′ ≤ p(n). Let
φ = φ[1..k]. By the above construction, for 1 ≤ i ≤ p(n) and 0 ≤ j < P , pij
CHAPTER 16. N P-COMPLETENESS 553

is true iff the ith instruction executed by M on input (x, φ) is the instruction
at program location j. Finally, because F7 must be satisfied, one of these
instructions must be a Halt(1) instruction.
Now suppose that for some φ ∈ B, M executes a Halt(1) instruction
on input (x, φ). By our choice of the polynomial p(n), we can assume that
|φ| ≤ p(|x|). Let us now set x = x and φ[1..|φ|] = φ. We will also set
b
φ[k] = true for 1 ≤ k ≤ |φ| and φ[k] b = false for |φ| < k ≤ p(n). We
can clearly assign truth values to the variables in the sub-formulas Fq for
2 ≤ q ≤ 6 so that all of these sub-formulas are satisfied. By the above
construction, we can then assign truth values to the variables in each of the
sub-formulas Fq′ for 1 ≤ q ≤ p(n) so that these formulas, along with F1 , are
satisfied. Such an assignment will yield pij = true iff pj is the ith instruction
executed by M on input (x, φ). Because M executes a Halt(1) instruction
on this input, F7 must also be satisfied. Therefore, F is satisfied by this
assignment.
We have therefore shown that X ≤pm Sat. Because X can be any prob-
lem in N P, it follows that Sat is N P-hard. Because Sat ∈ N P, it follows
that Sat is N P-complete.

16.9 Summary
The N P-complete problems comprise a large class of decision problems for
which no polynomial-time algorithms are known. Furthermore, if a poly-
nomial time algorithm were found for any one of these problems, we would
be able to construct polynomial-time algorithms for all of them. For this
reason, along with many others that are beyond the scope of this book, we
tend to believe that none of these problems can be solved in polynomial
time. Note, however, that this conjecture has not been proven. Indeed,
this question — whether P = N P — is the most famous open question in
theoretical computer science.
Proofs of N P-completeness consist of two parts: membership in N P
and N P-hardness. Without knowledge of any N P-complete problems, it is
quite tedious to prove a problem to be N P-hard. However, given one or
more N P-complete problems, the task of proving additional problems to be
N P-hard is greatly eased using polynomial-time many-one reductions.
Some general guidelines for finding a reduction from a known N P-
complete problem to a problem known to be in N P are as follows:

• Look for a known N P-complete problem that has similarities with the
problem in question.
CHAPTER 16. N P-COMPLETENESS 554

• If all else fails, try reducing from 3-Sat.

• Look at the proofs of membership in N P and try to transform proofs

φ ∈ B for the known N P-complete problem to proofs φ′ ∈ B for the
problem in question.

Large numbers play an interesting role in the theory of N P-completeness.

In particular, some problems become N P-hard simply because very large
numbers can be given as input using comparatively few bits. The definitions
of strong N P-completeness and strong N P-hardness exclude such problems.
A refinement of polynomial many-one reducibility, namely, pseudopolyno-
mial reducibility, is used to prove strong N P-hardness.

16.10 Exercises
Exercise 16.1 Prove that if X, Y , and Z are decision problems such that
X ≤pm Y and Y ≤pm Z, then X ≤pm Z.

Exercise 16.2 Adapt BoolEval (Figure 16.2) to evaluate a CNF expres-

sion F in O(|F|) time.

Exercise 16.3 Implement the reduction from CSat to 3-Sat, as outlined

in Section 16.4, to run in O(n) time, where n is the size of the given CNF
formula.

Exercise 16.4 Give an algorithm that takes as input an undirected graph

G, a natural number k, and an array A[0..m−1] of booleans, and determines
whether A denotes a vertex cover of G with size k. Your algorithm must
run in O(n + a) time, where n and a are the number of vertices and edges,
respectively, of G. For the purpose of analyzing the running time, you may
assume that G is implemented as a ListGraph.

Exercise 16.5 Implement the reduction from 3-Sat to VC, as outlined in

Section 16.5, to run in O(m + n) time, where m is the number of variables
and n is the number of clauses in the given 3-CNF formula.

* Exercise 16.6 Let NotAllEqual-3-Sat be the problem of deciding,

for a given 3-CNF formula f , whether there is an assignment of boolean
variables such that each clause in f contains at least one true literal and at
least one false literal. Prove that this problem is N P-complete.
CHAPTER 16. N P-COMPLETENESS 555

Exercise 16.7 A clique is a complete undirected graph — i.e., a graph such

that for every pair of distinct vertices u and v, {u, v} is an edge. The clique
problem (Clique) is the problem of deciding, for a given undirected graph
G and natural number k, if G has a subgraph that is a clique with k vertices.
Show that Clique is N P-complete.

Exercise 16.8 Two graphs G and G′ are said to be isomorphic if the ver-
tices of G can be renamed so that the resulting graph is G′ . Given two
graphs G and G′ and a natural number k, we wish to decide whether G and
G′ contain isomorphic subgraphs with k vertices. Show that this problem is
N P-complete. You may use the result of Exercise 16.7.

** Exercise 16.9 A Hamiltonian cycle in a graph G is a cycle that contains

each vertex in G exactly once. Prove that the problem of deciding whether
a given undirected graph contains a Hamiltonian cycle is N P-complete.
[Hint: Reduce VC to this problem.]

Exercise 16.10 Repeat Exercise 16.9 for directed graphs. You may use the
result of Exercise 16.9.

Exercise 16.11 As was defined in Exercise 10.34, a Hamiltonian path in

a graph G is a simple path that contains each vertex in G exactly once.
Prove that problem of deciding whether a given undirected graph has a
Hamiltonian path is N P-complete. You may use the results of Exercises
16.9 and 16.10.

Exercise 16.12 Repeat Exercise 16.11 for directed graphs. You may use
the results of Exercises 16.9, 16.10, and 16.11.

Exercise 16.13 Given a directed graph G = (V, E) and a positive integer

k, we wish to determine whether there is a subset V ′ ⊆ V of size k such that
every cycle in G contains at least one vertex in V ′ . Show that this problem
is N P-complete.

* Exercise 16.14 Given an undirected graph G = (V, E) and a positive

integer k, we wish to decide whether V can be partitioned into two disjoint
sets, V1 and V2 , such that V1 contains exactly k vertices and for every vertex
u ∈ V2 , there is a vertex v ∈ V1 such that {u, v} ∈ E. Show that this
problem is N P-complete.
CHAPTER 16. N P-COMPLETENESS 556

Exercise 16.15 Give an algorithm that takes an instance (X, Y, Z, W ) of

3DM and a bit string φ and determines whether φ is a proof that (X, Y, Z, W )
has a matching, as defined in Section 16.6. You may assume that (X, Y, Z, W )
is represented by a natural number m and an array W [1..n] of triples of the
form (i, j, k) such that m ≤ n and each i, j, and k is a positive integer no
greater than m. Your algorithm should run in O(m2 lg n) time.

Exercise 16.16 Given a finite sequence of finite sets and a natural number
k, we wish to decide whether the sequence contains at least k mutually
disjoint sets. Show that this problem is N P-complete.

Exercise 16.17 Prove that Part, as defined in Section 16.7, is in N P.

Exercise 16.18 Suppose we modify the 0-1 knapsack problem (see Section
12.4) by including a target value V as an additional input. The problem
then is to decide whether there is a subset of the items whose total weight
does not exceed the weight bound W and whose total value is at least V .
Prove that this problem is N P-complete.

Exercise 16.19 Suppose we are given a set of items a1 , . . . , an , each having

a positive integer weight wi , and positive integers k and W . We wish to
decide whether the items can be partitioned into k mutually disjoint subsets
A1 , . . . , Ak , such that
 2
k
X X
 wi  ≤ W.
j=1 ai ∈Aj

Show that this problem is N P-complete.

** Exercise 16.20 Suppose we are given a sequence S1 , . . . , Sn of finite

sets. We wish to partition
[n
Si
i=1

into two disjoint sets S and S′

such that for 1 ≤ i ≤ n, Si 6⊆ S and Si 6⊆ S ′ .
Show that this problem is N P-complete. [Hint: Reduce 3-Sat to this
problem.]

** Exercise 16.21 Suppose we are given an undirected graph G = (V, E)

with exactly 3k vertices. We wish to partition V into k disjoint subsets such
that each subset forms a path of length 2 in G. Show that this problem is
N P-complete. [Hint: Reduce 3DM to this problem.]
CHAPTER 16. N P-COMPLETENESS 557

** Exercise 16.22 Given a directed graph G = (V, E), we wish to decide

whether each vertex vi ∈ V can be assigned a label Li ∈ N such that Li is
the least natural number that is not in the set

{Lj | (vi , vj ) ∈ E}.

Show that this problem is N P-complete. [Hint: Reduce 3-Sat to this

problem.]

** Exercise 16.23 Show that the problem of deciding whether a given

undirected graph has a 3-coloring is N P-complete. (See Exercise 13.12 for
the definition of a 3-coloring.) [Hint: Reduce 3-Sat to this problem.]

Exercise 16.24 Show that the problem of deciding whether a given undi-
rected graph has a k-coloring is N P-complete for each fixed k ≥ 4. You
may use the result of Exercise 16.23.

** Exercise 16.25 Certain aspects of the board game Axis and AlliesTM
can be modeled as follows. The game is played on an undirected graph.
The playing pieces include fighters and aircraft carriers, each of which has
a natural number range. These pieces are each assigned to a vertex of
the graph. Each vertex may be assigned any number of pieces. A combat
scenario is valid if it is possible to move each piece to a new vertex (possibly
the same one) so that

• for each move, the distance (i.e., number of edges) from the starting
vertex to the ending vertex is no more than the range of piece moved;
and

• after the pieces are moved, each vertex has no more than twice as
many fighters as aircraft carriers.

Prove that the problem of determining whether a combat scenario is valid

is N P-complete.

Exercise 16.26 Let k-Part be as defined in Section 16.7.

a. Prove that k-Part ∈ N P for all k ≥ 1.

** b. Prove that 3-Part is N P-complete in the strong sense. [Hint: Show

that 4-Part ≤pp
m 3-Part.]

c. Prove that 2-Part ∈ P.

CHAPTER 16. N P-COMPLETENESS 558

Exercise 16.27 The bin packing problem (BP) is to decide whether a given
set of items, each having a weight wi , can be partitioned into k disjoint sets
each having a total weight of at most W , where k and W are given positive
integers. Show that BP is N P-complete in the strong sense.

Exercise 16.28 Suppose we are given a complete undirected graph G with

positive integer edge weights and a positive integer k. The traveling sales-
person problem (TSP) is to determine whether there is a Hamiltonian cycle
in G with total weight no more than k. Show that this problem is N P-
complete in the strong sense. You may use the result of Exercise 16.9.

* Exercise 16.29 We are given a set of n tasks, each having an execution

time ei ∈ N, a ready time ri ∈ N, and a deadline di ∈ N. We wish to decide
whether there is a nonpreemptive schedule that meets the constraints of
all of the tasks. In other words, we wish to know if there is a function
f : [1..n] → N such that for 1 ≤ i ≤ n,
• ri ≤ f (i);
• f (i) + ei ≤ di ; and
• for 1 ≤ j ≤ i and j 6= i, either f (j) + ej ≤ f (i) or f (j) ≥ f (i) + ei .
Show that this problem is N P-complete in the strong sense.

* Exercise 16.30 We are given an undirected graph G = (V, E), a se-

quence hw1 , . . . , w|E| i of natural numbers, and a positive integer k. We wish
to decide whether there is a 1-1 function f : E → {1, . . . , |E|} such that if
each edge e ∈ E is assigned a length of wf (e) , then for every pair of vertices
u and v, there is a path from u to v with length at most k. Prove that this
problem is N P-complete in the strong sense.

Exercise 16.31 Define the predicate Pq (i) for the case in which the in-
struction at location q is Goto(q ′ ). Show that the resulting sub-formula Fq′
can be constructed in O(p(n)) time.

Exercise 16.32 Define the predicates Eq (i) and Uq (i) for the case in which
the instruction at location q is Copy(l1 , l2 ). Show that the resulting sub-
formula Fq′ can be constructed in O(p4 (n)) time.

Exercise 16.33 Define the predicates Eq (i) and Uq (i) for the case in which
the instruction at location q is Copy(∗l1 , ∗l2 ). Show that the resulting sub-
formula Fq′ can be constructed in O(p5 (n)) time.
CHAPTER 16. N P-COMPLETENESS 559

Exercise 16.34 Define the predicates Eq (i), Uq (i), and Pq (i) for the case
in which the instruction at location q is IfLeq(∗l1 , ∗l2 , q ′ ). Show that the
resulting sub-formula Fq′ can be constructed in O(p5 (n)) time.

Exercise 16.35 Define the predicates Eq (i) and Uq (i) for the case in which
the instruction at location q is Add(∗l1 , ∗l2 ). Show that the resulting sub-
formula Fq′ can be constructed in O(p5 (n)) time.

Exercise 16.36 Define the predicates Eq (i) and Uq (i) for the case in which
the instruction at location q is Subtract(∗l1 , ∗l2 ). Show that the resulting
sub-formula Fq′ can be constructed in O(p5 (n)) time.

Exercise 16.37 Define the predicates Eq (i) and Uq (i) for the case in which
the instruction at location q is Shift(∗l). Show that the resulting sub-
formula Fq′ can be constructed in O(p4 (n)) time.

* Exercise 16.38 Define the predicates Iq (i), Eq (i), and Uq (i) for the case
in which the instruction at location q is Input(1, ∗l). Show that the resulting
sub-formula Fq′ can be constructed in O(p5 (n)) time.

16.11 Chapter Notes

N P-completeness was introduced by Cook [23], who proved that Sat and
CSat are N P-complete. Karp [74] then demonstrated the importance of
this topic by proving N P-completeness of 21 problems, including VC, 3DM,
Part, and the problems described in Exercises 16.7, 16.9, 16.10, 16.13,
16.16, 16.18, and 16.23. The original definition of N P was somewhat dif-
ferent from the one given here — it was based on nondeterministic Turing
machines, rather than on algorithms or RAMs. The definition given in Sec-
tion 16.1 is based on a definition given by Brassard and Bratley [18]. All of
these definitions are equivalent.
The notion of strong N P-completeness was introduced by Garey and
Johnson [51]. They provided the definitions of strong N P-completeness,
pseudopolynomial algorithms, and pseudopolynomial reductions. They had
earlier given N P-completeness proofs for k-Part for k ≥ 3 [49] and for the
problem described in Exercise 16.29 [50]. As it turned out, their reductions
were pseudopolynomial. Their book on N P-completeness [52] is an excellent
resource.
Exercise 16.20 is solved by Lovasz [86]. Exercise 16.21 is solved by Kirk-
patrick and Hell [75]. Exercise 16.22 is solved by van Leeuwen [107]. The
CHAPTER 16. N P-COMPLETENESS 560

solution to Exercise 16.30 is attributed to Perl and Zaks by Garey and John-
son [52].
Axis and AlliesTM (mentioned in Exercise 16.25) is a registered trade-
mark of Hasbro, Inc.
Chapter 17

Approximation Algorithms

In Chapter 16, we examined decision problems that appear to be intractable.

As we might expect, there are other types of problems that are also in-
tractable. For example, consider the following version of the vertex cover
problem (cf. Section 16.5). Instead of being given a target size as input,
we are given simply an undirected graph from which we must find a vertex
cover of minimum size. Let us call this optimization problem VCOpt. We
can easily reduce VC to VCOpt, though because an optimization problem
is not a decision problem, the reduction is not a many-one reduction. How-
ever, it is clear that if VCOpt has a polynomial-time solution, then so does
VC. We can therefore conclude that unless P = N P, VCOpt cannot be
solved in polynomial time.
With hard optimization problems, however, it may not be necessary
to obtain an exact solution. In this chapter, we will explore techniques
for obtaining approximate solutions to hard optimization problems. We
will see that for some problems, we can obtain reasonable approximation
algorithms. On the other hand, we will use the theory of N P-completeness
to show limitations to these techniques. Before looking at specific problems,
however, we must first extend some of the definitions from Chapter 16 to
include problems other than decision problems.

17.1 Polynomial Turing Reducibility

In this section, we will extend the definition of N P-hardness to include
problems that are not decision problems. As we have already observed, we
can reduce VC to VCOpt in a way that proves that VCOpt cannot be
solved in polynomial time unless P = N P; however, this reduction is not

561
CHAPTER 17. APPROXIMATION ALGORITHMS 562

a many-one reduction. We will therefore define a new kind of reducibility

that will include this kind of reduction.
Suppose we can reduce a problem X to another problem Y in such a
way that for some polynomial p(n) and any instance x of X:

• the time required to obtain a solution for x, excluding any time needed
to solve instances of Y , is bounded above by p(|x|); and

• the values of all variables are bounded above by p(|x|).

We then say that X is polynomially Turing reducible to Y , or X ≤pT Y . Note

that if X ≤pm Y , then clearly X ≤pT Y .
It is easily seen that VC ≤pT VCOpt. More generally, consider any
minimization problem Y with objective function f . We can construct a
decision problem X from Y by adding an additional natural number input,
k. X is simply the set of all pairs (y, k) such that y is an instance of Y with
a candidate solution s for which f (s) ≤ k. It is easily seen that X ≤pT Y , for
if we can find the minimum value of f for a given instance y, then we can
quickly decide whether there is a candidate solution s for which f (s) ≤ k.
We can now extend the notion of N P-hardness by saying that Y is N P-
hard with respect to Turing reducibility if for every X ∈ N P, X ≤pT Y .
Note that we have not modified the definition of N P — it contains only
decision problems. As a result, it makes no sense to extend the definition
of N P-completeness beyond decision problems. Because N P-hardness with
respect to Turing reducibility is the natural version of N P-hardness to use
when discussing optimization problems, we will simply refer to this version
as “N P-hardness” in this chapter. The following theorem can now be shown
in a manner similar to the proof of Theorem 16.2.

Theorem 17.1 If X ≤pT Y and there is a deterministic polynomial-time

algorithm for solving Y , then there is a deterministic polynomial-time algo-
rithm for solving X.
The above theorem shows that if there is a deterministic polynomial-time
algorithm for solving an N P-hard problem Y , then there is a deterministic
polynomial-time algorithm for deciding every problem in N P. We therefore
have the following corollary, which highlights the importance of the notion
of N P hardness with respect to Turing reducibility.

Corollary 17.2 If Y is N P-hard and there is a deterministic polynomial-

time algorithm for solving Y , then P = N P.
CHAPTER 17. APPROXIMATION ALGORITHMS 563

17.2 Knapsack
The first problem we will examine is the 0-1 knapsack problem, as defined
in Section 12.4. As is suggested by Exercise 16.18, the associated decision
problem is N P-complete; hence, the optimization problem is N P-hard.
Consider the following greedy strategy for filling the knapsack. Suppose
we take an item whose ratio of value to weight is maximum. If this item
won’t fit, we discard it and solve the remaining problem. Otherwise, we
include it in the knapsack and solve the problem that results from removing
this item and decreasing the capacity by its weight. We have thus reduced
the problem to a smaller instance of itself. Clearly, this strategy results
in a set of items whose total weight does not exceed the weight bound.
Furthermore, it is not hard to implement this strategy in O(n lg n) time,
where n is the number of items.
Because the problem is N P-hard, we would not expect this greedy strat-
egy to yield an optimal solution in all cases. What we need is a way to
measure how good an approximation to an optimal solution it provides. In
order to motivate an analysis, let us consider a simple example. Consider
the following instance consisting of two items:
• The first item has weight 1 and value 2.
• The second item has weight 10 and value 10.
• The weight bound is 10.
The value-to-weight ratios of the two items are 2 and 1, respectively. The
greedy algorithm therefore takes the first item first. Because the second item
will no longer fit, the solution provided by the greedy algorithm consists of
the first item by itself. The value of this solution is 2. However, it is easily
seen that the optimal solution is the second item by itself. This solution has
a value of 10.
A common way of measuring the quality of an approximation is to form
a ratio with the actual value. Specifically, for a maximization problem, we
define the approximation ratio of a given approximation to be the ratio of
the optimal value to the approximation. Thus, the approximation ratio for
the above example is 5. For a minimization problem, we use the reciprocal
of this ratio, so that the approximation ratio is always at least 1. As the ap-
proximation ratio approaches 1, the approximation approaches the optimal
value.
Note that for a minimization problem, the approximation ratio cannot
take a finite value if the optimal value is 0. For this reason, we will restrict
CHAPTER 17. APPROXIMATION ALGORITHMS 564

our attention to optimization problems whose optimal solutions always make

their objective functions positive. In addition, we will restrict our attention
to problems whose objective functions have integer values for all candidate
solutions.
We would like to show some fixed upper bound on the approximation
ratio of our greedy algorithm. However, we can modify the above example
by replacing 10 with an arbitrarily large x in order to achieve an arbitrarily
large approximation ratio of x/2. Thus, this approximation algorithm can
perform arbitrarily poorly.
With a bit more work, however, we can modify this algorithm so that it
has a bounded approximation ratio. Specifically, we find n different packings
and take the one with the highest value. For the ith packing, we take the
ith item first, then apply the greedy strategy to finish the packing. Thus,
we expend additional work in making sure that we get started correctly.
The algorithm is shown in Figure 17.1. For simplicity, we assume that the
items are given in nondecreasing order of value-to-weight ratios, and that no
item’s weight exceeds the weight bound. It is easily seen that this algorithm
produces a solution in Θ(n2 ) time. The following theorem shows how well
it approximates an optimal solution in the worst case.

Theorem 17.3 KnapsackApprox yields an approximation ratio of at

most 2 on all inputs that satisfy the precondition. Furthermore, for ev-
ery ǫ ∈ R>0 , there is some input for which the approximation ratio is at
least 2 − ǫ.

Proof: We begin by showing the lower bound. Let ǫ ∈ R>0 , and without
loss of generality, assume ǫ < 1. We first define the weight bound as

4
W =2 .
ǫ

We then construct the following set of three items:

• The first item has weight 1 and value 2.

• The second and third items each have a weight and value of W/2.

The optimal solution clearly consists of the second and third items. This
solution has value W . Each iteration of the outer loop of KnapsackAp-
prox yields a solution containing the first item and one of the other two.
The solution returned by this algorithm therefore has a value of W/2 + 2.
CHAPTER 17. APPROXIMATION ALGORITHMS 565

Figure 17.1 An approximation algorithm for the 0-1 knapsack problem

Precondition: W is a positive Nat, n ≥ 1, and w[1..n] and v[1..n] are

arrays of positive Nats such that for 1 ≤ i ≤ j ≤ n, v[i]/w[i] ≥ v[j]/w[j]
and w[i] ≤ W .
Postcondition: Returns an array A[1..n] of Bools such that if

S = {i | 1 ≤ i ≤ n, A[i] = true},

then X
w[i] ≤ W.
i∈S

KnapsackApprox(W, w[1..n], v[1..n])

maxValue ← 0
for i ← 1 to n
A ← new Array[1..n]
for j ← 1 to n
A[j] ← false
A[i] ← true; value ← v[i]; weight ← w[i]
for j ← 1 to n
if j 6= i and weight + w[j] ≤ W
weight ← weight + w[j]; value ← value + v[j]; A[j] ← true
if value > maxValue
M ←A
return M
CHAPTER 17. APPROXIMATION ALGORITHMS 566

The approximation ratio is therefore

W 2W
W
=
2 +2 W +4
8
=2−
W +4
8
=2−
2⌈4/ǫ⌉ + 4
8
≥2−
8/ǫ
= 2 − ǫ.

Now consider an arbitrary input to KnapsackApprox. For a given

solution X, let V (X) denote the value of X. Suppose KnapsackApprox
returns a solution A, and let S be an optimal solution. Let i be the index
of some element with maximum value in S, and consider iteration i of the
outer loop. Let Ai be the solution chosen by this iteration. We will show
Ai has an approximation ratio of at most 2. Because V (A) ≥ V (Ai ), the
theorem will follow.
In computing an upper bound V (S) − V (Ai ), we can ignore all items
that belong to S ∩ Ai . Suppose these common items have a total weight
of C. Suppose further that V (Ai ) < V (S). Then the greedy loop must
reject at least one element belonging to S. Let item k be the first element
from S to be rejected by the greedy loop in the construction of Ai . Then
the total weight of all items in Ai \ S chosen prior to item k is greater than
W − C − w[k]. Because their value-to-weight ratios are all at least v[k]/w[k],
their total value is greater than
v[k](W − C − w[k])
.
w[k]
The items in S \ Ai must have total weight at most W −C. Furthermore,
all of their value-to-weight ratios are at most v[k]/w[k]; hence their total
value is at most
v[k](W − C)
.
w[k]
We therefore have
v[k](W − C) v[k](W − C − w[k])
V (S) − V (Ai ) < −
w[k] w[k]
= v[k].
CHAPTER 17. APPROXIMATION ALGORITHMS 567

Because items i and k both belong to S and v[i] ≥ v[k], v[k] ≤ V (S)/2.
We therefore have

V (S) − V (Ai ) < V (S)/2

V (S) < 2V (Ai )
V (S)/V (Ai ) < 2
V (S)/V (A) < 2.

Though we have a bounded approximation ratio, an approximation ratio

of 2 may seem unsatisfactory, as in the worst case we may only achieve half
the actual maximum value. It turns out that we can improve the approxima-
tion ratio by examining all pairs of items, then using the greedy algorithm
to complete each of these packings. More generally, we can achieve an upper
bound of 1 + k1 by examining all sets of k items and completing each packing
using the greedy algorithm. (If there are fewer than k items, we simply
do an exhaustive search and return the optimal solution.) The proof is a
straightforward generalization of the proof of Theorem 17.3 — the details
are left as an exercise.
It is not hard to see that the algorithm outlined above can be imple-
mented to return a solution in Θ(nk+1 ) time. If k is a fixed constant, the
running time is polynomial. We therefore have an infinite sequence of al-
gorithms, each of which is polynomial, such that if an approximation ratio
of 1 + ǫ is needed (for some positive ǫ), then one of these algorithms will
provide such an approximation. Such a sequence of algorithms is called a
polynomial approximation scheme.
Although each of the algorithms in the above sequence is polynomial
in the length of the input, it is somewhat unsatisfying that to achieve an
approximation ratio of 1 + k1 , a running time in Θ(nk+1 ) is required. We
would be more satisfied with a running time that is polynomial in both n
and k. More generally, suppose we have an approximation algorithm that
takes as an extra input a natural number k such that for any fixed k, the
algorithm yields an approximation ratio of no more than 1 + k1 . Suppose
further that this algorithm runs in a time polynomial in k and the length
of its input. We call such an algorithm a fully polynomial approximation
scheme.
We can obtain a fully polynomial approximation scheme for the 0-1 knap-
sack problem using one of the dynamic programming algorithms suggested
in Section 12.4. The algorithm based on recurrence (12.5) on page 403 runs
CHAPTER 17. APPROXIMATION ALGORITHMS 568

in Θ(nV ) time, where n is the number of items and V is the sum of their
values. We can make V as small as we wish by replacing each value v by
⌊v/d⌋ for some positive integer d. If some of the values become 0, we re-
move these items. Observe that because we don’t change any weights or the
weight bound, any packing for the new instance is a packing for the origi-
nal. However, because we take the floor of each v/d, the optimal packing
for the new instance might not be optimal for the original. The smaller we
make d, the better our approximation, but the less efficient our dynamic
programming algorithm.
In order to determine an appropriate value for d, we need to analyze
the approximation ratio of this approximation algorithm. Let S be some
optimal set of items. The optimal value is then
X
V∗ = vi .
i∈S

With the modified values, this packing has a value of

X j vi k X vi − d
≥
d d
i∈S i∈S
X vi X
= − 1
d
i∈S i∈S
V∗
≥ − n.
d
If we remove from S the items whose new values are 0, we obtain a pack-
ing for the revised instance with same value as above. Because the dynamic
programming algorithm selects an optimal packing for the revised instance,
it will yield a packing with a value at least this large. If we substitute the
original values into the packing chosen by the dynamic programming algo-
rithm, we obtain a value of at least V ∗ − nd. The approximation ratio is
therefore at most
V∗
.
V ∗ − nd
We need to ensure that the approximation ratio is at most 1+ k1 for some
CHAPTER 17. APPROXIMATION ALGORITHMS 569

positive integer k. We therefore need

V∗ 1
≤1+
V ∗ − nd k
V ∗ − nd
V ∗ ≤ V ∗ − nd +
k
V ∗ − (k + 1)nd
0≤
k
0 ≤ V ∗ − (k + 1)nd
V∗
d≤ .
(k + 1)n

Let v be the largest value of any item in the original instance. Assuming
that no item’s weight exceeds the weight bound, we can conclude that V ∗ ≥
v. Thus, if v ≥ (k + 1)n, we can satisfy the above inequality by setting d to
⌊v/((k + 1)n)⌋. However, if v < (k + 1)n, this value becomes 0. In this case,
we can certainly set d to 1, as the dynamic programming algorithm would
then give the optimal solution. We therefore set

v
d = max ,1 .
(k + 1)n

We can clearly compute the scaled values in O(n) time. If v ≥ 2(k + 1)n,
the sum of the scaled values is no more than
nv nv
=j k
d v
(k+1)n
nv
≤ v−(k+1)n
(k+1)n
(k + 1)n2 v
=
v − (k + 1)n
(k + 1)n2 v
≤
v/2
= 2(k + 1)n2 .

In this case, the dynamic programming algorithm runs in O(kn3 ) time.

If v < 2(k + 1)n, then d = 1, so that we use the original values. In this
case, the sum of the values is no more than

nv < 2(k + 1)n2 ,

CHAPTER 17. APPROXIMATION ALGORITHMS 570

so that again, the dynamic programming algorithm runs in O(kn3 ) time.

Thus, the total running time of the approximation algorithm is in O(kn3 ).
Because this running time is polynomial in k and n, and because the ap-
proximation ratio is no more than 1 + k1 , this algorithm is a fully polynomial
approximation scheme.

17.3 Bin Packing

Exercise 16.27 introduced the bin packing problem as a decision problem.
Its input consists of a set of items, each having a positive integer weight wi ,
a positive integer weight bound W , and a positive integer k. The question
we ask is whether the items can be partitioned into k disjoint subsets, each
having a total weight of no more than W . The corresponding optimization
problem does not include the input k, but instead asks for the minimum
number of subsets into which the items can be partitioned such that the
weight bound is satisfied. As is suggested by Exercise 16.27, the decision
problem BP is strongly N P-complete. As a result, it is easily seen that the
optimization problem is N P-hard in the strong sense.
Ideally, we would like to have a fully polynomial approximation scheme
for bin packing. However, the following theorem tells us that unless P =
N P, a fully polynomial approximation scheme does not exist.

Theorem 17.4 Let p(x, y) be an integer-valued polynomial, and let X be

an optimization problem whose optimal value on any input x is a natural
number bounded above by p(|x|, µ(x)). If there is a fully polynomial ap- Recall that |x|
denotes the
proximation scheme for X, then there is a pseudopolynomial algorithm for number of bits in
obtaining an optimal solution for X. the encoding of x
and µ(x) denotes
the maximum
Proof: The pseudopolynomial algorithm operates as follows. Given an value of any
input x, it first computes k = p(|x|, µ(x)). It then uses the fully polynomial integer encoded
within x.
approximation scheme to approximate a solution with an approximation
ratio bounded by 1 + k1 . Let V be the value of the approximation, and let
V ∗ be the value of an optimal solution. If the problem is a minimization
CHAPTER 17. APPROXIMATION ALGORITHMS 571

problem, we have
V 1
∗
≤1+
V k
V∗
V ≤V∗+
k
V∗
V −V∗ ≤
k
< 1.

Because both V and V ∗ are natural numbers and V ≥ V ∗ , we conclude that

V = V ∗ . Furthermore, because the fully polynomial approximation scheme
runs in time polynomial in |x| and p(|x|, µ(x)), it is a pseudopolynomial
algorithm.
An analogous argument applies to maximization problems.

Because the minimum number of bins needed is clearly no more than

the length of the input to the bin packing problem, Theorem 17.4 applies to
this problem. Indeed, the condition that the optimal solution is bounded by
a polynomial in the length of the input and the largest integer in the input
holds for most optimization problems. In these cases, if the given problem
is strongly N P-hard (as is bin packing), there can be no fully polynomial
approximation scheme unless P = N P.
If we cannot obtain a fully polynomial approximation scheme for bin
packing, we might still hope to find a polynomial approximation scheme.
However, the theory of N P-hardness tells us that this is also unlikely. In
particular, for a fixed positive integer k, let k-BP denote the problem of
deciding whether, for a given instance of bin packing, there is a solution
using at most k bins. It is easily seen that Part ≤pm 2-BP, so that 2-BP is
N P-hard. Now for a fixed positive real number ǫ, let ǫ-ApproxBP be the
problem of approximating a solution to a given instance of bin packing with
an approximation ratio of no more than 1+ǫ. We will now show that for any
ǫ < 1/2, 2-BP ≤pT ǫ-ApproxBP, so that ǫ-ApproxBP is N P-hard. As a
result, there can be no polynomial approximation scheme for bin packing
unless P = N P.

Theorem 17.5 For 0 < ǫ < 1/2, ǫ-ApproxBP is N P-hard.

Proof: As we noted above, we will show that 2-BP ≤pT ǫ-ApproxBP.

Given an instance of 2-BP, we first find an approximate solution with ap-
proximation ratio at most ǫ. If the approximate solution uses no more than
CHAPTER 17. APPROXIMATION ALGORITHMS 572

2 bins, then we can answer “yes”. If the approximate solution uses 3 or

more bins, then the optimal solution uses at least
3 3
>
1+ǫ 3/2
=2

bins. We can therefore answer “no”.

Ignoring the time needed to compute the approximation, this algorithm
runs in Θ(1) time. Therefore, 2-BP ≤pT ǫ-ApproxBP, and ǫ-ApproxBP
is N P-hard.

From Theorem 17.5, we can conclude that there is no approximation

algorithm for bin packing with approximation ratio less than 3/2 unless
P = N P. As a result, there can be no polynomial approximation scheme
for bin packing unless P = N P.
On the other hand, there do exist approximation algorithms which yield
approximation ratios that come close to the lower bound of 3/2 for bin
packing. The algorithm we will present here is a simple greedy strategy
known as first fit. For each item, we try each bin in turn to see if the item
will fit. If we find a bin in which the item fits, we place it in that bin;
otherwise, we place it in a new bin. The algorithm is shown in Figure 17.2.
This algorithm is easily seen to run in Θ(n2 ) time in the worst case. We will
now show that it yields an approximation ratio of at most 2.

Theorem 17.6 BinPackingFF yields an approximation ratio of no more

than 2 on all inputs that satisfy the precondition.

Proof: We will first show as an invariant of the for loop that at most one
bin is no more than half full. This clearly holds initially. Suppose it holds
at the beginning of some iteration. If w[i] > W/2, then no matter where
w[i] is placed, it cannot increase the number of bins that are no more than
half full. Suppose w[i] ≤ W/2. Then if there is a bin that is no more than
half full, w[i] will fit into this bin. Thus, the only case in which the number
of bins that are no more than half full increases is if there are no bins that
are no more than half full. In this case, the number cannot be increased to
more than one.
We conclude that the packing returned by this algorithm has at most
one bin that is no more than half full. Suppose this packing consists of k
bins. The total weight must therefore be strictly larger than (k − 1)W/2.
CHAPTER 17. APPROXIMATION ALGORITHMS 573

Figure 17.2 First-fit approximation algorithm for bin packing

Precondition: W is a positive Nat, and w[1..n] is an array of positive

Nats such that for 1 ≤ i ≤ n, w[i] ≤ W .
Postcondition: Returns an array B[1..k] of ConsLists of Nats i such
that 1 ≤ i ≤ n. For 1 ≤ i ≤ n, i occurs in exactly one ConsList in B[1..k].
For 1 ≤ i ≤ k, if S is the set of integers in B[i], then
X
wj ≤ W.
j∈S

BinPackingFF(W , w[1..n])
B ← new Array[1..n]; slack ← new Array[1..n]; numBins ← 0
for i ← 1 to n
j←1
while j ≤ numBins and w[i] > slack[j]
j ←j+1
if j > numBins
numBins ← numBins + 1; B[j] ← new ConsList(); slack[j] ← W
B[j] ← new ConsList(i, B[j]); slack[j] ← slack[j] − w[i]
return B[1..numBins]

The optimal packing must therefore contain more than (k −1)/2 bins. Thus,
the number of bins in the optimal packing is at least
k−1

k+1
+1=
2 2
≥ k/2.
The approximation ratio is therefore at most 2.

It can be shown via a much more complicated argument that if the

optimal packing uses B ∗ bins, then BinPackingFF gives a packing using
no more than ⌈ 17 ∗ ∗
10 B ⌉ bins. Thus, as B increases, the upper bound on
the approximation ratio approaches 17/10. If we first sort the items by
nonincreasing weight, it can be shown that this strategy (known as first-
fit decreasing) gives a packing using no more than 11 ∗
9 B + 4 bins. Note
∗
that although this upper bound is less than 3/2 as B increases, this does
CHAPTER 17. APPROXIMATION ALGORITHMS 574

not give a polynomial-time algorithm for ǫ-ApproxBP for any ǫ < 3/2,
as the proof of Theorem 17.5 essentially shows the hardness of deciding
whether B ∗ = 2. Furthermore, Theorem 17.5 does not preclude the existence
of a pseudopolynomial algorithm with an approximation ratio bounded by
some value less than 3/2. We leave it as an exercise to show that dynamic
programming can be combined with the first-fit decreasing strategy to yield,
for any positive ǫ, an approximation algorithm with an approximation ratio
bounded by 11 9 + ǫ.

17.4 The Traveling Salesperson Problem

Exercise 16.28 introduced the traveling salesperson problem as a decision
problem, TSP. Its input consists of a complete undirected graph G with
positive integer edge weights and a positive integer k. The question we ask
is whether there is a Hamiltonian cycle in G with total weight no more
than k. As is suggested by Exercise 16.28, TSP is strongly N P-complete.
The corresponding optimization problem does not include the input k, but
instead asks for the Hamiltonian cycle in G with minimum weight. It is
easily seen that this problem is N P-hard in the strong sense. Clearly, a
minimum weight Hamiltonian cycle has weight no more than nW , where n
is the number of vertices in G and W is the maximum weight of any edge in
G; hence, by Theorem 17.4, there can be no fully polynomial approximation
scheme for the optimization problem unless P = N P.
For ǫ > 0, let ǫ-ApproxTSP be the problem of finding, for a given
undirected graph G with positive integer edge weights, a Hamiltonian cycle
with approximation ratio no more than 1 + ǫ. In what follows, we will
show that ǫ-ApproxTSP is N P-hard in the strong sense for every positive
ǫ. As a result, there can be no polynomial or pseudopolynomial algorithm
for finding an approximation with any bounded approximation ratio unless
P = N P.

Theorem 17.7 For every positive ǫ, ǫ-ApproxTSP is N P-hard in the

strong sense.

Proof: Let ǫ > 0, and let HC be the problem of deciding whether a given
undirected graph G contains a Hamiltonian cycle. By Exercise 16.9, HC
is N P-complete. Since there are no integers in the problem instance, it is
strongly N P-complete. We will show that HC ≤pp T ǫ-ApproxTSP, where
pp
≤T denotes a pseudopolynomial Turing reduction. It will then follow that
ǫ-ApproxTSP is N P-hard in the strong sense.
CHAPTER 17. APPROXIMATION ALGORITHMS 575

Let G = (V, E) be an undirected graph. We first construct a complete

undirected graph G′ = (V, E ′ ). Let k = ⌊ǫ⌋ + 2. We define the weight of an
edge e ∈ E ′ as follows:
• If e ∈ E, then the weight of e is 1.
• If e 6∈ E, then the weight of e is nk, where n is the size of V .
Note that because k is a fixed constant, the weights are bounded by a poly-
nomial in the size of G.
We now show how we can use an approximation of a minimum-weight
Hamiltonian cycle in G′ to decide whether G has a Hamilton cycle. Suppose
we can obtain an approximation with an approximation ratio of no more
than 1 + ǫ. If the weight of this approximation is n, then the correspond-
ing Hamiltonian cycle must contain only edges with weight 1; hence, it is a
Hamiltonian cycle in G, so we can answer “yes”. Otherwise, the approxima-
tion contains at least one edge with weight nk, and n > 0. The weight of the
approximation is therefore at least nk + n − 1. Because the approximation
ratio is no more than 1 + ǫ, the minimum-weight Hamiltonian path has a
weight of at least
nk + n − 1 n(⌊ǫ⌋ + 2) + n − 1
=
1+ǫ 1+ǫ
n(1 + ǫ)
>
1+ǫ
= n.
Hence, there is no Hamiltonian cycle whose edge weights are all 1. Because
this implies that G contains no Hamiltonian cycle, we can answer “no”.
The running time for this algorithm, excluding any time needed to com-
pute the approximation, is linear in the size of G. Furthermore, all integers
constructed have values polynomial in the size of G. We therefore conclude
that ǫ-ApproxTSP is N P-hard in the strong sense.

As a result of Theorem 17.7, we have little hope of finding a polynomial-

time approximation algorithm yielding a bounded approximation ratio for
the traveling salesperson problem. However, if we make a certain restric-
tion to the problem, we can find such an algorithm. The metric traveling
salesperson problem is the restriction of the traveling salesperson problem
to inputs in which the edges of the graph satisfy the triangle inequality; i.e.,
if u, v, and w are vertices, then
weight({u, w}) ≤ weight({u, v}) + weight({v, w}).
CHAPTER 17. APPROXIMATION ALGORITHMS 576

The triangle inequality is satisfied, for example, if the vertices represent

points in the plane, and the edge weights represent distances. In what
follows, we will present a polynomial-time approximation algorithm yielding
an approximation ratio bounded by 2 for this problem.
We first observe that if we remove any edge from a Hamiltonian cycle,
we obtain a spanning tree of the graph. Furthermore, the weight of this
spanning tree must be less than the weight of the Hamiltonian cycle. Hence,
an MST will have a weight strictly less than the weight of a minimum-weight
Hamiltonian cycle. Now consider a tour of an MST that follows a depth-
first search — that is, we go from vertex u to vertex v when the call on u
makes a call on v, and we go from v to u when the call on v returns. In
this way, we traverse each edge exactly twice and reach each vertex at least
once, returning to the vertex from which we started. Clearly, the weight of
the edges in this tour (counting each edge exactly twice) is less than twice
the weight of a minimum-weight Hamiltonian cycle.
We now wish to convert this tour to a Hamiltonian cycle by taking
shortcuts. Specifically, when the tour would return to a vertex that it has
already reached, we skip ahead to the next vertex in the tour that has not
yet been reached (see Figure 17.3). When we have reached all vertices, we
return to the starting point.
It is easily seen by induction that if the triangle inequality is satisfied,
then the weight of edge {u, v} is no more than the sum of the weights of the
edges on any simple path from u to v. It is easily seen that when a path
from u to v is replaced by edge {u, v} in the above conversion, that path is
a simple path, because all edges in the tour that reach vertices that have
already been reached must go from children to parents; hence, the path in
the tour from u to v takes edges from children to parents, followed by a
single edge from a parent to v, which is reached for the first time in the
tour. Clearly, no vertex can be repeated in such a path. As a result, the
weight of this Hamiltonian cycle is less than twice the weight of an optimal
Hamiltonian cycle.
Notice that because this Hamiltonian cycle reaches the vertices in the
same order that they are first reached in the depth-first search, the vertices
are ordered by their preorder traversal numbers. Therefore, it is easy to con-
struct this Hamiltonian cycle while doing the depth-first search on the MST.
A Searcher for the depth-first search needs only a VisitCounter pre for
recording the preorder traversal numbers and a readable array order[0..n−1]
such that the Hamiltonian cycle will be horder[0], order[1], . . . , order[n −
1], order[0]i. Such a Searcher is defined in Figure 17.4.
For constructing an MST, we can use either Kruskal’s algorithm (Figure
CHAPTER 17. APPROXIMATION ALGORITHMS 577

Figure 17.3 Conversion of an MST to a Hamiltonian cycle

Figure 17.4 An implementation of Searcher for use in the metric travel-

ing salesperson approximation algorithm

MetricTspSearcher(n)
pre ← new VisitCounter(n); order ← new Array[0..n − 1]

MetricTspSearcher.PreProc(i)
pre.Visit(i); order[pre.Num(i)] ← i
CHAPTER 17. APPROXIMATION ALGORITHMS 578

Figure 17.5 Approximation algorithm for the metric traveling salesperson

problem

Precondition: G is a Graph representing a complete undirected graph

with at least one vertex, whose edges contain positive Nat weights satisfying
the triangle inequality.
Postcondition: Returns an array order[0..n − 1] in which each Nat
less than n occurs exactly once. The sum of the weights of the edges
{order[0], order[1]}, {order[1], order[2]}, . . . , {order[n − 1], order[0]} is less
than twice the weight of an optimal Hamiltonian cycle.
MetricTsp(G)
n ← G.Size; L ← Prim(G)
G′ ← new ListMultigraph(n)
while not L.IsEmpty()
e ← L.Head(); L ← L.Tail()
i ← e.Source(); j ← e.Dest(); x ← e.Data()
G′ .Put(i, j, x); G′ .Put(j, i, x)
G ← new ListGraph(G′ ); sel ← new Selector(n)
′′

s ← new MetricTspSearcher(n); Dfs(G′′ , 0, sel, s)

return s.Order()

11.1, page 379) or Prim’s algorithm (Figure 11.2, page 382). Our graph is
complete, so that the number of edges is in Θ(n2 ), where n is the number of
vertices. If we are using a ListGraph representation, Kruskal’s algorithm
is more efficient, running in Θ(n2 lg n) time. However, if we are using a
MatrixGraph representation, Prim’s algorithm is more efficient, running
in Θ(n2 ) time. Because we can construct a MatrixGraph from a List-
Graph in Θ(n2 ) time, we will use Prim’s algorithm. The entire algorithm
is shown in Figure 17.5.
Assuming G is a MatrixGraph, the call to Prim runs in Θ(n2 ) time.
The ListMultigraph constructor then runs in Θ(n) time. Because the
ConsList returned by Prim contains exactly n − 1 edges, and the List-
Multigraph.Put operation runs in Θ(1) time, the loop runs in Θ(n) time.
As was shown in Section 9.5, the ListGraph constructor runs in Θ(n)
time. The Selector constructor runs in Θ(n) time, and the MetricTsp-
Searcher constructor clearly runs in Θ(1) time. Because the MetricTsp-
CHAPTER 17. APPROXIMATION ALGORITHMS 579

Searcher.PreProc operation runs in Θ(1) time and G′′ is a ListGraph,

with n − 1 edges, the call to Dfs runs in Θ(n) time. The total running time
is therefore in Θ(n2 ).

17.5 The Maximum Cut and Minimum Cluster

Problems
We conclude this chapter by examining two optimization problems that are
essentially the same, but which yield entirely different results with respect to
approximation algorithms. Let G = (V, E) be a complete undirected graph
with positive integer edge weights. For a natural number k ≥ 2, a k-cut for
G is a partition of V into k disjoint sets, S1 , S2 , . . . , Sk . The weight of this
cut is the sum of the weights of all edges {u, v} ∈ E such that u and v are in
different partitions. The maximum cut problem is to find, for a given natural
number k ≥ 2 and complete undirected graph G = (V, E) with edge weights
and more than k vertices, a k-cut with maximum weight. The minimum
cluster problem is to find a k-cut that minimizes the sum of the weights of
all edges {u, v} ∈ E such that u and v are in the same partition. Clearly, a
k-cut has maximum weight iff it minimizes this latter sum.
Let Cut be the problem of deciding, for given natural numbers k ≥ 2
and B, and complete undirected graph G = (V, E) with positive integer
edge weights and more than k vertices, whether there is a k-cut with weight
at least B. For each k ≥ 2, we also define the k-Cut problem to be the
Cut problem restricted to cuts of exactly k sets (i.e, k is not given as input,
but is fixed). We further define the Cluster and k-Cluster problems
analogously. We leave as exercises to show that k-Cluster is strongly N P-
complete for every k ≥ 2, and that Cluster is strongly N P-complete. It
then follows that Cut and k-Cut for k ≥ 2 are all strongly N P-complete.
Given the above results, the problems of finding either a maximum cut
or a minimum cluster are N P-hard in the strong sense. Thus, from Theorem
17.4, there is no fully polynomial approximation scheme for either of these
problems unless P = N P. However, there is a simple greedy strategy that
yields good approximation ratios for maximum cut. We begin with k empty
sets, and add vertices one by one to the set that gives us the largest cut. The
algorithm is shown in Figure 17.6. Note that this algorithm actually uses
the amount by which the total weight of the clusters would increase when
choosing the set in which to place a given vertex — these values are easier
to compute than the weights of the resulting cuts. It is easily seen that this
algorithm runs in Θ(n2 ) time, if G is represented by a MatrixGraph. The
CHAPTER 17. APPROXIMATION ALGORITHMS 580

Figure 17.6 Approximation algorithm for maximum cut

Precondition: G is a Graph representing a complete undirected graph

with positive Nat edge weights, and k is a Nat such that 2 ≤ k < n, where
n is the number of vertices in G.
Postcondition: Returns an array cut[0..n − 1] such that for 0 ≤ i < n,
1 ≤ cut[i] ≤ k. If W is the sum of the weights of the edges in G, then the
weight of the cut described by cut[0..n − 1] is at least W (k − 1)/k.
MaxCut(G, k)
n ← G.Size(); cut ← new Array[0..n − 1]
clusterInc ← new Array[1..k]
for i ← 0 to n − 1
for j ← 1 to k
clusterInc[j] ← 0
for j ← 0 to i − 1
clusterInc[cut[j]] ← clusterInc[cut[j]] + G.Get(i, j)
m←1
for j ← 2 to k
if clusterInc[j] < clusterInc[m]
m←j
cut[i] ← m
return cut[0..n − 1]

following theorem gives bounds for its approximation ratio.

Theorem 17.8 For each k ≥ 2, MaxCut yields an approximation ratio of

no more than
1
1+ ;
k−1
thus, the approximation ratio is never more than 2.

Proof: For a given vertex i, let Wi denote the sum of the weights of all
edges {i, j} such that 0 ≤ j < i. At the end of iteration i, the value of the
cut increases by Wi − clusterInc[m], where clusterInc[m] is the sum of the
weights of the edges from i to other vertices in partition m. m is chosen
so that clusterInc[m] is minimized; hence, for each partition other than m,
the sum of the weights of the edges from i to vertices in that partition is at
CHAPTER 17. APPROXIMATION ALGORITHMS 581

least clusterInc[m]. We therefore have

clusterInc[m] ≤ Wi /k.

The value of the cut therefore increases by at least Wi (k − 1)/k on

iteration i. Because the value of the cut is initially 0, the final value of the
cut is at least
n−1 n−1
X Wi (k − 1) k−1X
= Wi
k k
i=0 i=0
k−1
= W,
k
where W is the sum of all edge weights in G. Clearly, the maximum cut can
be no more than W . The approximation ratio is therefore bounded above
by
W k
=
(k − 1)W/k k−1
1
=1+ .
k−1

Though the algorithm MaxCut yields a fixed bound on the approxi-

mation ratio for approximating a maximum cut, it is perhaps surprising,
that the same algorithm yields unbounded approximation ratios for approx-
imating a minimum cluster, even though the two optimization problems are
essentially the same. We can see why this is the case by examining in-
stances that cause the approximation ratio for MaxCut to approach the
upper bound shown in Theorem 17.8.
For k ≥ 2, consider a complete undirected graph G = (V, E), where
V = {i ∈ N | i < k 2 }. We partition V into k groups such that for 0 ≤ j < k,
group j is the set {i ∈ N | jk ≤ i < (j + 1)k}; thus, each group contains
k vertices. We now assign weights to the edges in G such that if vertices
u and v are in the same group, then {u, v} has weight 1; otherwise, {u, v}
has weight x, where x is some sufficiently large natural number. (See Figure
17.7 for the case in which k = 2.)
We claim that if x ≥ k 3 , the maximum cut for G partitions the vertices
so that each group forms a cluster. To see this, first note that when we
partition G in this way, the resulting cut includes exactly those edges with
CHAPTER 17. APPROXIMATION ALGORITHMS 582

Figure 17.7 A bad case for MaxCut

1
0 1

x x
x
x
max cut

2 3
1

approximation

weight x. Because the sum of all other edges is less than x, the maximum cut
must contain all of the edges with weight x. Note that connecting any two of
the vertices 0, k, . . . , k2 − k is an edge with weight x; hence, a maximum cut
must place all of these k vertices into different clusters. Then for 0 ≤ i < k
and 1 ≤ j < k, there is an edge of weight x between vertex ik + j and i′ k
for each i′ 6= i. As a result, vertex ik + j must be placed in the same cluster
as vertex ik. Hence, the maximum cut partitions the vertices so that each
group forms a cluster.
Now consider the behavior of MaxCut on G. It will first place vertices
0, 1, . . . , k − 1 into different clusters. Then vertex k is adjacent to exactly
one vertex in each of the clusters via an edge with weight x. Because placing
k in any of the clusters would increase the weight of that cluster by x, k is
placed in the first cluster with vertex 0. Placing k + 1 in this cluster would
increase its weight by x+1; however, placing k +1 in any other cluster would
increase that cluster’s weight by only x. As a result, k + 1 is placed in the
second cluster with vertex 1. It is easily seen that the algorithm continues
by placing vertices into clusters in round-robin fashion, so that each cluster
ultimately contains exactly one vertex from each group.
We can use symmetry to help us to evaluate the approximation ratio of
MaxCut on G. In the maximum cut, each vertex is adjacent to (k − 1)k
CHAPTER 17. APPROXIMATION ALGORITHMS 583

vertices in other clusters via edges whose weights are all x. In the cut
produced by MaxCut, each vertex is also adjacent to (k − 1)k vertices in
other clusters; however, only (k − 1)2 of these edges have weight x, while the
remaining edges each have weight 1. The approximation ratio is therefore
(k − 1)kx kx
= ,
(k − 1)2 x + k − 1 (k − 1)x + 1
which approaches
k 1
=1+
k−1 k−1
as x approaches ∞. The bound of Theorem 17.8 is therefore tight.
Let us now analyze the approximation ratio for MaxCut as an approx-
imation algorithm for the minimum cluster problem. Again we can use
symmetry to simplify the analysis. In the optimal solution, each vertex is
adjacent to k − 1 vertices in the same cluster via edges whose weights are
all 1. In the solution given by MaxCut, each vertex is adjacent to k − 1
vertices in the same cluster via edges whose weights are all x. Thus, the
approximation ratio is x, which can be chosen to be arbitrarily large. We
can therefore see that even though the maximum cut and minimum clus-
ter optimization problems are essentially the same, the MaxCut algorithm
yields vastly different approximation ratios relative to the two problems.
To carry this idea a step further, we will now show that the minimum
cluster problem has no approximation algorithm with a bounded approxi-
mation ratio unless P = N P. For a given ǫ ∈ R>0 and integer k ≥ 2, let the
ǫ-Approx-k-Cluster problem be the problem of finding, for a given com-
plete undirected graph G with positive integer edge weights, a k-cut whose
sum of cluster weights is at most W ∗ (1 + ǫ), where W ∗ is the minimum sum
of cluster weights. Likewise, let ǫ-Approx-Cluster be the corresponding
problem with k provided as an input. We will show that for every positive ǫ
and every integer k ≥ 3, the ǫ-Approx-k-Cluster problem is N P-hard in
the strong sense. Because ǫ-Approx-3-Cluster ≤pp T ǫ-Approx-Cluster,
it will then follow that this latter problem is also N P-hard in the strong
sense. Whether the result extends to ǫ-Approx-2-Cluster is unknown at
the time of this writing.

Theorem 17.9 For every ǫ ∈ R>0 and every k ≥ 3, ǫ-Approx-k-Cluster

is N P-hard in the strong sense.

Proof: As is suggested by Exercises 16.23 and 16.24, the problem of de-

ciding whether a given undirected graph is k-colorable is N P-complete for
CHAPTER 17. APPROXIMATION ALGORITHMS 584

each k ≥ 3. Let us refer to this problem as k-Col. Because k-Col contains

no large integers, it is N P-complete in the strong sense. We will now show
that k-Col ≤pp T ǫ-Approx-k-Cluster, so that ǫ-Approx-k-Cluster is
N P-hard in the strong sense for k ≥ 3.
Let G = (V, E) be a given undirected graph. Let n be the number
of vertices in G. We can assume without loss of generality that n > k,
for otherwise G is clearly k-colorable. We construct G′ = (V, E ′ ) to be
the complete graph on V . We assign an edge weight of n2 ⌈1 + ǫ⌉ to edge
{u, v} ∈ E ′ if {u, v} ∈ E; otherwise, we assign it a weight of 1. Clearly,
this construction can be completed in time polynomial in n. Furthermore,
because ǫ is a fixed constant, all integers have values polynomial in n.
Suppose we have a k-cut of G′ such that the ratio of its cluster weight
to the minimum cluster weight is at most 1 + ǫ. If the given cluster weight
is less than n2 ⌈1 + ǫ⌉, then all edges connecting vertices in the same cluster
must have weight less than n2 ⌈1 + ǫ⌉; hence none of them belong to E.
This k-cut is therefore a k-coloring of G. In this case, we can answer “yes”.
Suppose the cluster weight is at least n2 ⌈1 + ǫ⌉. Because the approximation
ratio is no more than 1 + ǫ, the minimum sum of cluster weights is at least
n2 . It is therefore impossible to k-color G, for a k-coloring of G would be
a k-cut of G′ in which each cluster contains only edges with weight 1, and
which would therefore have total weight less than n2 . In this case, we can
answer “no”. We conclude that k-Col ≤pp T ǫ-Approx-k-Cluster, so that
ǫ-Approx-k-Cluster is N P-hard in the strong sense.

17.6 Summary
Using Turing reducibility, we can extend the definition of N P-hardness from
Chapter 16 to apply to problems other than decision problems in a natural
way. We can then identify certain optimization problems as being N P-hard,
either in the strong sense or the ordinary sense. One way of coping with
N P-hard optimization problems is by using approximation algorithms.
For some N P-hard optimization problems we can find polynomial ap-
proximation schemes, which take as input an instance x of the problem and
a positive real number ǫ and return, in time polynomial in |x|, an approxi-
mate solution with approximation ratio no more than 1+ǫ. If this algorithm
runs in time polynomial in |x| and 1/ǫ, it is called a fully polynomial ap-
proximation scheme.
However, Theorem 17.4 tells us that for most optimization problems, if
CHAPTER 17. APPROXIMATION ALGORITHMS 585

the problem admits a fully polynomial approximation scheme, then there

is a pseudopolynomial algorithm to solve the problem exactly. As a result,
we can use strong N P-hardness to show for a number of problems that
unless P = N P, that problem cannot have a fully polynomial approximation
scheme. Furthermore, by showing N P-hardness of certain approximation
problems, we can show that unless P = N P, the corresponding optimization
problem has no approximation algorithm with approximation ratio bounded
by some — or in some cases any — given value.
Finally, there are some pairs of optimization problems, such as the maxi-
mum cut problem and the minimum cluster problem, that are essentially the
same problem, but which yield vastly different results concerning approxi-
mation algorithm. For example, the maximum k-cut, can be approximated
1
in Θ(n2 ) time with an approximation ratio of no more than 1+ k−1 ; however,
unless P = N P, there is no polynomial-time algorithm with any bounded
approximation ratio for finding the minimum weight of clusters formed by
a k-cut if k ≥ 3.

17.7 Exercises
Exercise 17.1 Give an approximation algorithm that takes an instance of
the knapsack problem and a positive integer k and returns a packing with
an approximation ratio of no more than 1 + k1 . Your algorithm must run in
O(nk+1 ) time. Prove that both of these bounds (approximation ratio and
running time) are met by your algorithm.

Exercise 17.2 Suppose we were to modify the knapsack problem to allow

as many copies as we wish of any of the items. Show that the greedy
algorithm yields an approximation ratio of no more than 2 for this variation.

Exercise 17.3 The best-fit algorithm for bin packing considers the items
in the given order, always choosing the largest-weight bin in which the item
will fit. Show that the approximation ratio for this algorithm is no more
than 2.

** Exercise 17.4 Let ǫ be any fixed positive real number. Demonstrate

how dynamic programming can be combined with the first-fit-decreasing
algorithm to obtain a pseudopolynomial-time approximation algorithm for
bin packing with approximation ratio no more than 11 9 + ǫ. You may use the
fact that the first-fit-decreasing algorithm always produces a packing using
at most 11 ∗ ∗
9 B + 4 bins, where B is the minimum number of bins possible.
CHAPTER 17. APPROXIMATION ALGORITHMS 586

* Exercise 17.5

a. Prove that NotAllEqual-3-Sat ≤pp m k-Cluster for each fixed k ≥

2, so that, from the result of Exercise 16.6, k-Cluster is N P-hard
in the strong sense for k ≥ 2.

b. Prove that Cluster is strongly N P-complete.

* Exercise 17.6 Give an approximation algorithm with approximation ra-

tio bounded by 2 for the problem of finding a minimum-sized vertex cover.
Your algorithm should run in O(n + a) time, assuming the graph is im-
plemented as a ListGraph. Prove both the approximation ratio and the
running time.

* Exercise 17.7

a. Given an undirected graph G = (V, E), we define G2 = (V 2 , E ′ ) such

that for u = hu1 , u2 i and v = hv1 , v2 i in V 2 , {u, v} ∈ E ′ iff for every
i, 1 ≤ i ≤ 2, either ui = vi or {ui , vi } ∈ E. Prove that the size of the
largest clique in G2 is k 2 , where k is the size of the largest clique in
G.

b. Use part a to prove that if there is a polynomial-time algorithm with

a bounded approximation ratio for approximating the size of a largest
clique in a given graph, then there is a polynomial approximation
scheme for this problem.

17.8 Chapter Notes

The concept of a polynomial-time approximation algorithm was first formal-
ized by Garey, Graham, and Ullman [48] and Johnson [70]. In fact, much of
the foundational work in this area is due to Garey and Johnson — see their
text [52] for a summary of the early work. For example, they proved The-
orem 17.4 [51]. A detailed analysis of bin packing, including the 11 ∗
9 B +4
upper bound on the approximation ratio for first-fit decreasing, is given by
17 ∗
Johnson [69]. The ⌈ 10 B ⌉ upper bound for the first-fit-decreasing algorithm
is due to Garey, Graham, Johnson, and Yao [47], and a close relationship
between best-fit and first-fit was established by Johnson, Demers, Ullman,
Garey, and Graham [68].
The polynomial approximation scheme suggested by Exercise 17.1 for the
knapsack problem is due to Sahni [95]. The fully polynomial approximation
CHAPTER 17. APPROXIMATION ALGORITHMS 587

scheme of Section 17.2 is due to Ibarra and Kim [66]. Theorem 17.7 was
shown by Sahni and Gonzalez [94].