Developing a user-friendly OpenResty

application
OpenResty Con 2017 - Beijing

OpenResty Con 2017 - Beijing 1

$ whoami

Thibault Charbonnier

● Lead Engineer @ Kong (https://konghq.com)

● Main contributor @ Kong API Gateway (https://github.com/Kong/kong)
● OpenResty Contributor on my spare time
● https://github.com/thibaultcha
● https://chasum.net

OpenResty Con 2017 - Beijing 2

A user-friendly OpenResty application?

Initial assumption: Most OpenResty applications that we know of seem

to be private, deployed on internal infrastructures.

If we were to ship an on-premise OpenResty application, it should be

easy to install and deploy:

➔ NGINX processes only (no other daemons in the system)

➔ Minimal libraries dependencies (pure LuaJIT/OpenResty)
➔ Horizontally scalable (clustering)
➔ Platform agnostic

OpenResty Con 2017 - Beijing 3

A user-friendly OpenResty application?
Example: recurring background jobs

➔ Cronjob

*/5 * * * * curl -XGET 'http://localhost:8000/job?hello=world'

vs

➔ ngx.timer API

ngx.timer.every(60 * 5, do_job(), "world")

OpenResty Con 2017 - Beijing 4

What is Kong?
Short introduction to API Gateways

OpenResty Con 2017 - Beijing 5

What is an API Gateway?
It’s a reverse proxy, sitting between your clients and your upstream services

ABSTRACTION LAYER

Items

Client API Gateway

(The client can be another service too) Authentication

Customers
Security

Logging

Transformations Orders

Load-Balancing

...and more.
Invoices

OpenResty Con 2017 - Beijing 6

What is an API Gateway?
Reduce Code Duplication, Orchestrate Common Functionalities

OpenResty Con 2017 - Beijing 7

Kong
Open Source API Gateway

Built with OpenResty.

■ 12,000+ Stars on GitHub
■ 70+ Contributors
■ 107 Meetups
● Open Source ■ 10K Community Members

● Extensible via Plugins (60+ available)

● Sub-millisecond latency on most
use-cases
● Platform Agnostic
● Horizontally Scalable

https://github.com/Mashape/kong https://getkong.org

OpenResty Con 2017 - Beijing 8

Kong
init_by_lua_block {
kong = require 'kong'
kong.init()
}
…
location / {
set $upstream_scheme '';
set $upstream_uri '';

rewrite_by_lua_block { kong.rewrite() }

access_by_lua_block { kong.access() }

proxy_http_version 1.1;
proxy_pass $upstream_scheme://kong_upstream$upstream_uri;

header_filter_by_lua_block { kong.header_filter() }

body_filter_by_lua_block { kong.body_filter() }

log_by_lua_block { kong.log() }
}

OpenResty Con 2017 - Beijing 9

Kong v0.1 dependencies

● Proxy + Lua middleware → OpenResty

● A command line interface (CLI) → PUC-Rio Lua 5.1 ☹
● DNS resolution → dnsmasq ☹
● Clustering of nodes → Serf ☹
● Generate UUIDs → libuuid ☹
● Database → Cassandra

Lots of dependencies for users to install...

OpenResty Con 2017 - Beijing 10

Kong v0.1 dependencies

Processes
● Kong’s CLI
● NGINX
● dnsmasq
● serf

Ports
● 80 + 8001 (NGINX)
● 8053 (dnsmasq)
● 7946 + 7373 TCP/UDP (serf)

Less than ideal for dockerized environments or

firewall rules...

OpenResty Con 2017 - Beijing 11

CLI
Choosing an interpreter

OpenResty Con 2017 - Beijing 12

Build an OpenResty CLI

~ $ kong
Usage: kong COMMAND
Database I/O
[OPTIONS]

The available commands are:

migrations
prepare Start NGINX
reload
restart
start
stop
version
Stop NGINX
Options:
--v verbose
--vv debug

OpenResty Con 2017 - Beijing 13

Build an OpenResty CLI

#!/usr/bin/env lua

require("kong.cmd.init")(arg)

● No FFI (LuaJIT only)

● No support for cosockets (LuaSocket + LuaSec fallback)
● Missing ngx.* API

Lots of fragmentation between our OpenResty and CLI code ☹

OpenResty Con 2017 - Beijing 14

Build an OpenResty CLI

#!/usr/bin/env luajit

require("kong.cmd.init")(arg)

● LuaJIT FFI available

● No support for cosockets (LuaSocket + LuaSec fallback)
● Missing ngx.* API

An improvement but fragmentation is still very much of an issue ☹

OpenResty Con 2017 - Beijing 15

Build an OpenResty CLI

#!/usr/bin/env resty

require("kong.cmd.init")(arg)

● Runs in timer context thanks to https://github.com/openresty/resty-cli

● Cosockets available
● ngx.* API available
● LuaJIT FFI available

We can reuse our OpenResty and CLI code

No PUC-Rio Lua dependency

OpenResty Con 2017 - Beijing 16

Build an OpenResty CLI

● Proxy + Lua middleware → OpenResty

● A command line interface (CLI) → PUC-Lua 5.1 OpenResty
● DNS resolution & Load balancing → dnsmasq
● Clustering of nodes → Serf
● Generate UUIDs → libuuid
● Database → Cassandra

OpenResty Con 2017 - Beijing 17

Using resty-cli in busted

OpenResty Con 2017 - Beijing 18

Using resty-cli in busted

describe('Busted unit testing framework', function()

it('should be easy to use', function()
assert.truthy('Yup.')
end)

it('should have lots of features', function()

Kong’s test framework is busted since 2014 -- deep check comparisons!
assert.same({ table = 'great'}, { table = 'great' })
https://github.com/Olivine-Labs/busted
-- or check by reference!
assert.is_not.equals({ table = 'great'},
{ table = 'great'})

assert.falsy(nil)
assert.error(function() error('Wat') end)
end)
end)

OpenResty Con 2017 - Beijing 19

Using resty-cli in busted

We changed the interpreter from PUC-Rio

Lua to resty-cli
-- ./rbusted
#!/usr/bin/env resty

-- Busted command-line runner

require 'busted.runner'({ standalone = false })

https://github.com/thibaultcha/lua-resty-busted

OpenResty Con 2017 - Beijing 20

Using resty-cli in busted

-- t/sanity_spec.lua
describe("openresty script", function()
it("should run in ngx_lua context", function()
assert.equal(0, ngx.OK)
assert.equal(200, ngx.HTTP_OK)
end)

it("should yield", function()

ngx.sleep(3)
assert.is_true(1 == 1)
end)
end)

OpenResty Con 2017 - Beijing 21

Using resty-cli in busted
Improving busted for OpenResty development

~ $ rbusted --o=tap t/sanity_spec.lua

ok 1 - openresty script should run in ngx_lua context
ok 2 - openresty script should yield
1..2

Now we can test our OpenResty code with busted!

OpenResty Con 2017 - Beijing 22

UUID generation
And PRNG seeding

OpenResty Con 2017 - Beijing 23

Removing the libuuid dependency

● PUC-Rio Lua - https://github.com/Tieske/uuid

○ Slowest implementation
○ Generates invalid v4 UUIDs
● libuuid binding (Lua C API) - https://github.com/Kong/lua-uuid
○ Safe underlying implementation
○ External dependency
● libuuid binding (LuaJIT FFI) - https://github.com/bungle/lua-resty-uuid
○ Safe underlying implementation
○ External dependency
● LuaJIT - https://github.com/thibaultcha/lua-resty-jit-uuid
○ Seems to be the fastest implementation
○ Uses LuaJIT’s PRNG

OpenResty Con 2017 - Beijing 24

Removing the libuuid dependency

LuaJIT 2.1.0-beta1 with 1e+06 UUIDs

UUID v4 (random) generation
1. resty-jit-uuid took: 0.064228s 0%
2. FFI binding took: 0.093374s +45%
3. C binding took: 0.220542s +243%
4. Pure Lua took: 2.051905s +3094%

UUID v3 (name-based and MD5) generation if supported

1. resty-jit-uuid took: 1.306127s

UUID v5 (name-based and SHA-1) generation if supported

1. resty-jit-uuid took: 4.834929s

UUID validation if supported (set of 70% valid, 30% invalid)

1. resty-jit-uuid (JIT PCRE enabled) took: 0.223060s
2. FFI binding took: 0.256580s
3. resty-jit-uuid (Lua patterns) took: 0.444174s

OpenResty Con 2017 - Beijing 25

Caution: PRNG seeding in NGINX workers

init_by_lua_block {
--math.randomseed(ngx.time()) <-- AVOID
}

init_worker_by_lua_block {
math.randomseed(ngx.time() + ngx.worker.pid())
math.randomseed = function()end -- ensure we prevent re-seeding
}

● Be wary of calling math.randomseed() in init_by_lua

● Seeding in init_worker_by_lua is safer
● Still, some external dependencies may call math.randomseed() again

A possible solution: https://github.com/openresty/lua-resty-core/pull/92

OpenResty Con 2017 - Beijing 26

Build an OpenResty CLI

● Proxy + Lua middleware → OpenResty

● A command line interface (CLI) → PUC-Lua 5.1 OpenResty
● DNS resolution & Load balancing → dnsmasq
● Clustering of nodes → Serf
● Generate UUIDs → libuuid OpenResty
● Database → Cassandra

OpenResty Con 2017 - Beijing 27

DNS Resolution

OpenResty Con 2017 - Beijing 28

DNS Resolution

NGINX does not use the system resolver, and needs a

user-specified name server.

More often than not, this deceives users:

➔ Ignores /etc/resolv.conf ☹
➔ Ignores /etc/hosts ☹
➔ No support for SRV records ☹
➔ Unusable with balancer_by_lua ☹

OpenResty Con 2017 - Beijing 29

DNS Resolution

Temporary solution: dnsmasq

~ $ kong start --vv

...
2017/03/01 14:45:35 [debug] found 'dnsmasq' executable at /usr/sbin/dnsmasq
2017/03/01 14:45:35 [debug] starting dnsmasq: /usr/sbin/dnsmasq -p 8053 --pid-file=/usr/local/kong/pids/dnsmasq.pid -N -o
--listen-address=127.0.0.1

http { ● Parses /etc/resolv.conf

resolver 127.0.0.1:8053 ipv6=off; ● Parses /etc/hosts
● Support for SRV records
...
} ● Still unusable with balancer_by_lua ☹
● New dependency ☹

dnsmasq daemon

OpenResty Con 2017 - Beijing 30

DNS Resolution

To remove our dnsmasq dependency, and use balancer_by_lua, we

must resolve DNS records in the Lua land.

Part of the solution: https://github.com/openresty/lua-resty-dns

● Pure Lua, bundled with OpenResty

● Resolves, A, AAAA, CNAME, SRV records (and more)
● No /etc/hosts parsing ☹
● No /etc/resolv.conf parsing ☹
● No results caching ☹
● No DNS load-balancing ☹

OpenResty Con 2017 - Beijing 31

DNS Resolution

lua-resty-dns-client - https://github.com/Kong/lua-resty-dns-client
Author: Thijs Schreijer (@tieske)

● Built on top of lua-resty-dns

● Parses /etc/hosts
● Parses /etc/resolv.conf
● Built-in cache & asynchronous querying
● Built-in DNS load-balancing
●

OpenResty Con 2017 - Beijing 32

DNS Resolution

● Proxy + Lua middleware → OpenResty

● A command line interface (CLI) → PUC-Lua 5.1 OpenResty
● DNS resolution & Load balancing → dnsmasq OpenResty
● Clustering of nodes → Serf
● Generate UUIDs → libuuid OpenResty
● Database → Cassandra

OpenResty Con 2017 - Beijing 33

Clustering

OpenResty Con 2017 - Beijing 34

Clustering

● Kong nodes connected to the same database (PostgreSQL or Cassandra)

share the same configuration.
● To limit database traffic, Kong nodes maintain their own cache.
● lua-shared-dict + lua-resty-lock allow Kong to avoid the “dogpile effect”
(cache stampede).

http {
lua_shared_dict kong_cache ${{MEM_CACHE_SIZE}};
…
}

OpenResty Con 2017 - Beijing 35

Clustering

Temporary solution: Serf (https://www.serf.io/)

~ $ kong start --vv

...
2017/05/22 14:30:13 [debug] found 'serf' executable in $PATH
2017/05/22 14:30:13 [debug] starting serf agent: nohup serf agent -profile 'wan' -bind '0.0.0.0:7946' -log-level 'err' -rpc-addr
'127.0.0.1:7373' -event-handler
'member-join,member-leave,member-failed,member-update,member-reap,user:kong=/usr/local/kong/serf/serf_event.sh' -node
'dev_0.0.0.0:7946_470b634076b94e2aa6a0bb7bce7673f7' > /usr/local/kong/logs/serf.log 2>&1 & echo $! > /usr/local/kong/pids/serf.pid
2017/05/22 14:30:14 [verbose] serf agent started
2017/05/22 14:30:14 [verbose] auto-joining serf cluster
2017/05/22 14:30:14 [verbose] registering serf node in datastore
2017/05/22 14:30:14 [verbose] cluster joined and node registered in datastore

● Provides inter-nodes gossiping

● New dependency ☹
● Additional ports and firewall rules ☹
● Additional cross-datacenter communication ☹

OpenResty Con 2017 - Beijing 36

Clustering
The overhead of the OpenResty + Serf pattern

LB LB

us-west-1 us-east-1

K K K K K K
Serf Serf Serf Serf Serf Serf

Cassandra Cassandra

37
OpenResty Con 2017 - Beijing 37
Clustering
Our desired high-level view of a Kong cluster

LB LB

us-west-1 us-east-1

K K K K K K

Cassandra Cassandra

38
OpenResty Con 2017 - Beijing 38
Clustering

We removed our Serf dependency by introducing a pub/sub mechanism between

OpenResty and PostgreSQL/Cassandra.

● Workers write in a “channel” with broadcast(channel, data, nbf)

● Other workers subscribe to it with subscribe(channel, callback)

● A combination of ngx.timer and lua-resty-lock allows for a safe polling mechanism

● Introducing new configuration properties:

db_update_frequency/db_update_propagation/db_cache_ttl

● Upon invalidation event received: ngx.shared.cache:delete(key)

https://github.com/Kong/kong/blob/master/kong/cluster_events.lua

OpenResty Con 2017 - Beijing 39

Clustering

lua-resty-mlcache
NGINX

● Multi-level caching (lua-resty-cache + worker worker worker

L1
lua_shared_dict) with LRU eviction
Lua cache Lua cache Lua cache
● TTL and negative (miss) TTL

● Built-in mutex mechanism with

lua-resty-lock to prevent dogpile L2 lua_shared_dict
effects

● Multiple instances supported

callback

I/O fetch

Database, API, I/O...

https://github.com/thibaultcha/lua-resty-mlcache

OpenResty Con 2017 - Beijing 40

DNS Resolution

● Proxy + Lua middleware → OpenResty

● A command line interface (CLI) → PUC-Lua 5.1 OpenResty
● DNS resolution & Load balancing → dnsmasq OpenResty
● Clustering of nodes → Serf OpenResty
● Generate UUIDs → libuuid OpenResty
● Database → Cassandra

OpenResty Con 2017 - Beijing 41

Inter-workers communication

OpenResty Con 2017 - Beijing 42

Inter-workers communication

Invalidating Lua-land cache (lua-resty-lru) requires inter-workers

communication, a long-requested OpenResty feature.

lua-resty-worker-events - https://github.com/Kong/lua-resty-worker-events
Author: Thijs Schreijer (@tieske)

● Pub/sub mechanism via lua_shared_dict

● Multiple channels
● Automatic polling via ngx.timer

Ideally, a binding API for cosockets will one day replace lua_shared_dict based
solutions!

OpenResty Con 2017 - Beijing 43

Conclusion

OpenResty Con 2017 - Beijing 44

Conclusion

One by one, we’ve eliminated all external dependencies. Kong now is a

pure OpenResty application.

● Proxy + Lua middleware → OpenResty

● A command line interface (CLI) → PUC-Lua 5.1 OpenResty
● DNS resolution & Load balancing → dnsmasq OpenResty
● Clustering of nodes → Serf OpenResty
● Generate UUIDs → libuuid OpenResty
● Database → Cassandra

OpenResty Con 2017 - Beijing 45

Conclusion
We’ve open sourced several libraries to the OpenResty community!

● https://github.com/Kong/lua-resty-dns-client

● https://github.com/Kong/lua-resty-worker-events

● https://github.com/thibaultcha/lua-resty-mlcache

● https://github.com/thibaultcha/lua-resty-jit-uuid

● https://github.com/thibaultcha/lua-resty-busted

● And more!

● https://github.com/thibaultcha/lua-cassandra (see my LuaConf

2017 talk in Rio de Janeiro: https://youtu.be/o8mbOT3Veeo)

● https://github.com/thibaultcha/lua-resty-socket

OpenResty Con 2017 - Beijing 46

Conclusion

Wishlist:
● Support for cosockets in init_by_lua

● Native inter-workers communication

● Support for SSL client certificates for cosockets

(https://github.com/openresty/lua-nginx-module/pull/997)

● Support for ngx.rawlog() API

(https://github.com/openresty/lua-resty-core/pull/128)

● Support for /etc/hosts parsing

(https://github.com/openresty/openresty/pull/247)

OpenResty Con 2017 - Beijing 47

Thank you!
Questions?

OpenResty Con 2017 - Beijing 48

Bonus

OpenResty Con 2017 - Beijing 49

lua-cjson empty array encoding
local cjson = require "cjson"
local rows = {} -- fetch from db

-- before
cjson.encode({ data = rows })

--[[
{
"data":{}
}
--]]

-- now
setmetatable(rows, cjson.empty_array_mt)
cjson.encode({ data = rows })

--[[
{
"data":[]
https://github.com/openresty/lua-cjson/pull/6
}
--]]

OpenResty Con 2017 - Beijing 50

lua-resty-socket

https://github.com/thibaultcha/lua-resty-socket

Compatibility module for cosocket/LuaSocket.

● Automatic fallback to LuaSocket in non-OpenResty, or non-supported
OpenResty contexts (e.g. init_by_lua)
● Support for SSL via LuaSec fallback
● Full interoperability

local socket = require "resty.socket"

local sock = socket.tcp()

sock:settimeout(1000) ---> 1000ms converted to 1s if LuaSocket

sock:getreusedtimes(...) ---> 0 if LuaSocket
sock:setkeepalive(...) ---> calls close() if LuaSocket
sock:sslhandshake(...) ---> LuaSec dependency if LuaSocket

OpenResty Con 2017 - Beijing 51

Friendly error logs with ngx.log

local errlog = require "ngx.errlog"

errlog.rawlog(ngx.NOTICE, "hello world")

2017/07/09 19:36:25 [notice] 25932#0: *1 [lua] content_by_lua(nginx.conf:51):5:

hello world, client: 127.0.0.1, server: localhost, request: "GET /log
HTTP/1.1", host: "localhost"

● Raw output to error_log

● Customizable stacktrace level report

https://github.com/openresty/lua-resty-core/pull/128

OpenResty Con 2017 - Beijing 52

Thank you!
Questions?

OpenResty Con 2017 - Beijing 53