rpxy: A simple and ultrafast reverse-proxy serving multiple domain names with TLS termination, written in Rust
WIP Project
Note
This project is an HTTP, i.e., Layer 7, reverse-proxy. If you are looking for a TCP/UDP, i.e., Layer 4, reverse-proxy, please check my another project, rpxy-l4
.
rpxy
[ahr-pik-see] is a simple and lightweight reverse-proxy implementation with additional features. The implementation is based on hyper
, rustls
and tokio
, i.e., written in Rust 1. rpxy
routes multiple hostnames to appropriate backend application servers while serving TLS connections.
The supported features are summarized as follows:
- Supported HTTP(S) protocols: HTTP/1.1, HTTP/2, and the brand-new HTTP/3 2
- gRPC is also supported
- Serving multiple domain names with TLS termination
- Mutual TLS authentication with client certificates
- Automated certificate issuance and renewal via TLS-ALPN-01 ACME protocol 3
- Post-quantum key exchange for TLS/QUIC 4
- TLS connection sanitization to avoid domain fronting 5
- Load balancing with round-robin, random, and sticky sessions
- and more...
This project is still work-in-progress. However, it is already working in some production environments and serves a number of domain names. Furthermore, it significantly outperforms NGINX and Caddy, e.g., 30% ~ 60% or more faster than NGINX, in very simple HTTP reverse-proxy scenarios (See bench
directory).
You can build an executable binary yourself by checking out this Git repository.
# Cloning the git repository
% git clone https://github.com/junkurihara/rust-rpxy
% cd rust-rpxy
# Update submodules
% git submodule update --init
# Build (default: QUIC and HTTP/3 is enabled using `quinn`)
% cargo build --release
# If you want to use `s2n-quic`, build as follows. You may need several additional dependencies.
% cargo build --no-default-features --features http3-s2n --release
Then you have an executable binary rust-rpxy/target/release/rpxy
.
You can find the Jenkins CI/CD build scripts for rpxy
in the ./.build directory.
Prebuilt packages for Linux RPM and DEB are available at https://rpxy.gamerboy59.dev, provided by @Gamerboy59.
Note that we do not have an installation option via crates.io
, i.e., cargo install
, at this point since some dependencies are not yet published. Alternatively, you can use the docker image (see below) as the easiest way for amd64
environments.
rpxy
always refers to a configuration file in TOML format, e.g., config.toml
. You can find an example of the configuration file, config-example.toml
, in this repository.
You can run rpxy
with a configuration file like
% ./target/release/rpxy --config config.toml
rpxy
tracks changes to config.toml
in real-time and applies changes immediately without restarting the process.
The full help message is as follows.
usage: rpxy [OPTIONS] --config <FILE>
Options:
-c, --config <FILE> Configuration file path like ./config.toml
-l, --log-dir <LOG_DIR> Directory for log files. If not specified, logs are printed to stdout.
-h, --help Print help
-V, --version Print version
If you set --log-dir=<log_dir>
, the log files are created in the specified directory. Otherwise, the log is printed to stdout.
${log_dir}/access.log
for access log
${log_dir}/rpxy.log
for system and error log
That's all!
The most basic configuration of config.toml
is given like the following.
listen_port = 80
[apps.app1]
server_name = 'app1.example.com'
reverse_proxy = [{ upstream = [{ location = 'app1.local:8080' }] }]
In the above setting, rpxy
listens on port 80 (TCP) and serves incoming cleartext HTTP requests that include app1.example.com
in their HOST header or URL in their Request line.
For example, request messages like the following.
GET http://app1.example.com/path/to HTTP/1.1\r\n
or
GET /path/to HTTP/1.1\r\n
HOST: app1.example.com\r\n
Otherwise, a request to other.example.com
is simply rejected with status code 40x
.
If you want to host multiple distinct domain names on a single IP address/port, simply create multiple app."<app_name>"
entries in the config file like
default_app = "app1"
[apps.app1]
server_name = "app1.example.com"
#...
[apps.app2]
server_name = "app2.example.org"
#...
Note that by specifying a default_app
entry, HTTP requests will be served by the specified application if the HOST header or URL in the Request line doesn't match any server_name
s in reverse_proxy
entries. For HTTPS requests, it will be rejected since a secure connection cannot be established for an unknown server name.
The request message will be routed to the backend application specified with the domain name app1.localdomain:8080
or IP address over cleartext HTTP. If the backend channel needs to serve TLS, like forwarding to https://app1.localdomain:8080
, you need to enable the tls
option for the location.
revese_proxy = [
{ location = 'app1.localdomain:8080', tls = true }
]
You can specify multiple backend locations in the reverse_proxy
array for load-balancing with an appropriate load_balance
option. Currently it works in a round-robin manner, randomly, or round-robin with session-persistence using cookies. If load_balance
is not specified, the first backend location is always chosen.
[apps."app_name"]
server_name = 'app1.example.com'
reverse_proxy = [
{ location = 'app1.local:8080' },
{ location = 'app2.local:8000' }
]
load_balance = 'round_robin' # or 'random' or 'sticky'
First of all, you need to specify a port listen_port_tls
listening for HTTPS traffic, separately from the HTTP port (listen_port
). Then, serving an HTTPS endpoint can be easily done for your desired application by simply specifying TLS certificates and private keys in PEM files.
listen_port = 80
listen_port_tls = 443
[apps."app_name"]
server_name = 'app1.example.com'
tls = { tls_cert_path = 'server.crt', tls_cert_key_path = 'server.key' }
reverse_proxy = [{ upstream = [{ location = 'app1.local:8080' }] }]
In the above setting, both cleartext HTTP requests to port 80 and encrypted HTTPS requests to port 443 are routed to the backend app1.local:8080
in the same manner. If you don't need to serve cleartext requests, just remove listen_port = 80
and specify only listen_port_tls = 443
.
Note that the private key specified by tls_cert_key_path
must be in PKCS8 format. (See TIPS to convert PKCS1 formatted private keys to PKCS8 format.)
In the current Web, it is common to serve everything through HTTPS rather than HTTP, and hence HTTPS redirection is often used for HTTP requests. When you specify both listen_port
and listen_port_tls
, you can enable such redirection by setting https_redirection
to true.
tls = { https_redirection = true, tls_cert_path = 'server.crt', tls_cert_key_path = 'server.key' }
If it is true, rpxy
returns status code 301
to the cleartext request with the new location https://<requested_host>/<requested_query_and_path>
served over TLS.
rpxy
can, of course, route requests to multiple backend destinations according to path information. The routing information can be specified for each application (server_name
) as follows.
listen_port_tls = 443
[apps.app1]
server_name = 'app1.example.com'
tls = { https_redirection = true, tls_cert_path = 'server.crt', tls_cert_key_path = 'server.key' }
[[apps.app1.reverse_proxy]]
upstream = [
{ location = 'default.backend.local' }
]
[[apps.app1.reverse_proxy]]
path = '/path'
upstream = [
{ location = 'path.backend.local' }
]
[[apps.app1.reverse_proxy]]
path = '/path/another'
replace_path = '/path'
upstream = [
{ location = 'another.backend.local' }
]
In the above example, a request to https://app1.example.com/path/to?query=ok
matches the second reverse_proxy
entry in a longest-prefix-matching manner, and will be routed to path.backend.local
while preserving path and query information, i.e., served as http://path.backend.local/path/to?query=ok
.
On the other hand, a request to https://app1.example.com/path/another/xx?query=ng
matching the third entry is routed with its path information being rewritten as specified by the replace_path
option. Namely, the matched /path/another
part is rewritten to /path
, and it is served as http://another.backend.local/path/xx?query=ng
.
Requests that don't match any paths will be routed by the first entry that doesn't have the path
option, which serves as the default destination. In other words, unless every reverse_proxy
entry has an explicit path
option, rpxy
rejects requests that don't match any paths.
This path-based routing option would be sufficient in many cases. For example, you can serve multiple applications with one domain by specifying a unique path for each application. More specifically, see the example below.
[apps.app]
server_name = 'app.example.com'
#...
[[apps.app.reverse_proxy]]
path = '/subapp1'
replace_path = '/'
upstream = [ { location = 'subapp1.local' } ]
[[apps.app.reverse_proxy]]
path = '/subapp2'
replace_path = '/'
upstream = [ { location = 'subapp2.local' } ]
[[apps.app.reverse_proxy]]
path = '/subapp3'
replace_path = '/'
upstream = [ { location = 'subapp3.local' } ]
This example configuration demonstrates a very common path-based routing situation. When a request to app.example.com/subappN
is routed to subappN.local
by replacing the path part /subappN
with /
.
Since this is currently a work-in-progress project, we are frequently adding new options. We first add new option entries in config-example.toml
as examples. Please refer to it for up-to-date options. We will prepare comprehensive documentation for all options.
You can also use the docker
image hosted on Docker Hub and GitHub Container Registry instead of directly executing the binary. See the ./docker
directory for more details.
The ./bench
directory contains a very simple example of rpxy
configuration. This can also serve as an example of a docker use case.
rpxy
can serve HTTP/3 requests thanks to quinn
, s2n-quic
and hyperium/h3
. To enable this experimental feature, add an entry experimental.h3
in your config.toml
as follows. Any values in the entry like alt_svc_max_age
are optional.
[experimental.h3]
alt_svc_max_age = 3600
request_max_body_size = 65536
max_concurrent_connections = 10000
max_concurrent_bidistream = 100
max_concurrent_unistream = 100
max_idle_timeout = 10
Client authentication is enabled when apps."app_name".tls.client_ca_cert_path
is set for the domain specified by "app_name"
like
[apps.localhost]
server_name = 'localhost' # Domain name
tls = { https_redirection = true, tls_cert_path = './server.crt', tls_cert_key_path = './server.key', client_ca_cert_path = './client_cert.ca.crt' }
However, currently we have a limitation on HTTP/3 support for applications that enable client authentication. If an application is configured with client authentication, HTTP/3 doesn't work for that application.
If [experimental.cache]
is specified in config.toml
, you can leverage the local caching feature using temporary files and on-memory objects. An example configuration is as follows.
# If this specified, file cache feature is enabled
[experimental.cache]
cache_dir = './cache' # optional. default is "./cache" relative to the current working directory
max_cache_entry = 1000 # optional. default is 1k
max_cache_each_size = 65535 # optional. default is 64k
max_cache_each_size_on_memory = 4096 # optional. default is 4k if 0, it is always file cache.
A storable (in the context of an HTTP message) response is stored if its size is less than or equal to max_cache_each_size
in bytes. If it is also less than or equal to max_cache_each_size_on_memory
, it is stored as an in-memory object. Otherwise, it is stored as a temporary file. Note that max_cache_each_size
must be greater than or equal to max_cache_each_size_on_memory
. Also note that once rpxy
restarts or the config is updated, the cache is completely eliminated not only from the in-memory table but also from the file system.
This is a brand-new feature and may still be unstable. Thanks to rustls-acme
, automatic issuance and renewal of certificates are finally available in rpxy
. To enable this feature, you need to specify the following entries in config.toml
.
# ACME enabled domain name.
# ACME will be used to get a certificate for the server_name with ACME tls-alpn-01 protocol.
# Note that acme option must be specified in the experimental section.
[apps.localhost_with_acme]
server_name = 'example.org'
reverse_proxy = [{ upstream = [{ location = 'example.com', tls = true }] }]
tls = { https_redirection = true, acme = true } # do not specify tls_cert_path and/or tls_cert_key_path
For the ACME enabled domain, the following settings are referred to acquire a certificate.
# Global ACME settings. Unless specified, ACME is disabled.
[experimental.acme]
dir_url = "https://localhost:14000/dir" # optional. default is "https://acme-v02.api.letsencrypt.org/directory"
email = "[email protected]"
registry_path = "./acme_registry" # optional. default is "./acme_registry" relative to the current working directory
The above configuration is common to all ACME-enabled domains. Note that the HTTPS port must be open to the public to verify domain ownership.
Consider a case where rpxy
is running in a container. When the container manager maps port A (e.g., 80/443) of the host to port B (e.g., 8080/8443) of the container for HTTP and HTTPS, rpxy
must be configured with port B for listen_port
and listen_port_tls
. However, when you want to set https_redirection=true
for some backend apps, rpxy
issues redirection response 301 with port B by default, which is not accessible from outside the container. To avoid this, you can set a custom port for the redirection response by specifying https_redirection_port
in config.toml
. In this case, port A should be set for https_redirection_port
, then redirection response 301 will be issued with port A.
listen_port = 8080
listen_port_tls = 8443
https_redirection_port = 443
If you obtain certificates and private keys from Let's Encrypt, you have PKCS1-formatted private keys. You need to convert such retrieved private keys into PKCS8 format to use them in rpxy
.
The easiest way is to use openssl
:
% openssl pkcs8 -topk8 -nocrypt \
-in your_domain_from_le.key \
-inform PEM \
-out your_domain_pkcs8.key.pem \
-outform PEM
First, you need to prepare a CA certificate used to verify client certificates. If you do not have one, you can generate a CA key and certificate using OpenSSL commands as follows. Note that rustls
accepts X509v3 certificates and rejects SHA-1, and that rpxy
relies on Version 3 extension fields of KeyID
s of Subject Key Identifier
and Authority Key Identifier
.
- Generate a CA key of
secp256v1
, CSR, and then generate a CA certificate that will be set fortls.client_ca_cert_path
for each server app inconfig.toml
.
% openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:prime256v1 -out client.ca.key
% openssl req -new -key client.ca.key -out client.ca.csr
...
-----
Country Name (2 letter code) []: ...
State or Province Name (full name) []: ...
Locality Name (eg, city) []: ...
Organization Name (eg, company) []: ...
Organizational Unit Name (eg, section) []: ...
Common Name (eg, fully qualified host name) []: <Should not input CN>
Email Address []: ...
% openssl x509 -req -days 3650 -sha256 -in client.ca.csr -signkey client.ca.key -out client.ca.crt -extfile client.ca.ext
- Generate a client key of
secp256v1
and certificate signed by the CA key.
% openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:prime256v1 -out client.key
% openssl req -new -key client.key -out client.csr
...
-----
Country Name (2 letter code) []:
State or Province Name (full name) []:
Locality Name (eg, city) []:
Organization Name (eg, company) []:
Organizational Unit Name (eg, section) []:
Common Name (eg, fully qualified host name) []: <Should not input CN>
Email Address []:
% openssl x509 -req -days 365 -sha256 -in client.csr -CA client.ca.crt -CAkey client.ca.key -CAcreateserial -out client.crt -extfile client.ext
Now you have a client key client.key
and certificate client.crt
(version 3). A pfx
(p12
) file can be generated as follows:
% openssl pkcs12 -export -inkey client.key -in client.crt -certfile client.ca.crt -out client.pfx
Note that on macOS, a pfx
generated by OpenSSL 3.0.6
cannot be imported to macOS Keychain Access. We generated the sample pfx
using LibreSSL 2.8.3
instead of OpenSSL
.
All sample certificate files can be found in the ./example-certs/
directory.
Basically, Docker automatically manages your iptables if you use the port-mapping option, i.e., --publish
for docker run
or ports
in docker-compose.yml
. This means you do not need to manually expose your port, e.g., 443 TCP/UDP for HTTPS, using ufw
-like management commands.
However, we found that if you want to use the brand-new UDP-based protocol, HTTP/3, on rpxy
, you need to explicitly expose your HTTPS port using ufw
-like commands.
% sudo ufw allow 443
% sudo ufw enable
Your Docker container can receive only TCP-based connections, i.e., HTTP/2 or earlier, unless you manually manage the port. We see that this is strange and expect that it is some kind of bug (of Docker? Ubuntu? or something else?). But at least for Ubuntu 22.04 LTS, you need to handle it as described above.
Check the third-party project Gamerboy59/rpxy-webui
to manage rpxy
via a web interface.
todo!
rpxy
cannot be built without the following projects and inspirations:
rpxy
is free, open-source software licensed under MIT License.
You can open issues for bugs you've found or features you think are missing. You can also submit pull requests to this repository.
Contributors are more than welcome!
Footnotes
-
It is questionable whether this can be claimed to be written in pure Rust since the current
rpxy
is based onaws-lc-rs
for cryptographic operations. ↩ -
HTTP/3 is enabled thanks to
quinn
,s2n-quic
andhyperium/h3
. HTTP/3 libraries are mutually exclusive. You need to explicitly specifys2n-quic
with--no-default-features
flag. Also note that if you buildrpxy
withs2n-quic
, then it requiresopenssl
just for building the package. ↩ -
rpxy
supports the automatic issuance and renewal of certificates via TLS-ALPN-01 (RFC8737) of ACME protocol (RFC8555) thanks torustls-acme
. ↩ -
rpxy
supports the hybridized post-quantum key exchangeX25519MLKEM768
4 for TLS/QUIC incoming and outgoing initiation thanks torustls-post-quantum
. This is already a default feature. Also note thatX25519MLKEM768
is still a draft version yet this is widely used on the Internet. ↩ ↩2 -
By default,
rpxy
provides TLS connection sanitization by correctly binding a certificate used to establish a secure channel with the backend application. Specifically, it always maintains consistency between the given SNI (server name indication) inClientHello
of the underlying TLS and the domain name given by the overlaid HTTP HOST header (or URL in Request line). We should note that NGINX doesn't guarantee such consistency by default. To achieve this, you have to add anif
statement in the NGINX configuration file. ↩