Skip to content
Fixedmark

Networking · 11 min read

Updated

Building a Static IP Proxy in Rust

A static IP proxy maps each credential to one source address, binds every upstream socket to that address, and relays bytes without decrypting them. The Fixedmark proxy does this in Rust on tokio, with rustls for its own TLS endpoints, a hashed credential table swapped whole on reload, a destination check before every dial, and a metering wrapper on every upstream socket.

On this page

What the proxy has to do

A static IP proxy has one promise to keep. Traffic from a given customer leaves from that customer's IP, and only from that IP.

The Fixedmark proxy is one Rust binary, fixedmark-proxy. One node holds several customer IPs. Each IP belongs to a tenant row with an address (the public IP a partner allowlists) and a bind_address (what the upstream socket binds). They are the same on most hosts. On a cloud that maps a reserved public IP to a private one with 1:1 NAT, the socket binds the private address and the destination still sees the public one.

  • Accept HTTP CONNECT, absolute-form HTTP, and SOCKS5, in plaintext and over TLS.
  • Authenticate every connection, and let the credential pick the tenant and its source IP.
  • Refuse destinations a shared IP must never reach, such as private ranges and outbound mail ports.
  • Dial from the tenant's address, never the node's primary address, and never over IPv6.
  • Relay bytes both ways, pace them to the tenant's limits, and count every byte for billing.
  • Take new tenants, tokens, and certificates without a restart, and drain cleanly on shutdown.

If you want the protocol background first, read how an HTTP CONNECT proxy works and SOCKS5 versus socks5h. This post is about the code.

One tokio task per connection

The binary builds a multi-threaded tokio runtime and runs everything on it. The listeners are not in the config file. They arrive in a snapshot from the control plane, and each row names a bind address, a port, and one of six kinds (http_forward, socks5, tls_front, tls_socks, and two inbound kinds).

Each listener runs an accept loop. The loop looks up the handler for the listener's kind in a small table and spawns one task per accepted socket. A kind with no handler closes the connection at once and logs result="no_handler". Handlers share state through Arc pointers to the credential store, the resolver, and the guard that holds limits and policy.

Two details matter under bursts. The listen backlog is 4096, because tokio's default of 1024 overflows when 2,000 clients connect at once, and each dropped SYN waits a full second to retry. A failed accept sleeps 50 ms and retries instead of spinning.

Every task also holds a drain guard, which counts open connections and watches an abort signal for shutdown.

Parsing CONNECT and SOCKS5 without trusting the client

The HTTP side reads the request head with a hard cap of 8 KiB, 32 headers, and 10 seconds. It parses with httparse and accepts HTTP/1.0 too, which ncat still sends. A CONNECT target must be host:port as RFC 9110 describes, with a bracketed IPv6 literal and a non-zero port. Bytes the client sends after the blank line are kept and written upstream first, so an eager TLS client loses nothing.

Authentication runs before the method check. A request with a bad credential always gets 407, even when the method is wrong, so the proxy never tells an unauthenticated client anything about what it serves. An absolute-form request such as GET http://host/ goes to the forward path, which uses a pooled hyper client and strips proxy and hop-by-hop headers.

The SOCKS5 side follows RFC 1928 with RFC 1929 username and password, where the username is the app id and the password is the token. It reads each message with exact lengths, so no tunnel byte is consumed early. It serves only CONNECT, only the username and password method, and only IPv4 or domain targets.

SOCKS5 exchange as the proxy serves it
client  05 01 02                        greeting: version 5, one method, username/password
proxy   05 02                           method chosen
client  01 <ulen> <app id> <plen> <token>
proxy   01 00                           auth ok (01 01 on failure, then close)
client  05 01 00 03 <len> <name> <port> CONNECT to a domain name
proxy   05 00 00 01 <ip> <port>         succeeded; BND.ADDR is the customer's public IP

Every parser here has a fuzz target under fuzz/ (libFuzzer through cargo-fuzz), with a seed corpus of valid and broken inputs. The unit tests replay the same corpus, so a crash found by fuzzing becomes a regular test.

Credentials stored as hashes and swapped whole

The node never holds a token. The snapshot carries sha256(token) as 64 hex digits. The store keys each entry by the first 8 bytes of that hash. A lookup hashes the presented token, finds the entry by prefix, and compares the full hash in constant time with the subtle crate.

src/auth.rs (excerpt)
pub fn lookup(&self, app_id: &str, token: &str) -> Option<Arc<Tenant>> {
    let hash = TokenHash::of(token);
    let found = self.by_prefix.get(&hash.prefix())?;
    let same_hash = found.hash.matches(&hash);
    let same_app = found.tenant.tenant_id == app_id;
    (same_hash & same_app).then(|| found.tenant.clone())
}

Both comparisons run before they are combined, and the single & joins them without short-circuiting, so a wrong token and a wrong app id cost the same work. Two tokens whose hashes share a prefix refuse the whole snapshot rather than shadow each other.

The store lives behind an ArcSwap. A reload builds a new table and swaps the pointer, so a lookup sees the old snapshot or the new one, never a mix. A tunnel that is already open keeps the tenant it was authorized as. A tenant blocked by a reload keeps its open tunnels and is refused new ones, and an integration test checks exactly that.

Secrets stay out of logs. The credential types print [redacted] in Debug, the log formatter redacts secret-looking fields, and a test asserts a wrong token never appears in any log line.

Dialing from the customer's IP

This is the part that makes it a static IP proxy. Every upstream socket binds the tenant's bind_address before it connects. A failed bind is an error. Nothing falls back to an unbound socket, because a fallback would leave from the node's own address and the partner's firewall would reject it. A bind address of 0.0.0.0 is refused outright.

src/bind.rs (condensed)
let socket = Socket::new(Domain::IPV4, Type::STREAM, Some(Protocol::TCP))?;
socket.set_reuse_address(true)?;
set_bind_address_no_port(&socket)?; // setsockopt(IPPROTO_IP, IP_BIND_ADDRESS_NO_PORT, 1)
socket.bind(&SocketAddr::new(bind_address.into(), 0).into())?;
socket.set_nonblocking(true)?;
let socket = TcpSocket::from_std_stream(socket.into());
// then socket.connect(target) under one deadline

Binding to a specific address with port 0 normally makes the kernel reserve an ephemeral port at bind() time. With many tunnels from one address, that caps the address at one ephemeral range across all destinations. Linux's IP_BIND_ADDRESS_NO_PORT (Linux 4.2 and later) defers the choice to connect(), so a source port can be shared as long as the 4-tuple is unique. The proxy sets it on every socket and treats a refusal as an error, not a warning.

Resolution keeps only A records, so an IPv6-only host fails with no_ipv4 instead of leaving from an address nobody allowlisted. If you have hit that mismatch, the post on IPv6 egress and IPv4-only allowlists covers the client side. connect_any then tries each A record in resolver order under one shared deadline. A bind or socket option failure stops at once, since every address would hit it.

The integration tests prove the source address with loopback aliases. Linux answers on all of 127.0.0.0/8, so each test lane gives its tenants 127.0.n.2 and 127.0.n.3. An echo server reports the caller's address, and the test asserts that each credential arrives from its own alias.

Checking the destination before every dial

A customer IP is only useful while partners trust it, so every dial is checked in a fixed order before any byte moves.

  1. Limits. A blocked tenant, a tenant over conns_per_sec, or a tenant or node over its concurrent budget is refused before anything else.
  2. Ports, before DNS. The deny ports, the ports the node's cloud provider blocks (25 by default), and 465 and 587 for tenants without mail access.
  3. Every resolved address. Loopback, private, link-local, 0.0.0.0/8, the RFC 6598 shared range 100.64.0.0/10, multicast, and 240.0.0.0/4 are refused, and so is anything on the operator's DROP list.
  4. The credential's own rules. A credential with destination rules may reach only what one rule matches. A credential with no rules may reach anything the node rules allow.

One address in a denied range refuses the whole answer set, so a name that resolves partly into a private range never reaches it. The dialer then connects to exactly the addresses that were checked. It never does a fresh lookup, which closes the DNS rebinding gap where a second lookup returns an address the first one did not.

The DROP list is a file of IPv4 CIDRs reloaded on SIGHUP. The parser aggregates it into sorted, disjoint ranges, so a lookup is a binary search with partition_point. A file that fails to parse leaves the old list serving.

How each refusal reaches the client
CauseHTTP responseSOCKS5 reply
Missing or wrong credentials407, X-Proxy-Error: auth_failedauth status 01 (05 FF if no password method)
Tenant blocked407, X-Proxy-Error: blockedREP 02
Over conns_per_sec429, rate_limited, Retry-After: 1REP 02
Over the concurrent limit429, too_many_connectionsREP 02
Private range or DROP list403, destination_deniedREP 02
No credential rule matches403, denied_by_policyREP 02
Port 25403, smtp_blocked_by_providerREP 02
Port 465 or 587 without mail access403, smtp_deniedREP 02
DNS failure, refused, or timeout502, upstream_unreachableREP 04, 05, or 06

Each refusal has exactly one response and one result value in the logs. The full debugging view is in proxy error codes explained and the troubleshooting docs.

The relay and its memory budget

Once the dial succeeds, the proxy answers HTTP/1.1 200 Connection Established (or SOCKS5 REP 00) and hands both sockets to one relay function. It is tokio's copy_bidirectional_with_sizes, wrapped in two adapters and raced against an idle timer.

src/relay.rs (condensed)
let mut upstream = Throttled::new(Counted::new(upstream, meter), paced);
let activity = upstream.activity();
let outcome = tokio::select! {
    copied = copy_bidirectional_with_sizes(client, &mut upstream, RELAY_UP, RELAY_DOWN) => ...,
    idle = activity.idle(limiter.idle()) => Err(Broken::Idle(idle)),
};

Counted wraps the socket that faces the destination and adds every byte to the tenant's meter as it moves. Throttled paces reads and writes to the tenant's bytes_per_sec with a governor rate limiter (GCRA), taking quota in chunks of at most 16 KiB with a quarter-second burst. It also stamps the last time a byte moved. A tunnel that moves nothing for the tenant's idle time (300 seconds by default) closes with idle_timeout.

tokio allocates both relay buffers up front, and they live as long as the tunnel. That makes buffer size the node's main memory cost, and the admission cap is derived from it. At 64 KiB each way, the buffers alone would cost 128 KiB per tunnel. The buffers are 16 KiB up (mostly requests) and 32 KiB down (where downloads go), plus a measured 20 KiB allowance for the task, socket handles, and rustls state. That is 68 KiB per tunnel, so the default 512 MiB budget admits about 7,700 tunnels. Kernel socket memory is outside this budget and is bounded separately by net.ipv4.tcp_mem.

TLS to the proxy on port 443

Plain CONNECT and SOCKS5 send the proxy credential in cleartext, which RFC 1929 itself warns about. So Fixedmark serves both protocols over TLS too, on ports 443 and 1443, with plain ports 8080 and 1080 kept for clients that cannot open TLS to a proxy. The post on HTTPS proxy endpoints explains which clients support that.

TLS uses rustls with the ring provider, TLS 1.2 and 1.3 only, and no client certificates. The 443 listener does not terminate TLS blindly. It first peeks the ClientHello without consuming it and reads the server name. A name under the proxy's own domain, an IP literal, or no name at all goes to the front, where rustls terminates the outer TLS and the decrypted stream goes to the same CONNECT and forward code as port 8080. A name that an inbound route serves is passed through undecrypted. A name nothing serves is refused before a ServerHello is sent.

Inside the front, the client's own TLS session to the destination still runs end to end. The proxy decrypts only its outer layer, which carries the CONNECT line and the credential. The relay does read the server name from the inner ClientHello for the connection event, the same SNI any network observer sees. The TLS passthrough post covers what a passthrough proxy can and cannot see.

Certificates reload without a restart. SIGHUP, a file change, or a 10-second metadata poll (which catches a renewal that swaps a symlink target) rebuilds the config and swaps it in. A pair that fails to parse, does not match its key, or has expired keeps the old one serving and counts fm_proxy_tls_reload_total{result="failed"}. A gauge exports the leaf's expiry time, and the daily log line turns into a warning under 14 days left.

Logs, metrics, metering, and shutdown

Every connection writes exactly one connection closed line when it ends, with the tenant, the egress IP, the destination host and port, bytes each way, the duration, a result, and a reason where the result alone is ambiguous. The same close also increments fm_proxy_connections_total by direction, listener, and result, and appends one JSON event to a rotating file. Prometheus metrics and /healthz are served on an admin address that the config requires to be loopback.

No log or event write may block a tunnel. Log lines go through a bounded queue of 4,096 lines to one OS thread that writes stdout, so a stalled journald never stalls a tokio worker. A full queue drops the line and counts it in fm_proxy_log_lines_dropped_total. Events follow the same rule with their own counter.

Usage metering follows that rule too. Meters are atomic counters per tenant, IP, and direction. A flusher swaps them to zero every window (10 seconds in the example config) and offers the batch to a channel without waiting. If the channel is full, the batch stays pending and the next window merges into it, so a slow consumer never blocks a connection or loses a row. The node agent reads batches over a gRPC service (tonic) on a Unix socket and acknowledges each one.

  1. 1
    Stop taking workSIGTERM sets /healthz to draining (503) and closes every listener. Open tunnels keep running.
  2. 2
    DrainWait up to drain_timeout_secs for the open count to reach zero, then abort what is left so no tunnel moves bytes after the last usage flush.
  3. 3
    Flush and confirmFlush the last partial usage window and, when usage goes to the control socket, wait for the agent to acknowledge it. Then stop the control socket and drain the events writer. Each step has its own timeout.

Two smaller choices keep a bad connection from taking down the node. The crate denies unwrap and expect with a clippy lint, and the release profile keeps panic = "unwind", so a panic ends one connection's task rather than the process. Snapshot reloads bind any new listener sockets before publishing the new tenants, so a reload whose new outbound listener hits a taken port changes nothing.

If you want to use this proxy rather than build one, Fixedmark is in early access. Each app gets a pair of static IPs in a region and one proxy URL. The concepts docs cover the four endpoints, and verify your egress IP shows how to check that traffic leaves from the right address.

Terminal
curl -v --proxy "$FIXEDMARK_PROXY_URL" https://api.partner.example/v1/orders

# Proxy refused the tunnel:
#   < HTTP/1.1 407 Proxy Authentication Required
#   < X-Proxy-Error: auth_failed
#   curl: (7) CONNECT tunnel failed, response 407   (exit code from curl 8.21)
#
# Tunnel open, destination answered:
#   < HTTP/1.1 200 Connection Established

Sources

Checked November 2026. Confirm current details on the vendor's site before you decide. Prices and limits change.

Blog

Frequently asked questions

Does the proxy decrypt HTTPS traffic?

No. A CONNECT or SOCKS5 tunnel carries the client's TLS session to the destination as opaque bytes. On the TLS endpoints the proxy terminates only its own outer TLS layer, which carries the CONNECT request and the proxy credential.

How does the proxy choose which IP a request leaves from?

The credential decides. Each token hash maps to a tenant row with a `bind_address`, and every upstream socket binds that address before it connects. A failed bind is an error, never a fallback to the node's own address.

Why does the proxy refuse IPv6 destinations?

Customer IPs are IPv4, and partners allowlist IPv4 addresses. Leaving over IPv6 would come from an address nobody allowlisted, so the resolver drops AAAA records and an IPv6-only host fails with a clear `no_ipv4` reason.

What is IP_BIND_ADDRESS_NO_PORT for?

It tells Linux not to reserve an ephemeral port when a socket binds an address with port 0. The port is picked at connect time and can be shared across destinations as long as the 4-tuple is unique, so one source address is not capped at one ephemeral range.

Make it fixed.

Fixedmark is in early access. Join the list to get dedicated static IPs when your region opens.