Tunnels: hostname routes to your machine

What a tunnel route is

• A route belongs to an app. Its IPs are placements, chosen by Fixedmark. A route on the shared pool is placed on two shared ingress IPs, one on each of two nodes in its region.
• fixedmarkd on your machine dials each of those IPs on 443 and holds one session to each. Each partner connection comes down one of those sessions to the local target you name, such as 127.0.0.1:8080.
• A node reads only the server name in the partner's TLS ClientHello, or the port for a TCP route, and splices the bytes. It never holds a key for your traffic.
• Every control you know from inbound IPs works the same: partner allowlists, inbound metering, connection logs, webhooks, and the ingress block.
• A shared route takes a connector origin, since a shared IP never dials out to a host. For a host origin, or for traffic that must arrive only at your own address, use the dedicated inbound IP add-on.

Route kinds

The kinds are spelled http, tls, tcp, and access in the CLI, the API, and fixedmarkd's plan. The API refuses any other kind with route.kind_invalid.

Before you start

• Tunnels on Pro and up, with a dedicated inbound IP as an add-on at $9 per IP-month. Your plan's routes, hostnames, connectors, and inbound bandwidth are in Limits. A plan without tunnels refuses a route with route.tunnel_not_entitled (402).
• An app. fm apps list prints its id and name. --app takes either.
• A Linux or macOS machine on amd64 or arm64 that can reach your service and make outbound connections to port 443. fixedmarkd does not run on Windows yet.
• The partner's source addresses, as IPv4 addresses or CIDRs. Every route except access admits only the addresses you list.

Quick start: a first hostname route

Create a connector, install fixedmarkd as a service on the machine that holds your service, then create an http route that names the connector.

fm connectors create --app shop --name laptop
sudo fixedmarkd install --token-file fm-connector-laptop.token

fm connectors create writes the token to a file readable by you only and prints the install line. Run the install line on the service's machine, from the directory that holds the file, then delete the file. Connectors covers the service and its files.

fm routes create --app shop --kind http --name web \
  --connector laptop --to 127.0.0.1:8080 \
  --partner 198.51.100.0/24

curl -X POST "https://api.fixedmark.com/v1/apps/APP_ID/routes" \
  -H "Authorization: Bearer $FIXEDMARK_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: $(uuidgen)" \
  -d '{"kind":"http","name":"web","origin":{"kind":"connector","connector_id":"CONNECTOR_ID","target":"127.0.0.1:8080"},"partner_cidrs":["198.51.100.0/24"]}'

The API answers 202 with the route and a job_id. fm prints the route:

route ROUTE_ID
name web
kind http
hostname web.acme.fmtunnel.test
mode shared
region REGION
state pending
dns unpublished
placements 2
  IP_ID  203.0.113.20:443
  IP_ID  203.0.113.21:443
job JOB_ID
watch it with: fm jobs watch JOB_ID

Within about a minute fixedmarkd reads the route from its plan, attaches a session to each IP, and orders the hostname's certificate. Then, from a partner address:

curl -sv https://web.acme.fmtunnel.test/

The request reaches 127.0.0.1:8080 on your machine as plain HTTP. fm routes show web reads the route's state, health, and DNS record, and fixedmarkd status shows the sessions and the certificate.

Hostnames

Every route except access gets the hostname <name>.<org-slug>.<zone>, such as web.acme.fmtunnel.test. Partners use that name for every kind. A tcp route adds its port, such as db.acme.fmtunnel.test:20000.

• <name> is the route's name, from --name. It is 1 to 40 lowercase letters, digits, and single hyphens, starts and ends with a letter or digit, and does not start with xn. It is unique among your org's live routes. Without --name the route is named its kind and the first 8 hex digits of its id, such as tcp-5f0c2b1e.
• Names such as www, api, app, login, and status, and any name starting pg- or ip-, are reserved and refused with route.name_reserved. So is any name containing fixedmark. The same rules hold for the org slug.
• <org-slug> is set by your org's first hostname route, from --org-slug or your org's name, and it is fixed after that. A later route always gets that slug. A first route that asks for a slug another org holds is refused with route.slug_taken.
• Your org's slug is its own label under the zone, but that label does not keep cookies apart between orgs yet. Browsers treat each org's hostnames as separate sites only once the tunnel domain is on the Public Suffix List, and that entry is pending. Until then, set cookies without a Domain attribute, so each one stays on the hostname that set it.
• Every org shares the weekly certificate budget, with a share for each org. See Certificates.

Passthrough and HTTP termination

A tls or tcp route is passthrough. The partner's bytes reach your target unchanged, and your target holds the certificate, if any. fixedmarkd does not hand its own certificates to a local TLS server yet, so a tls target needs a certificate of its own that its partners accept.

An http route terminates TLS in fixedmarkd, on your machine. The node still passes the encrypted bytes through, and the key never leaves your machine. fixedmarkd then sends each request to your target as plain HTTP.

• Partners speak HTTP/2 or HTTP/1.1, picked by ALPN. Your target always gets HTTP/1.1.
• WebSockets work over HTTP/1.1 upgrades and over HTTP/2 extended CONNECT. Other upgrade tokens are not passed.
• The Host header is kept as the partner sent it. A request for any other host than the route's hostname gets 421. A Host that is not all visible ASCII gets 400.
• An upgrade request that declares a body, by a Content-Length other than 0 or by a Transfer-Encoding, gets 400.
• A plain CONNECT gets 405, so no partner can tunnel past your target.
• A target that refuses or fails the request gets the partner a bare 502, and fixedmarkd logs upstream failed.
• Up to 256 requests and WebSockets go to your targets at once. A request past that cap gets 503, and so does one that needs a new target connection while the connection cap is full.
• A connection with nothing in flight closes after 75 seconds. A request's headers must arrive within 30 seconds and fit in 64 KiB, else the partner gets 431 for size. fixedmarkd waits 10 seconds for your target to accept.

fixedmarkd drops these headers from the partner's request: every header whose name starts X-Forwarded, plus Forwarded, X-Real-IP, Fixedmark-Tunnel-Request, and the client-address headers X-Client-IP, True-Client-IP, CF-Connecting-IP, X-Cluster-Client-IP, Fastly-Client-IP, X-Original-Forwarded-For. A name matches in any case and with _ in place of -, so X_Forwarded_For is dropped too. CGI, WSGI, Rack, and PHP read that name as X-Forwarded-For. Then fixedmarkd sets its own headers. It never appends to a partner's value.

fixedmarkd sets no other header. Any header it does not drop, such as Authorization or Cookie, reaches your target as the partner sent it.

Certificates

fixedmarkd orders the certificate for each http route's hostname from Let's Encrypt, by the ACME DNS-01 challenge. It makes the key on your machine. Fixedmark only writes the challenge's TXT record in the tunnel zone for it, so no key and no certificate ever leaves your machine.

1. fixedmarkd makes a key, opens an order, and asks the control plane to publish the challenge value at _acme-challenge.<hostname>.
2. It waits until every Fixedmark nameserver answers the value, then asks the CA to validate.
3. It stores the certificate and serves it from the next handshake. The challenge record is removed.
4. It renews in the window the CA's renewal information names, else at two thirds of the lifetime, and checks once an hour. A renewal never counts against the budget below.

Until the first certificate is issued, a partner's connection to the route is refused with cert_unavailable, and fixedmarkd status shows the certificate as queued or issuing.

• The directory holds the ACME account and one key and certificate per hostname. Every file is mode 0600.
• fixedmarkd refuses a state directory that is a symbolic link or that another user owns, and exits 2 with fixedmarkd.state_dir.
• sudo fixedmarkd uninstall leaves the state directory in place, so the keys stay on the machine. Remove the directory yourself when you retire the machine.
• On macOS the directory is owned by nobody, which other services on the machine may also run as. Run fixedmarkd on a machine where that is acceptable.

Until the tunnel domain is on the Public Suffix List, Let's Encrypt allows the zone a small number of new certificates a week. Fixedmark shares it out: at most 45 first certificates across every org in 7 days, and at most 9 for one org. A first certificate past either share waits, and fixedmarkd retries after the time the control plane names. Renewals are exempt.

Two sessions per route

• fixedmarkd holds one session to each IP that carries one of its routes. A shared route has two placements, so it has two sessions.
• A partner reaches either IP. Each node sends the connection down its own session, so both sessions serve traffic all the time.
• fixedmarkd reads its plan every 60 seconds. When a dial to an IP fails, it reads the plan again within about 8 seconds, then backs off to once a minute, so it follows a route that moved without a restart. It stops these extra reads 10 minutes after the first failure, or as soon as the plan changes.
• One fixedmarkd per token. A second one with the same token replaces the first one's sessions.

Partners

Every route except access admits only the partner addresses you list. The node checks the list before anything reaches your machine. Setting partners replaces the whole list.

fm routes partners set web --partner 198.51.100.0/24 --partner 192.0.2.44

A public website takes --partner 0.0.0.0/0 --allow-any. Without --allow-any, that entry, or any entry wider than a /8, is refused with route.partner_open_to_all, and with it the route carries the warning open_to_all.

DNS records

Each hostname has one CNAME record in the tunnel zone. A shared route's record points at its placement group's name, pg-<id>.<zone>, which holds one A record for each of the group's two IPs. Every record has a TTL of 60 seconds.

web.acme.fmtunnel.test.            60  IN  CNAME  pg-0123456789ab.fmtunnel.test.
pg-0123456789ab.fmtunnel.test.     60  IN  A      203.0.113.20
pg-0123456789ab.fmtunnel.test.     60  IN  A      203.0.113.21

• The record is written about two seconds after the route is created. The route's dns.state reads unpublished right after the create, as fm prints it, then pending, then synced within seconds.
• When only one of the route's placements is healthy, the CNAME points at that IP's own name, ip-<id>.<zone>, and returns to the group when both are.
• Health never removes the record. It goes only while the route is disabled or has no placement, and when you delete the route.
• dns.state refused means the DNS provider refused the record. Fixedmark retries it when the route changes and at an hourly check. dns.error says why.

Health and failover

Fixedmark judges each placement from its node's heartbeat, its listener, and your connector's session, and rolls those up into the route's health. Failover is by DNS, in three layers.

• These bounds are measured with 2 second heartbeats in Fixedmark's test lanes and scaled to the 10 second heartbeats production nodes send.
• A route never moves to another region by default. When every group of its region is down, it waits there, and Fixedmark is paged. Cross-region moves are off until Fixedmark turns them on.
• A failure on your side, such as no_session or origin_down, never moves a route, since a move would not fix it.
• A route on a dedicated IP is never re-placed. It follows its IP.
• Partners that dial the hostname follow every move. A partner that dials an address it looked up once must look it up again.

Status fields

fm routes show web
fixedmarkd status

fixedmarkd status prints one line per IP with its session state, connecting, attached, detached, or stopped, and a CERTS table with one row per hostname: queued, issuing, valid, renewing, renewal_failing, or failed, with the expiry and the last error. --json prints the same file, with each http route's request and error counters.

Move a route to a dedicated IP

A route keeps its name and hostname when it moves between the shared pool and a dedicated IP of its app.

fm routes placement web --dedicated-ip IP_ID
fm routes placement web --shared

• PUT /v1/apps/APP_ID/routes/ROUTE_ID/placement takes {"mode":"shared"} or {"mode":"dedicated","ip_id":"IP_ID"}. It answers 202 when the route moves.
• Asking for the placement the route already has changes nothing. The API answers 200 with changed false, and fm prints unchanged: the route is already there.
• A tcp route with partners gets a new address and maybe a new port, which its partners dial. It moves only with --confirm, or confirm: true, else route.placement_unconfirmed (409).
• A dedicated placement needs the add-on. A plan without it refuses with route.dedicated_not_entitled (402).

fm routes reference

A route is named by its id or its name. Commands that take one route search every app your key reads unless you pass --app.

A tcp route takes the same flags with --kind tcp, such as fm routes create --app shop --kind tcp --name db --connector laptop --to 127.0.0.1:5432 --partner 198.51.100.0/24. Partners then reach db.acme.fmtunnel.test on the port the route prints.

Limits and quotas

• Every route with a hostname counts against hostnames, and every route against routes. Past either, a create is refused with route.hostname_quota or route.quota (409), and the error's limit names the quota.
• Tunnels come with paid plans only. The trial has none.
• A tcp route takes one port from 20000 to 29999, the same port on both IPs of its group.
• A route admits 1 to 256 partner entries, IPv4 only.
• fixedmarkd holds up to 256 partner connections at once by default, set by --max-streams.
• Inbound bytes count toward your plan's inbound bandwidth, apart from outbound. Ingress packs add more for one period.

See inbound limits for the rest. Each count comes from the plan catalog; 3 plans on sale include tunnels.

Troubleshooting

fm exits 2 on a malformed request (400 or 422) and 1 on any other refusal, such as 402, 409, or 503. Either way it prints the code. When you write to support, include the route id and the connector_id, never the token.