Skip to content
Fixedmark

Networking · 11 min read

Updated

Rotating Proxy Credentials Without Downtime

Never swap a proxy credential in one step. Create the new token while the old one still works, move every client to it, confirm the old token has gone quiet in your logs, then revoke it. Make the overlap longer than the slowest path a secret takes to reach your code, and have clients reload the token and retry once when the proxy answers `407`.

On this page

Why a credential swap breaks outbound calls

A proxy credential is usually a username and password, or an app ID and a token, inside a URL such as https://APP_ID:TOKEN@proxy.example.com:443. Every call your app sends through the proxy depends on it. If the proxy stops accepting the credential before every copy is replaced, those calls fail with 407 Proxy Authentication Required. They fail together, because they all share one secret.

A credential has more copies than the one you just changed in the secret store.

  • Caches in front of the store. The AWS Parameters and Secrets Lambda Extension keeps a secret for up to 300 seconds by default and says it "doesn't detect changes" before that TTL runs out.
  • Environment variables of running processes. A process reads its environment once, when it starts. Kubernetes states that ConfigMaps consumed as environment variables "are not updated automatically and require a pod restart."
  • Build artifacts. On Vercel, changes to environment variables "only apply to new deployments." An old deployment URL keeps the old value.
  • Open connections. An HTTP CONNECT tunnel is authenticated once, when it opens. A pool of kept-alive tunnels can keep running on a credential that no config file holds any more.

A one-step swap starts a race between the provider and all of those copies. AWS describes the same race for database passwords in its single-user rotation strategy. There is "a short period of time" between the password change and the secret update, and calls can be denied during it. For a proxy that carries every partner call your app makes, a short period still means a burst of failed payouts, orders, or database connections.

Keep two valid credentials during the switch

The fix is an overlap. For a window of time the proxy accepts both the old and the new credential. Clients move across at their own pace, and none of them ever holds a dead credential.

The order of the steps is what makes this work. Vercel's guide to rotating secrets says to update your app "before you invalidate the old credential." Create, distribute, verify, revoke. Never revoke first.

Three ways a provider can handle rotation
StrategyHow it worksExample
Rotate with a grace periodThe provider issues a new secret and keeps the old one valid until a deadline.Stripe keeps both keys working for up to 7 days when you rotate a key in the Dashboard.
Two independent credentialsYou create a second credential, move clients to it, then revoke the first. Next time you rotate the other one.AWS Secrets Manager's alternating-users strategy, after which "both user and user_clone credentials are valid."
Hard cutThe old secret stops working the moment the new one exists.Choosing Now as the expiry when you rotate a Stripe key, which deletes the old key.

Two independent credentials give you the most control. The overlap lasts as long as you need, not as long as the provider's grace period, and you revoke on evidence instead of on a timer. Stripe gives the same advice when its 7 days are not enough. Create a new key by hand, migrate to it, then expire the old one.

A grace-period rotate is fine when you can finish well inside the window. Keep the hard cut for a leaked credential, where a short outage is the smaller harm.

Size the overlap to the slowest copy

The overlap has to outlast the slowest path from your secret store to the code that makes the call. Add up the delays your setup really has.

When each kind of copy picks up a new value
Where the token livesWhen it changesSource
Mounted Kubernetes Secret volumeEventually. The delay can be the kubelet sync period plus the cache propagation delay. A subPath mount never updates.Kubernetes Secrets
Environment variable in a containerAt the next container start.Kubernetes ConfigMaps
Lambda extension cacheWhen SECRETS_MANAGER_TTL expires. 300 seconds is both the default and the maximum.AWS Systems Manager
Vercel environment variableAt the next deployment. Old deployments keep the old value.Vercel
Open tunnel or pooled connectionWhen the connection closes and the client opens a new one.Your client's keep-alive and pool settings

Then add the delays that are not timers. A deployment that fails halfway leaves some instances on the old value. A rollback to last week's build brings back last week's token, which is why Vercel warns that an old deployment "will still use the old credential." A job that runs once a month only notices the change on its next run, and that run is often how teams learn they revoked too early.

So set the window longer than your longest job interval and longer than the oldest build you would roll back to. Then make the window matter less, by checking usage before you revoke. The runbook below covers that check.

Make clients reload the token without a deploy

Overlap protects you from slow copies. The other half of the job is making your copies fast. Code that reads the proxy URL once at startup needs a restart or a redeploy for every rotation. Code that reads it from a mounted file or a secret store, with a short cache, picks up a rotation on its own.

Two rules keep this safe. First, build a new proxy agent when the value changes, instead of editing headers on a live one, and let requests in flight finish on the old agent. Second, when the proxy answers 407, reload the secret and retry once. RFC 9110 allows exactly this. After a 407, "the client MAY repeat the request with a new or replaced Proxy-Authorization header field."

The retry is safe even for a POST. A proxy refuses the CONNECT before any byte of your request leaves, so the partner never saw the first attempt. Retry only once, so a token that is really wrong fails fast instead of looping.

import { readFile } from "node:fs/promises";
import { fetch, ProxyAgent, type RequestInit } from "undici";

// Your secret store keeps this file current, for example a mounted
// Kubernetes Secret or a sidecar that writes the latest version.
const PROXY_URL_FILE =
	process.env.PROXY_URL_FILE ?? "/var/run/secrets/proxy/url";
const MAX_AGE_MS = 60_000;

let current: { url: string; agent: ProxyAgent; loadedAt: number } | undefined;

async function proxyAgent(force = false): Promise<ProxyAgent> {
	if (current && !force && Date.now() - current.loadedAt < MAX_AGE_MS) {
		return current.agent;
	}
	const url = (await readFile(PROXY_URL_FILE, "utf8")).trim();
	if (current?.url === url) {
		current.loadedAt = Date.now();
		return current.agent;
	}
	const previous = current?.agent;
	current = { url, agent: new ProxyAgent(url), loadedAt: Date.now() };
	// close() lets requests already on the old agent finish first.
	previous?.close().catch(() => {});
	return current.agent;
}

// undici reports a refused CONNECT as an error whose message carries
// the proxy's status code, wrapped in fetch's TypeError.
function isProxyAuthError(err: unknown): boolean {
	for (let e: unknown = err; e instanceof Error; e = e.cause) {
		if (e.message.includes("(407)")) return true;
	}
	return false;
}

export async function proxiedFetch(url: string, init: RequestInit = {}) {
	try {
		return await fetch(url, { ...init, dispatcher: await proxyAgent() });
	} catch (err) {
		if (!isProxyAuthError(err)) throw err;
		// The proxy refused the token before the request left, so a
		// retry is safe even for a POST. Reload the secret and try once.
		return fetch(url, { ...init, dispatcher: await proxyAgent(true) });
	}
}

We ran both versions against a small CONNECT proxy that started refusing the old token partway through, with undici 7.30 and 8.12 and httpx 0.28. undici reports a refused tunnel as Proxy response (407) !== 200 when HTTP Tunneling, two levels down the cause chain of fetch's TypeError: fetch failed. It tunnels plain http:// destinations too, so one check covers both. httpx raises httpx.ProxyError for a refused tunnel and returns an ordinary 407 response for a plain http:// destination.

A cache of a minute means a rotation reaches every warm process about a minute after the file changes, without a call to the secret store on every request. If you route only some calls through the proxy, keep this helper on those calls alone, as routing only some fetches through a proxy describes.

Proxy details that catch people during rotation

  • Open tunnels outlive revocation at many proxies. A CONNECT proxy checks Proxy-Authorization when the tunnel opens, then, in RFC 9110's words, restricts itself "to blind forwarding of data." In our test, a kept-alive tunnel opened with the old token went on serving requests after the proxy stopped accepting that token. That helps availability. It also means revoking a leaked token does not close a tunnel an attacker already holds, unless your proxy closes it. Ask your provider which behaviour it has. How a CONNECT proxy works shows why.
  • New tokens need URL-encoding. A generated token can contain @, :, /, or %. Inside a proxy URL those must be percent-encoded, or the client splits the URL in the wrong place and sends a broken credential.
  • Copy and paste leaves marks. A trailing newline makes a valid token fail, and so do quotes left around the value. Trim whitespace when you read the value, as both examples above do, and fix stray quotes where the secret is stored.
  • Cleartext proxy ports expose the token. On a plain http:// proxy URL the credential crosses the network as base64 in a Proxy-Authorization: Basic header, which anyone on the path can decode. Rotate such a token more often, or use an HTTPS proxy endpoint, which encrypts it.
  • One shared token means one big rotation. If staging, production, and the nightly job share a token, rotating it touches all three at once. Give each its own token, so a rotation or a leak affects one service.
  • Database pools through SOCKS5 hold connections for a long time. A pool that never recycles a connection keeps using the old credential until it reconnects. Set a maximum connection lifetime. Pooling through SOCKS5 covers the pool settings.

When something does fail, find out who refused you. A 407 on the CONNECT comes from the proxy. A 401 after the tunnel opened comes from the destination, which means the partner's own API key is the problem, not the proxy token. Proxy error codes explained walks through telling them apart with curl -v.

A rotation runbook

  1. 1
    List every holderWrite down each service, environment, scheduled job, CI workflow, and laptop that holds the credential. If you cannot list them, the rotation will find them for you, so do this first.
  2. 2
    Create the new credentialCreate it while the old one still works. Copy it straight into your secret store, because many providers show a token only once.
  3. 3
    Test it aloneSend one request through the proxy with the new credential before any app uses it. The command below prints the proxy's answer to the CONNECT and the destination's status separately.
  4. 4
    Store it as the current versionAdd it as a new version of the same secret, not as a new secret name, so code that reads the current version picks it up. In AWS Secrets Manager, PutSecretValue moves AWSCURRENT to the new version and AWSPREVIOUS to the old one, so the last good value stays one call away.
  5. 5
    Roll out in stagesLet reloading clients pick it up, then restart or redeploy everything that reads the value at startup. Start with one service and watch for 407s before you do the rest.
  6. 6
    Wait for the old credential to go quietUse the proxy's logs to confirm nothing has used the old credential for a while. Stripe's advice for its own keys is to expire the old one "only after its request volume has been at zero for a few hours or days."
  7. 7
    Revoke, then keep watchingRevoke the old credential and watch 407 rates for a day. If a holder you missed breaks, you will know which credential it held and where to look.
Terminal
# Test the new proxy URL before any app depends on it.
curl -sS -o /dev/null \
  -w "proxy %{http_connect}, destination %{http_code}\n" \
  --proxy "$NEW_PROXY_URL" https://api.partner.example/health
# proxy 200, destination 200   new credential works
# proxy 407, destination 000   proxy refused it

# Store it as the current version. Reading it from a file keeps
# the token out of your shell history and process list.
aws secretsmanager put-secret-value \
  --secret-id prod/proxy-url \
  --secret-string file://new-proxy-url.txt
Script these steps once you have done them by hand. A script that runs create to revoke, with the quiet-period check in the middle, beats a person working from memory once a year.

When to rotate, and how often

Rotate at once when a credential is exposed. That includes a token committed to a repository, printed in a log, pasted into a chat, or known to someone who has left the team. Stripe's guidance on exposed keys is to rotate "immediately even if you aren't sure anyone saw it." For a leak, the hard cut is usually right, and the overlap is the risk you are closing.

Scheduled rotation earns its place for a different reason. Stripe says periodic rotation "confirms that you know where each key is used and that your team can replace a key on short notice." A team that rotates on a schedule has a tested runbook on the day of a leak. A team that never rotates builds its inventory during the incident.

Pick an interval you will keep, and automate it.

Rotating a Fixedmark token

Fixedmark's proxy URL carries your app ID as the username and a token as the password. A few properties from the concepts docs shape how rotation works there.

  • Rotating a token never changes your IPs. Partners allowlist your static IP pair, not your token, so a rotation needs no ticket to a bank or broker.
  • Each app can hold several tokens. Give each environment or service its own, and rotate one without touching the others.
  • Fixedmark stores only a hash of each token, so copy the token when you create it. The CLI prints it once, on create and on rotate.
  • A token can carry a destination allowlist, so a leaked token reaches only the hosts and ports you listed.
  • Proxy nodes check tokens locally from a cached copy, so authentication keeps working if the Fixedmark control plane is down.

The REST API supports both overlap strategies. POST /v1/apps/{id}/credentials/{cred}/rotate replaces a credential's token, and the old one works until its grace ends. The reply holds the new credential with its token, and the previous credential with its expires_at. For an overlap you control, create a second credential, move your clients to it, and revoke the old one.

Terminal
# Create a second credential. The reply's token is shown only here.
curl -X POST https://api.fixedmark.com/v1/apps/APP_ID/credentials \
  -H "Authorization: Bearer $FIXEDMARK_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"label": "prod-2026-11"}'

# After rollout and a quiet period, revoke the old one.
curl -X DELETE https://api.fixedmark.com/v1/apps/APP_ID/credentials/OLD_CRED_ID \
  -H "Authorization: Bearer $FIXEDMARK_API_KEY"

The fm CLI wraps the same routes as fm credentials list|create|rotate|revoke --app <app>, and fm api <METHOD> <path> calls any route directly. Listing shows token prefixes only, which is enough to tell tokens apart.

Before you revoke, check the connection logs. Each row carries the credential_id that opened the connection, never the token, so you can confirm the old credential has stopped opening connections. A revoked credential applies to new connections. A tunnel opened before the revocation stays open until it closes.

fm tunnel reads its token from its config file or FIXEDMARK_SOCKS_URL and checks it when it starts, so restart the tunnel after you change either one. If a client still gets 407 after a rotation, the troubleshooting guide lists the usual causes. Fixedmark is in early access, with planned pricing and a security model you can read first.

Sources

Checked November 2026. Confirm current details on the vendor's site before you decide. Prices and limits change.

Blog

Frequently asked questions

Does rotating a proxy token change my outbound IP?

Not with a static IP proxy. The IP belongs to the proxy or your plan, and the token only proves who you are. On Fixedmark, rotating a token never changes your IPs, so partners do not need to update their allowlists.

What happens to open connections when I revoke a proxy token?

It depends on the proxy. A CONNECT proxy usually checks credentials only when a tunnel opens, so tunnels that are already open keep working until they close. On Fixedmark, a revoked credential applies to new connections, and a tunnel opened before the change stays open until it closes.

Can I rotate a proxy token on Vercel without redeploying?

Not if the token lives in an environment variable, because Vercel applies changes only to new deployments. Either read the token at runtime from a secret store with a short cache, or plan the redeploy inside the overlap window. Remember that old deployments keep the old value.

Is it safe to retry a POST after a 407?

Yes, once. A 407 means the proxy refused your credential before it forwarded anything, so the destination never saw the request. Reload the token, retry one time, and let a second failure surface as an error.

How long should the overlap window be?

Longer than the slowest path your token takes to reach running code. Count cache TTLs, restarts, deploys, your longest scheduled job, and the oldest build you might roll back to. Then revoke on evidence, after the old credential has shown no use in your logs.

Make it fixed.

Fixedmark is in early access. Join the list to get dedicated static IPs when your region opens.