---
title: "Rotating Proxy Credentials Without Downtime"
description: "Rotate proxy tokens and passwords without breaking outbound calls. Overlap windows, two-token rotation, secret store caches, and a safe retry on a 407."
canonical: https://fixedmark.com/blog/rotating-proxy-credentials-without-downtime
updated: 2026-11-10
source: Fixedmark (https://fixedmark.com), BitMask LLP
---

Networking · 11 min read

# Rotating Proxy Credentials Without Downtime

Never swap a proxy credential in one step. Create the new token while the old one still works, move every client to it, confirm the old token has gone quiet in your logs, then revoke it. Make the overlap longer than the slowest path a secret takes to reach your code, and have clients reload the token and retry once when the proxy answers \`407\`.

## Why a credential swap breaks outbound calls

A proxy credential is usually a username and password, or an app ID and a token, inside a URL such as `https://APP_ID:TOKEN@proxy.example.com:443`. Every call your app sends through the proxy depends on it. If the proxy stops accepting the credential before every copy is replaced, those calls fail with `407 Proxy Authentication Required`. They fail together, because they all share one secret.

A credential has more copies than the one you just changed in the secret store.

- Caches in front of the store. The [AWS Parameters and Secrets Lambda Extension](https://docs.aws.amazon.com/systems-manager/latest/userguide/ps-integration-lambda-extensions.html) keeps a secret for up to 300 seconds by default and says it "doesn't detect changes" before that TTL runs out.
- Environment variables of running processes. A process reads its environment once, when it starts. Kubernetes states that [ConfigMaps consumed as environment variables](https://kubernetes.io/docs/concepts/configuration/configmap/) "are not updated automatically and require a pod restart."
- Build artifacts. On Vercel, [changes to environment variables](https://vercel.com/docs/environment-variables/managing-environment-variables) "only apply to new deployments." An old deployment URL keeps the old value.
- Open connections. An HTTP CONNECT tunnel is authenticated once, when it opens. A pool of kept-alive tunnels can keep running on a credential that no config file holds any more.

A one-step swap starts a race between the provider and all of those copies. AWS describes the same race for database passwords in its [single-user rotation strategy](https://docs.aws.amazon.com/secretsmanager/latest/userguide/rotation-strategy.html). There is "a short period of time" between the password change and the secret update, and calls can be denied during it. For a proxy that carries every partner call your app makes, a short period still means a burst of failed payouts, orders, or database connections.

## Keep two valid credentials during the switch

The fix is an overlap. For a window of time the proxy accepts both the old and the new credential. Clients move across at their own pace, and none of them ever holds a dead credential.

The order of the steps is what makes this work. Vercel's [guide to rotating secrets](https://vercel.com/docs/environment-variables/rotating-secrets) says to update your app "before you invalidate the old credential." Create, distribute, verify, revoke. Never revoke first.

_Three ways a provider can handle rotation_

| Strategy                    | How it works                                                                                                   | Example                                                                                                                              |
| --------------------------- | -------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ |
| Rotate with a grace period  | The provider issues a new secret and keeps the old one valid until a deadline.                                 | Stripe keeps both keys working [for up to 7 days](https://docs.stripe.com/keys#rolling-keys) when you rotate a key in the Dashboard. |
| Two independent credentials | You create a second credential, move clients to it, then revoke the first. Next time you rotate the other one. | AWS Secrets Manager's alternating-users strategy, after which "both `user` and `user_clone` credentials are valid."                  |
| Hard cut                    | The old secret stops working the moment the new one exists.                                                    | Choosing Now as the expiry when you rotate a Stripe key, which deletes the old key.                                                  |

Two independent credentials give you the most control. The overlap lasts as long as you need, not as long as the provider's grace period, and you revoke on evidence instead of on a timer. Stripe gives the same advice when its 7 days are not enough. Create a new key by hand, migrate to it, then expire the old one.

A grace-period rotate is fine when you can finish well inside the window. Keep the hard cut for a leaked credential, where a short outage is the smaller harm.

## Size the overlap to the slowest copy

The overlap has to outlast the slowest path from your secret store to the code that makes the call. Add up the delays your setup really has.

_When each kind of copy picks up a new value_

| Where the token lives               | When it changes                                                                                                         | Source                                                                                                                    |
| ----------------------------------- | ----------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- |
| Mounted Kubernetes Secret volume    | Eventually. The delay can be the kubelet sync period plus the cache propagation delay. A `subPath` mount never updates. | [Kubernetes Secrets](https://kubernetes.io/docs/concepts/configuration/secret/)                                           |
| Environment variable in a container | At the next container start.                                                                                            | [Kubernetes ConfigMaps](https://kubernetes.io/docs/concepts/configuration/configmap/)                                     |
| Lambda extension cache              | When `SECRETS_MANAGER_TTL` expires. 300 seconds is both the default and the maximum.                                    | [AWS Systems Manager](https://docs.aws.amazon.com/systems-manager/latest/userguide/ps-integration-lambda-extensions.html) |
| Vercel environment variable         | At the next deployment. Old deployments keep the old value.                                                             | [Vercel](https://vercel.com/docs/environment-variables/rotating-secrets)                                                  |
| Open tunnel or pooled connection    | When the connection closes and the client opens a new one.                                                              | Your client's keep-alive and pool settings                                                                                |

Then add the delays that are not timers. A deployment that fails halfway leaves some instances on the old value. A rollback to last week's build brings back last week's token, which is why Vercel warns that an old deployment "will still use the old credential." A job that runs once a month only notices the change on its next run, and that run is often how teams learn they revoked too early.

So set the window longer than your longest job interval and longer than the oldest build you would roll back to. Then make the window matter less, by checking usage before you revoke. The runbook below covers that check.

## Make clients reload the token without a deploy

Overlap protects you from slow copies. The other half of the job is making your copies fast. Code that reads the proxy URL once at startup needs a restart or a redeploy for every rotation. Code that reads it from a mounted file or a secret store, with a short cache, picks up a rotation on its own.

Two rules keep this safe. First, build a new proxy agent when the value changes, instead of editing headers on a live one, and let requests in flight finish on the old agent. Second, when the proxy answers `407`, reload the secret and retry once. [RFC 9110](https://www.rfc-editor.org/rfc/rfc9110.html#name-407-proxy-authentication-re) allows exactly this. After a 407, "the client MAY repeat the request with a new or replaced Proxy-Authorization header field."

The retry is safe even for a `POST`. A proxy refuses the `CONNECT` before any byte of your request leaves, so the partner never saw the first attempt. Retry only once, so a token that is really wrong fails fast instead of looping.

**Node.js (undici)**

```typescript
import { readFile } from "node:fs/promises";
import { fetch, ProxyAgent, type RequestInit } from "undici";

// Your secret store keeps this file current, for example a mounted
// Kubernetes Secret or a sidecar that writes the latest version.
const PROXY_URL_FILE =
	process.env.PROXY_URL_FILE ?? "/var/run/secrets/proxy/url";
const MAX_AGE_MS = 60_000;

let current: { url: string; agent: ProxyAgent; loadedAt: number } | undefined;

async function proxyAgent(force = false): Promise<ProxyAgent> {
	if (current && !force && Date.now() - current.loadedAt < MAX_AGE_MS) {
		return current.agent;
	}
	const url = (await readFile(PROXY_URL_FILE, "utf8")).trim();
	if (current?.url === url) {
		current.loadedAt = Date.now();
		return current.agent;
	}
	const previous = current?.agent;
	current = { url, agent: new ProxyAgent(url), loadedAt: Date.now() };
	// close() lets requests already on the old agent finish first.
	previous?.close().catch(() => {});
	return current.agent;
}

// undici reports a refused CONNECT as an error whose message carries
// the proxy's status code, wrapped in fetch's TypeError.
function isProxyAuthError(err: unknown): boolean {
	for (let e: unknown = err; e instanceof Error; e = e.cause) {
		if (e.message.includes("(407)")) return true;
	}
	return false;
}

export async function proxiedFetch(url: string, init: RequestInit = {}) {
	try {
		return await fetch(url, { ...init, dispatcher: await proxyAgent() });
	} catch (err) {
		if (!isProxyAuthError(err)) throw err;
		// The proxy refused the token before the request left, so a
		// retry is safe even for a POST. Reload the secret and try once.
		return fetch(url, { ...init, dispatcher: await proxyAgent(true) });
	}
}
```

**Python (httpx)**

```python
import os
import pathlib
import threading
import time

import httpx

# Your secret store keeps this file current.
PROXY_URL_FILE = pathlib.Path(
    os.environ.get("PROXY_URL_FILE", "/var/run/secrets/proxy/url")
)
MAX_AGE = 60.0

class ProxiedClient:
    def __init__(self) -> None:
        self._lock = threading.Lock()
        self._url: str | None = None
        self._client: httpx.Client | None = None
        self._retired: httpx.Client | None = None
        self._loaded = 0.0

    def _get(self, force: bool = False) -> httpx.Client:
        with self._lock:
            fresh = time.monotonic() - self._loaded < MAX_AGE
            if self._client and fresh and not force:
                return self._client
            url = PROXY_URL_FILE.read_text().strip()
            self._loaded = time.monotonic()
            if url != self._url or self._client is None:
                # Close the client from two rotations ago. The one just
                # replaced may still have requests in flight.
                if self._retired:
                    self._retired.close()
                self._retired = self._client
                self._url = url
                self._client = httpx.Client(proxy=url, timeout=10.0)
            return self._client

    def request(self, method: str, url: str, **kwargs) -> httpx.Response:
        try:
            resp = self._get().request(method, url, **kwargs)
        except httpx.ProxyError:
            # The proxy refused the CONNECT, so nothing reached the
            # destination. Reload the secret and retry once.
            return self._get(force=True).request(method, url, **kwargs)
        if resp.status_code == 407:
            # Plain http:// destinations get the 407 as a response.
            return self._get(force=True).request(method, url, **kwargs)
        return resp
```

We ran both versions against a small CONNECT proxy that started refusing the old token partway through, with undici 7.30 and 8.12 and httpx 0.28. undici reports a refused tunnel as `Proxy response (407) !== 200 when HTTP Tunneling`, two levels down the `cause` chain of fetch's `TypeError: fetch failed`. It tunnels plain `http://` destinations too, so one check covers both. httpx raises `httpx.ProxyError` for a refused tunnel and returns an ordinary 407 response for a plain `http://` destination.

A cache of a minute means a rotation reaches every warm process about a minute after the file changes, without a call to the secret store on every request. If you route only some calls through the proxy, keep this helper on those calls alone, as [routing only some fetches through a proxy](https://fixedmark.com/blog/nextjs-route-only-some-fetches-through-a-proxy) describes.

## Proxy details that catch people during rotation

- Open tunnels outlive revocation at many proxies. A CONNECT proxy checks `Proxy-Authorization` when the tunnel opens, then, in RFC 9110's words, restricts itself "to blind forwarding of data." In our test, a kept-alive tunnel opened with the old token went on serving requests after the proxy stopped accepting that token. That helps availability. It also means revoking a leaked token does not close a tunnel an attacker already holds, unless your proxy closes it. Ask your provider which behaviour it has. [How a CONNECT proxy works](https://fixedmark.com/blog/http-connect-proxy-explained) shows why.
- New tokens need URL-encoding. A generated token can contain `@`, `:`, `/`, or `%`. Inside a proxy URL those must be percent-encoded, or the client splits the URL in the wrong place and sends a broken credential.
- Copy and paste leaves marks. A trailing newline makes a valid token fail, and so do quotes left around the value. Trim whitespace when you read the value, as both examples above do, and fix stray quotes where the secret is stored.
- Cleartext proxy ports expose the token. On a plain `http://` proxy URL the credential crosses the network as base64 in a `Proxy-Authorization: Basic` header, which anyone on the path can decode. Rotate such a token more often, or use an [HTTPS proxy endpoint](https://fixedmark.com/blog/https-proxy-endpoints-tls-to-the-proxy), which encrypts it.
- One shared token means one big rotation. If staging, production, and the nightly job share a token, rotating it touches all three at once. Give each its own token, so a rotation or a leak affects one service.
- Database pools through SOCKS5 hold connections for a long time. A pool that never recycles a connection keeps using the old credential until it reconnects. Set a maximum connection lifetime. [Pooling through SOCKS5](https://fixedmark.com/blog/database-connections-through-socks5-pooling) covers the pool settings.

When something does fail, find out who refused you. A `407` on the CONNECT comes from the proxy. A `401` after the tunnel opened comes from the destination, which means the partner's own API key is the problem, not the proxy token. [Proxy error codes explained](https://fixedmark.com/blog/proxy-error-codes-explained) walks through telling them apart with `curl -v`.

## A rotation runbook

1. 1

   **List every holder** Write down each service, environment, scheduled job, CI workflow, and laptop that holds the credential. If you cannot list them, the rotation will find them for you, so do this first.
2. 2

   **Create the new credential** Create it while the old one still works. Copy it straight into your secret store, because many providers show a token only once.
3. 3

   **Test it alone** Send one request through the proxy with the new credential before any app uses it. The command below prints the proxy's answer to the CONNECT and the destination's status separately.
4. 4

   **Store it as the current version** Add it as a new version of the same secret, not as a new secret name, so code that reads the current version picks it up. In AWS Secrets Manager, [PutSecretValue](https://docs.aws.amazon.com/secretsmanager/latest/apireference/API_PutSecretValue.html) moves `AWSCURRENT` to the new version and `AWSPREVIOUS` to the old one, so the last good value stays one call away.
5. 5

   **Roll out in stages** Let reloading clients pick it up, then restart or redeploy everything that reads the value at startup. Start with one service and watch for 407s before you do the rest.
6. 6

   **Wait for the old credential to go quiet** Use the proxy's logs to confirm nothing has used the old credential for a while. Stripe's advice for its own keys is to expire the old one "only after its request volume has been at zero for a few hours or days."
7. 7

   **Revoke, then keep watching** Revoke the old credential and watch 407 rates for a day. If a holder you missed breaks, you will know which credential it held and where to look.

**Terminal**

```bash
# Test the new proxy URL before any app depends on it.
curl -sS -o /dev/null \
  -w "proxy %{http_connect}, destination %{http_code}\n" \
  --proxy "$NEW_PROXY_URL" https://api.partner.example/health
# proxy 200, destination 200   new credential works
# proxy 407, destination 000   proxy refused it

# Store it as the current version. Reading it from a file keeps
# the token out of your shell history and process list.
aws secretsmanager put-secret-value \
  --secret-id prod/proxy-url \
  --secret-string file://new-proxy-url.txt
```

Script these steps once you have done them by hand. A script that runs create to revoke, with the quiet-period check in the middle, beats a person working from memory once a year.

## When to rotate, and how often

Rotate at once when a credential is exposed. That includes a token committed to a repository, printed in a log, pasted into a chat, or known to someone who has left the team. Stripe's [guidance on exposed keys](https://docs.stripe.com/keys-best-practices) is to rotate "immediately even if you aren't sure anyone saw it." For a leak, the hard cut is usually right, and the overlap is the risk you are closing.

Scheduled rotation earns its place for a different reason. Stripe says periodic rotation "confirms that you know where each key is used and that your team can replace a key on short notice." A team that rotates on a schedule has a tested runbook on the day of a leak. A team that never rotates builds its inventory during the incident.

Pick an interval you will keep, and automate it.

## Rotating a Fixedmark token

Fixedmark's proxy URL carries your app ID as the username and a token as the password. A few properties from the [concepts docs](https://fixedmark.com/docs/concepts#tokens) shape how rotation works there.

- Rotating a token never changes your IPs. Partners allowlist your [static IP pair](https://fixedmark.com/docs/concepts#ip-pairs), not your token, so a rotation needs no ticket to a bank or broker.
- Each app can hold several tokens. Give each environment or service its own, and rotate one without touching the others.
- Fixedmark stores only a hash of each token, so copy the token when you create it. The CLI prints it once, on create and on rotate.
- A token can carry a [destination allowlist](https://fixedmark.com/docs/allowlists), so a leaked token reaches only the hosts and ports you listed.
- Proxy nodes check tokens locally from a cached copy, so authentication keeps working if the Fixedmark control plane is down.

The [REST API](https://fixedmark.com/docs/api) supports both overlap strategies. `POST /v1/apps/{id}/credentials/{cred}/rotate` replaces a credential's token, and the old one works until its grace ends. The reply holds the new credential with its token, and the previous credential with its `expires_at`. For an overlap you control, create a second credential, move your clients to it, and revoke the old one.

**Terminal**

```bash
# Create a second credential. The reply's token is shown only here.
curl -X POST https://api.fixedmark.com/v1/apps/APP_ID/credentials \
  -H "Authorization: Bearer $FIXEDMARK_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"label": "prod-2026-11"}'

# After rollout and a quiet period, revoke the old one.
curl -X DELETE https://api.fixedmark.com/v1/apps/APP_ID/credentials/OLD_CRED_ID \
  -H "Authorization: Bearer $FIXEDMARK_API_KEY"
```

The [fm CLI](https://fixedmark.com/docs/cli#commands) wraps the same routes as `fm credentials list|create|rotate|revoke --app <app>`, and `fm api <METHOD> <path>` calls any route directly. Listing shows token prefixes only, which is enough to tell tokens apart.

Before you revoke, check the [connection logs](https://fixedmark.com/docs/logs). Each row carries the `credential_id` that opened the connection, never the token, so you can confirm the old credential has stopped opening connections. A revoked credential applies to new connections. A tunnel opened before the revocation stays open until it closes.

`fm tunnel` reads its token from its config file or `FIXEDMARK_SOCKS_URL` and checks it when it starts, so restart the tunnel after you change either one. If a client still gets `407` after a rotation, the [troubleshooting guide](https://fixedmark.com/docs/troubleshooting#407) lists the usual causes. Fixedmark is in early access, with [planned pricing](https://fixedmark.com/pricing) and a [security model](https://fixedmark.com/security) you can read first.

## Sources

Checked November 2026. Confirm current details on the vendor's site before you decide. Prices and limits change.

- [AWS Secrets Manager: Lambda function rotation strategies](https://docs.aws.amazon.com/secretsmanager/latest/userguide/rotation-strategy.html)
- [AWS Secrets Manager API: PutSecretValue](https://docs.aws.amazon.com/secretsmanager/latest/apireference/API_PutSecretValue.html)
- [AWS Systems Manager: Parameter Store in Lambda functions](https://docs.aws.amazon.com/systems-manager/latest/userguide/ps-integration-lambda-extensions.html)
- [Kubernetes: Secrets](https://kubernetes.io/docs/concepts/configuration/secret/)
- [Kubernetes: ConfigMaps](https://kubernetes.io/docs/concepts/configuration/configmap/)
- [Vercel: Managing environment variables](https://vercel.com/docs/environment-variables/managing-environment-variables)
- [Vercel: Rotating environment variables](https://vercel.com/docs/environment-variables/rotating-secrets)
- [Stripe: API keys, rotate an API key](https://docs.stripe.com/keys)
- [Stripe: Best practices for managing secret API keys](https://docs.stripe.com/keys-best-practices)
- [RFC 9110: HTTP Semantics](https://www.rfc-editor.org/rfc/rfc9110.html)

Keep reading

## Related guides

- [Use case Partner API allowlisting Give Salesforce, SOAP vendors, and payment gateways one dedicated IP pair to allowlist, wherever your code runs.](https://fixedmark.com/use-cases/partner-api-allowlisting)
- [Integration Vercel Fixed egress IPs for Vercel Functions without the $100 per project add-on.](https://fixedmark.com/integrations/vercel)
- [Integration AWS Lambda A fixed IP for Lambda without VPC, subnets, and two NAT Gateways.](https://fixedmark.com/integrations/aws-lambda)
- [Guide Security TLS passthrough, encrypted proxy credentials, and per-token allowlists.](https://fixedmark.com/security)
- [Guide Docs Proxy URLs and client guides for Node.js, Python, Go, Java, and more.](https://fixedmark.com/docs)

## Frequently asked questions

### Does rotating a proxy token change my outbound IP?

Not with a static IP proxy. The IP belongs to the proxy or your plan, and the token only proves who you are. On Fixedmark, rotating a token never changes your IPs, so partners do not need to update their allowlists.

### What happens to open connections when I revoke a proxy token?

It depends on the proxy. A CONNECT proxy usually checks credentials only when a tunnel opens, so tunnels that are already open keep working until they close. On Fixedmark, a revoked credential applies to new connections, and a tunnel opened before the change stays open until it closes.

### Can I rotate a proxy token on Vercel without redeploying?

Not if the token lives in an environment variable, because Vercel applies changes only to new deployments. Either read the token at runtime from a secret store with a short cache, or plan the redeploy inside the overlap window. Remember that old deployments keep the old value.

### Is it safe to retry a POST after a 407?

Yes, once. A 407 means the proxy refused your credential before it forwarded anything, so the destination never saw the request. Reload the token, retry one time, and let a second failure surface as an error.

### How long should the overlap window be?

Longer than the slowest path your token takes to reach running code. Count cache TTLs, restarts, deploys, your longest scheduled job, and the oldest build you might roll back to. Then revoke on evidence, after the old credential has shown no use in your logs.
