Resilient deployments

This guide configures a stopped cloud standby machine that starts automatically if your primary deployment becomes unreachable, using AWS, Azure, or Google Cloud. See /docs if you're new to Nethera.

Why cold cloud failover

Self-hosting normally leaves you with two unappealing options when your primary machine goes down: accept the downtime, or run a second machine at full price around the clock just in case. Neither is great - on-prem hardware doesn't come with a cloud provider's uptime guarantees, but paying for standby compute you almost never use eats into the whole reason you're self-hosting.

A cold standby splits the difference. It sits stopped, so you pay for storage rather than compute, and only starts when your primary is actually unreachable. Because it's paired with the same Nethera workspace as your primary, there's no separate deploy pipeline to maintain for the failover path - it just reconciles to your latest revision when it wakes up.

How it works

1. Nethera attempts to route a request to a healthy backend.
2. If at least one backend is reachable, traffic is served normally.
3. If no backend is reachable, Nethera invokes your service failure webhook.
4. Your cloud provider starts the standby machine.
5. The Nethera agent on the standby reconnects and reconciles to the latest deployment revision.
6. Nethera begins routing traffic as soon as the service port responds.

The request that triggered failover receives:

503 Service Unavailable
Retry-After: 30

Clients should retry until the standby is ready. Startup time depends on your cloud provider, machine size, and how long your container takes to become healthy.

Before you start

You'll need:

• A Nethera app already deploying to your primary machine.
• A standby machine paired with Nethera and added to targets.
• A service failure webhook configured on the service.

appName: inference-api
appId: <generated by neth init>
 
targets:
  - office-gpu
  - cloud-gpu
 
services:
  api:
    image: ghcr.io/acme/inference-api:latest
    nethera:
      public: 8000
      auth: token
      onServiceFailure:
        after: 60s
        emails:
          - owners
        webhookSecrets:
          - FAILOVER_WEBHOOK

Set the webhook secret before your first deploy:

$neth secrets set FAILOVER_WEBHOOK

Deploy once while the cloud machine is running, to confirm it works, then stop or scale it down. When it starts again later, either from real failover or manually, the agent reconciles to the latest deployment revision automatically rather than replaying old jobs.

Set up the standby machine

import {
  EC2Client,
  StartInstancesCommand,
} from "@aws-sdk/client-ec2";

const client = new EC2Client({});

export const handler = async () => {
  await client.send(
    new StartInstancesCommand({
      InstanceIds: [process.env.INSTANCE_ID!],
    }),
  );

  return {
    statusCode: 202,
    body: JSON.stringify({ starting: true }),
  };
};

INSTANCE_ID=i-0123456789abcdef0

$neth secrets set FAILOVER_WEBHOOK

subscription
resource group
VM name

await computeClient.virtualMachines.beginStartAndWait(
  resourceGroupName,
  vmName,
);

$neth secrets set FAILOVER_WEBHOOK

await instancesClient.start({
  project,
  zone,
  instance: instanceName,
});

$neth secrets set FAILOVER_WEBHOOK

Failing over stateful services

Cold failover works most cleanly for stateless services. Both machines run the same image, and either instance can handle a request without depending on files or data stored only on its local disk.

For a stateful production service, the primary and standby need access to the same data, or you need a separate replication and recovery plan. Nethera starts replacement compute and routes traffic to it; it does not replicate application data.

Common approaches include:

• Use a shared managed database. Run Postgres, MySQL, or another database somewhere both the on-prem and cloud instances can reach. The application instances can then fail over without creating separate copies of the database.
• Store files in object storage. Put uploads, generated files, and other durable objects in S3, Azure Blob Storage, Google Cloud Storage, or another S3-compatible service instead of a machine-local Docker volume.
• Use database-native replication. For stricter availability requirements, configure replication and promotion using the database's own tooling or a managed high-availability database service.
• Restore from backups or snapshots. For workloads that can tolerate some data loss, define a recovery point objective and restore the latest known-good snapshot before allowing the standby to serve traffic.

Persistent disks on the standby are still useful for Docker images, model weights, package caches, and other data that can be prepared ahead of time. They should not be treated as a copy of data written to the primary machine.

Before enabling automatic failover for a stateful service, decide:

1. Where the authoritative data lives.
2. How the standby accesses or restores it.
3. How much data loss is acceptable.
4. Whether both instances may write concurrently.
5. When the standby is safe to place into service.

FAQ

Does failover replicate my data automatically?

No. This guide fails over compute only. If your service writes local data, see Failing over stateful services above before relying on this for anything other than stateless workloads.

What does the standby cost while it's idle?

Typically only storage and other reserved resources. You're not billed for compute until the webhook actually triggers a start.

How fast does failover happen?

Depends on machine size and how long your container takes to report healthy. A warm pool avoids full cold-boot time (image launch, driver init), but there's still a real startup window, this is cold failover, not instant high availability.

Does the standby need its own deployment step?

No. It's a normal Nethera machine in targets. When it starts, whether manually or via failover, the agent reconciles to your latest deployed revision automatically.

Notes

• This is cold failover, not instant high availability, expect a real gap while the standby starts.
• Large models or datasets should live on persistent disk on the standby, not be re-downloaded on every start.
• Standby machines stay paired with Nethera while stopped, and converge to the latest deployment on reconnect.
• Scaling the standby back down after recovery is a manual step in every provider section above. Automating it fully is reasonable for stateless services; for stateful ones, confirm data has synced back before scaling to zero.