Sandbox sessions

Sandbox sessions keep your agent's work safe when a client drops, goes idle or stops. A running session costs compute. A paused session costs storage.

A sandbox session is one fresh Graviton instance in Sydney. The instance serves that one session alone. This page sets out what a session does when a client drops, goes idle, is cancelled, is paused, is resumed or is interrupted. For each case, the page also sets out what happens to the session's files, processes, credentials, policy and billing. The page "The sandbox and its gateway" sets out the instance, the gateway and the prices. The page "The sandbox reset" sets out how a session ends. The page "Sandbox session routes" sets out each route.

Modes#

A session has one of two modes. The mode field sets the mode when the session starts:

• agent is the default. The session runs your agent's command as its main process. The session ends when that command ends.
• commands: the session starts with no command of its own. The agent container waits, and the session runs the commands you send. Only a session in this mode can pause.

Session states#

• launching: Hangar is starting the session's instance.
• running: the instance runs and the worker on the instance reports to Hangar. Compute is charged.
• pausing: you asked for a pause. The worker stops the running commands and closes the workspace. Then Hangar stops the instance. Compute is charged until the session is paused.
• paused: the instance is stopped. The workspace stays on the session volume. Storage is charged.
• resuming: you asked for a resume. Hangar is starting the same instance again.
• interrupted: the instance stopped, or the worker stopped reporting, and nobody asked for a pause. A session in mode commands waits up to 30 minutes for the instance to run again. A session in mode agent does not wait. A session in mode agent ends with the outcome outcome_unknown.
• stopping: the sandbox reset is running.
• ended: the sandbox reset is complete.
• failed: the session ended before its instance existed, and the reset is complete.
• reset_incomplete: Hangar has not confirmed one reset step. The session certificate waits until Hangar confirms every step.

The behaviour table#

Each row is one event. The From column gives the state in which the event applies. any means every state before the end. The last column says whether Hangar does the event in this release. Hangar refuses an unsupported event with the error named in the row. Hangar never approximates an unsupported event.

The rows idle_timeout_pause, idle_timeout_end, pause, resume and interruption_recovered apply to a session in mode commands only. A session in mode agent ends when its agent exits. An interruption ends a session in mode agent as interruption_lost. A recovery would start the agent's command again from the beginning.

Files means the workspace at /workspace. The workspace is the only part of a session that Hangar writes to a disk. Everything else is in the memory of the instance. That includes the containers' own files, the companion data, /tmp and the state of containerd. That memory is gone at a pause, at an interruption and at the end.

How each event happens#

Disconnection#

A client that drops loses nothing. The session's commands keep running. Hangar keeps the output of each command by byte offset. The client reattaches to the session by session id with GET /v1/sandboxes/{id}/attach. That route returns the state of the session. For each command, the route also returns the offset up to which Hangar has stored the output. The client then reads each command's output from the last offset the client acknowledged. Hangar replays the gap. The command hangar sandbox attach <id> does these steps for you.

Reattaching to a session that has ended returns the final state of the session and the outcome of each command. Reattaching never starts a session.

Idle timeout#

Idle applies to a session in mode commands. A session in mode agent runs its agent's command for as long as the session exists, so the session is never idle. Hangar refuses a start request for a session in mode agent that sets idle_timeout_seconds or idle_action. Hangar refuses idle_action pause with sandbox_pause_unavailable. Hangar refuses any other value with sandbox_request_invalid.

A session is idle when two things are true. The worker reports no command running. No client has attached in the last 30 seconds. A client counts as attached for 30 seconds after each call to attach. When a session has been idle for the idle timeout, the session pauses or ends, as idle_action says:

• idle_timeout_seconds is from 60 to 86,400 seconds. The default is 900 seconds, which is 15 minutes.
• idle_action is end, the default, or pause. Only a session in mode commands can pause.

Hangar charges idle time at the session's vCPU rate, as Hangar charges running time, until the session pauses or ends. The record of each idle minute has an idle flag.

Hard stop#

The field stop_after_seconds on POST /v1/sandboxes sets the most compute time the session runs. The value is from 3,600 seconds (one hour) to 604,800 seconds (seven days). The limit counts the metered compute seconds of the session across all attempts, so a pause does not use the limit. At the limit, the instance stops itself. The session ends with the stop reason stop_after. Hangar bills no compute second after the limit. The command hangar run --cpu --stop-after and a CPU job booked on the website both set this limit.

The session shows stop_after_seconds and compute_at_most_cents. The value compute_at_most_cents is the most the compute takes from your credit. It counts every minute up to the stop. The true-up then returns what the last minute did not use. Hangar charges the storage of a paused session per started hour, as the Pause section sets out. The session also shows memory_gib and minimum_charge_cents. The value memory_gib is the memory that the shape of the session gives. The value minimum_charge_cents is the A$5.00 minimum. Hangar charges the minimum once, when the instance starts running, and charges nothing within the free hour. Hangar charges use above the minimum at the metered rate.

When the session is over, exit_code is the agent's exit code in mode agent. The field certificate_signed shows when Hangar signed the session certificate.

Cancellation#

POST /v1/sandboxes/{id}/stop cancels a session. POST /v1/sandboxes/{id}/revoke-all cancels a session as Revoke All. Both routes work in every state before the end, including a paused session. A session token with the control scope does the same through POST /v1/sandbox-session/stop and POST /v1/sandbox-session/revoke-all.

Hangar does these steps in the transaction that stops the session. Hangar revokes every session token of the session. Hangar withdraws every pending approval. Hangar cancels every queued command. Nobody can approve a withdrawn approval, and the gateway sends nothing for a withdrawn approval. The sandbox reset then runs. A session ends in the same way in these cases:

• the credit of the session runs out;
• the idle timeout ends the session;
• the session has been paused for 7 days;
• the credit cannot pay for the paused storage;
• the instance is lost.

Pause#

POST /v1/sandboxes/{id}/pause pauses a running session in mode commands. POST /v1/sandbox-session/pause with a control token does the same. A session in mode agent returns sandbox_pause_unavailable, because the agent is the main process and processes do not survive a pause. A session that is not running returns sandbox_not_running.

The worker stops each running command. Each of those commands ends with the status killed_at_pause. The worker closes the encrypted workspace and reports that the workspace is ready. A command that is still queued stays queued. A queued command runs after the session resumes. Hangar then stops the instance and records the AWS StopInstances response. If the worker has not reported within 90 seconds, Hangar stops the instance anyway. The session's credential for Hangar ends. The gateway destroys the session's certificate authority and drops the session's attachments. Hangar withdraws every pending approval.

Resume#

POST /v1/sandboxes/{id}/resume resumes a paused session. POST /v1/sandbox-session/resume with a control token does the same. Hangar checks three things. Your credit must cover ten minutes. The gateway must be reporting. Your account policy must allow the resume as the policy reads now. Hangar then starts the same instance, so the instance id does not change. Each resume is a new attempt.

Hangar issues these items fresh at resume:

• a new credential for Hangar, after the instance proves again what the instance is;
• a new certificate authority at the gateway, so no first use that you approved before the pause carries over;
• the allowlist that the session asked for, less any destination that your account's egress list no longer has;
• each attachment whose secret still exists and whose destination is still listed, with the value read from the store at resume.

A token that you revoked never comes back. A session that ended never comes back. Files in the workspace come back from the session volume. Processes do not come back. The containers start fresh, and the worker never runs a command again. If the instance does not start, the session returns to paused. The attempt ends with the reason start_failed.

Interruption#

A session is interrupted in two cases. In the first case, the worker has not reported for 60 seconds and the instance is not running. In the second case, the instance runs but the worker is lost. Hangar asks AWS for the state of the instance:

• Stopped by a spot reclamation: the session is interrupted. If the instance runs again within 30 minutes, the session recovers as a new attempt.
• Running with its worker lost: Hangar stops the instance and starts the instance again. The session recovers as a new attempt.
• Terminated or gone: the session ends with the outcome outcome_unknown. The sandbox reset runs.

Only a session in mode commands recovers. A session in mode agent ends with the outcome outcome_unknown in each of the three cases. The agent's command is the work of that session. A recovery would start the command again from the beginning on the kept workspace. A reclamation that comes while a session is pausing is an interruption, not a pause.

A recovery is a resume in every other way. The credential, the certificate authority, the attachments and the policy are all fresh. A session that does not recover within 30 minutes ends with the outcome outcome_unknown.

Each command that was running at the interruption ends as outcome_unknown and never runs again, because Hangar cannot know what the command did. Each command that was still queued runs after recovery. Hangar charges compute up to the last heartbeat before the interruption. The interruption is a record on the session certificate.

Live-state snapshot#

POST /v1/sandboxes/{id}/snapshots with {"kind": "live_state"} returns 501 sandbox_live_snapshot_unsupported. A session starts from a template. The Templates section sets out templates.

Commands#

A command is one program with its arguments. Send a command with POST /v1/sandboxes/{id}/commands or with hangar sandbox exec <id> -- <program> <arguments>. The command runs in the session's agent container. In that container, the workspace is at /workspace. Each command has its own id and its own number in the session, starting from 1. Up to 16 commands run at once. A session takes up to 247 commands, with at most 64 not yet finished. Each command is a job on the session certificate. One certificate has room for 247 jobs beside the agent and the companions.

The output of each command is one stream of bytes. Each byte has an offset, starting from 0. Each chunk is marked stdout or stderr. A client reads the output with GET .../commands/{cid}/output?after=<offset>. That route waits up to 25 seconds for output past the offset.

A command has one of these statuses:

• queued: you sent the command, and the worker has not started the command.
• running: the worker started the command.
• cancelling: you asked for a cancel while the command ran, and the worker is ending the command.
• exited: the command ended by itself, with its exit code when the process gave an exit code.
• cancelled: the command was cancelled, before or while the command ran.
• killed_at_pause: the session paused while the command ran.
• timed_out: the command ran past its timeout_seconds, and the worker ended the command. The value of timeout_seconds is from 1 to 86,400 seconds, with a default of 3,600.
• outcome_unknown: the command was running when the session was interrupted.

The last five statuses are final. A command never leaves a final status. A cancelled command never runs. Send a cancelled command again with the same Idempotency-Key and you get sandbox_command_cancelled.

Each command is a job on the session certificate, with the job id <session id>/cmd/<number>:

Attempts and retention#

Every boot of the session's instance is an attempt, numbered from 1. The cause of an attempt is launch, resume or recovery. An attempt ends with one of these reasons: paused, interrupted, host_lost, worker_lost, start_failed or session_ended. GET /v1/sandboxes/{id} lists each attempt with its cause, its end and its compute time.

Every command output and every container log belongs to the attempt that made the output or log. Hangar seals each output and log under a key that Hangar makes for that attempt alone. Hangar keeps each output for the session's output_retention_days, counted from the end of the session. The value is from 1 to 90 days, with a default of 30. Then Hangar destroys the output key of the attempt and records the erasure. Nobody can open the stored output after that. A read after that returns 410 sandbox_output_erased.

The workspace is on a session volume of its own. Hangar attaches the volume to the session's instance at launch. The volume is encrypted under a 64-byte key that Hangar makes for the session. Hangar seals the key under your organisation's key. Hangar hands the key only to the instance of the current attempt, after that instance proves what it is. AWS KMS never has the key. Hangar deletes the volume with the instance. Hangar destroys the key when the session ends.

Only /workspace survives a pause or an interruption. The instance's own disk has the measured image, read-only, and nothing of the session. The session's containers, the companion data, /tmp and the state of containerd are in memory. They start empty at each attempt.

Images#

The agent runs an image by digest, and so does each companion. The image comes from your organisation's registry, from a template, or from the agent runtime that Hangar provides. The session's instance is Graviton and runs linux/arm64. So each image from your registry is a linux/arm64 image, or a multi-platform image with a linux/arm64 entry that the session runs. The page "How images are staged" sets out each step.

When Hangar opens the session, each image from your registry must pass two checks. The image must pass its check for linux/arm64. The image must fit the CPU pool's allowance of 8 GiB compressed. Hangar judges a multi-platform image on its linux/arm64 image. A layer counts at the larger of two sizes: the size its manifest declares and the size Hangar stores. Hangar refuses an image that fails a check before an instance starts. The refusal uses the check's own code:

• preflight_architecture for an image that has no linux/arm64 image;
• preflight_too_large for an image larger than 8 GiB;
• image_preflight_pending while the check runs.

The field detail.preflight in the reply gives the reason and the next step for the CPU pool.

At each attempt, the worker on the host stages each image from your registry before any container exists:

• The worker fetches each stored object of the image through its own presigned URL. Each URL is valid for 15 minutes and for that one object. The worker decrypts each object on the instance and checks each object.
• The worker renews the URLs of an image through the control plane when less than 3 minutes of the URLs is left. The worker also renews once after the store refuses a URL. So staging runs for as long as the bytes take. A renewal names only objects that the session manifest names. The worker checks every object against its digests after each fetch.
• An image pushed to the registry arrives as blobs. The worker checks the blobs against their digests and writes the blobs into one OCI image layout.
• The worker reads the config of a pushed image and the layer sizes before any layer. The worker refuses an image built for another platform with preflight_architecture. The worker refuses an image larger than 8 GiB with preflight_too_large.
• The worker imports each image under the digest that the session names.

A refused image ends the session before the containers start. The failed step shows the code.

Templates#

A template is an image that a session starts from. A template is named name@version, for example --template python-dev@1.2. Each template:

• is built by the image pipeline of Hangar, under a declared build network policy, as part of a promoted sandbox release;
• is recorded with its image digest and the digest of its software bill of materials;
• has no credential and no customer data.

Hangar registers a template only when the signed manifest of the template's release lists three items. The manifest lists the image by digest. It lists the software bill of materials, with the digest recorded. It lists the build network policy, with the destinations recorded.

A template is shared with every organisation, or is your organisation's own. GET /v1/sandbox-templates lists the templates that your sessions can start from. A session sets template in place of image.

Tenants and session tokens#

A vendor serves its own customers as tenants. The field tenant on POST /v1/sandboxes names the tenant that a session serves. A tenant name is a lower-case letter or digit, then up to 62 lower-case letters, digits, dots, hyphens or underscores. The tenant name is in the start record of the session, so the tenant name is on the session certificate. A session for a tenant attaches the secrets of that tenant by default. That session never finds the secrets of another tenant.

A session token lets your service, or your customer's client, act on one session without your organisation's sign-in. A member or an owner makes a token with POST /v1/sandboxes/{id}/tokens or hangar sandbox token create. Each token:

• starts hgt_, is shown once, and Hangar stores only its SHA-384 digest;
• works on the one session that the token was made for, and has the tenant of that session;
• is bound to a P-256 public key, and every request with the token has a Hangar-Proof header signed by the matching private key, as the page "Proof of possession: the Hangar-Proof header" sets out;
• lasts from 60 seconds to 24 hours, and never past the maximum length of the session;
• has one or more of these scopes.

A token ends when the token expires, when a member revokes the token, or when its session ends. Hangar's database stores the end of the token. The database refuses any change that would bring the token back. So a revoked token stays revoked after a retry, a restart or a resume. A session has at most 32 active tokens.

When a token ends, the authority of that token ends and nothing else. Hangar refuses the token at once. Hangar cancels each command that the token sent and that has not started. Those commands never run, also after a pause and a resume. A command that the token started runs to its end, as the session's work. The session's gateway credentials and pending approvals belong to the session, not to a token. They end when the session pauses, is interrupted or ends. They also end at Revoke All.

After the session ends, one kind of token can still do three things. The token must have the scope read, must not have expired and must not have been revoked by a member. That token can read the final state of the session, attach, and read each command's output. So a client that dropped just before the end learns how the session ended. That token can do nothing else.

Requests this release refuses#

Hangar refuses each of these requests with the error named. Hangar never approximates a refused request:

• a live-state snapshot: sandbox_live_snapshot_unsupported;
• a pause of a session in mode agent: sandbox_pause_unavailable;
• an idle timeout or idle action for a session in mode agent: sandbox_pause_unavailable for idle_action pause, otherwise sandbox_request_invalid;
• a request that gives a command standard input, a terminal or an interactive shell: bad_request, because the command request has no such field.

The operators of TALOS Defence have no route that sends a command to a session.