How do you test an AI agent sandbox's file, network, and credential boundaries?

An assistant is allowed to run a customer’s uploaded script against one sample file. Its prompt says “stay in the task folder,” but the process also inherits a home-directory mount and a cloud key. A polite model response would not repair that execution boundary. Before allowing the job, the team needs evidence that the process can complete the intended task and is denied access to everything outside it.

The permission ladder decides which action the agent may request. A tool contract defines what a backend must validate when an operation is called. This trial tests a different layer: what the actual runner can reach, read, change, or exhaust if the model or uploaded code tries. It also tests the route from that runner to any credentialed adapter.

Draw the execution boundary before choosing a container flag

Record the untrusted entry point, exact image or runtime revision, execution identity, host or cluster, and the assets that must remain outside. Separate three zones: the model’s proposal and task material; the short-lived runner that processes that material; and a service adapter that holds credentials for approved external operations. Put the adapter outside the runner’s filesystem and process environment. A prompt rule, hidden user-interface field, or environment variable that the process can read does not create a sandbox.

Write one allowed action and one forbidden action for each surface. For example, reading /task/input/example.csv may be necessary while reading a synthetic file in another tenant’s directory must fail. A successful task alone proves availability; a failed probe plus an unaffected protected target proves a boundary. Use synthetic marker files and test accounts, never live customer secrets, for the probes.

Surface	Configuration to inspect	Denial observation to collect
Files	Exact bind mounts or volumes, read-only input, bounded writable output, root filesystem mode, mounted sockets	A controlled read probe reports the denied open to a host-collected trace and emits no synthetic canary value; a separate write probe leaves the host-side checksum unchanged.
Network	Runtime network mode or cluster policy, DNS and egress paths, local sockets	Blocked test endpoint receives no request while a separately permitted route still works.
Commands and resources	Entrypoint, callable tools, image contents, user and capabilities, CPU, memory, process count, and deadline	A controlled over-limit test is stopped; it cannot start a host operation or affect another job.
Credentials	Environment, mounted Secrets, service identity token, metadata access, adapter channel	The runner cannot read a synthetic credential marker or use a credential to call an external system.

Docker bind mounts can be writable unless made read-only, and mounting a host socket can expose an unexpected control channel. A read-only root filesystem does not make a separately mounted volume read-only. Record the effective mounts from the launched container rather than inferring them from a template. Also check for --privileged, added Linux capabilities, and the effective default or custom seccomp profile. Docker documents that privileged mode grants broad host-device and capability access and disables default security restrictions; a capability or seccomp choice still cannot replace file mounts or network policy. Limit CPU, memory, process count, and wall time; test a bounded workload under those limits in a disposable environment. In Kubernetes, a resource request affects scheduling while a limit governs runtime use; verify the limits actually set for this Pod. An agent-facing command allowlist helps restrict requested commands, but uploaded code already executing in the runner can make its own system calls. Its file, network, identity, and resource limits still need enforcement below the prompt and tool list.

Make network denial observable in the actual runtime

For a Docker runner that does not need network access, --network none gives it an isolated network stack with loopback, not a guarantee that every possible data path is closed. Inspect mounts and sockets as well. First confirm from an allowed control environment that the synthetic endpoint is healthy and logs a hit; then request it from the sandbox and verify both the runner error and absence of a new hit. If a task needs one service, use an explicit mediated route and test a second, forbidden destination.

In Kubernetes, a NetworkPolicy object is effective only when the network plugin implements its enforcement. Its selectors, ingress or egress direction, and allowed peers must match the actual Pod. An ingress-only policy does not deny egress. A stored policy or a failed DNS lookup is not by itself evidence that outbound traffic is blocked. Run allow-and-deny probes from the deployed Pod after the policy takes effect, and check the destination’s log. Re-run after plugin, namespace, label, or policy changes. Network controls cannot compensate for a credential or powerful socket already present inside the runner.

Keep credentials with the adapter and test the handoff

The sandbox should receive task data and an operation proposal, not the key that can update the target system. An orchestrator can pass a structured proposal to a separate adapter. That adapter binds the user and tenant from trusted application context, checks the target, fields, approval and version, then performs the narrow call with its own credential. The runner should have no unrestricted direct route to that adapter; otherwise a forged proposal can still become a write. Test a valid proposal and altered target or action, then inspect the authoritative target system for what changed.

Kubernetes normally assigns a default ServiceAccount when none is specified and automatically mounts its token into the Pod. Record the assigned ServiceAccount and set automountServiceAccountToken: false when the runner does not need Kubernetes API access; also inspect any other mounted Secret or injected credential. That setting stops automatic token mounting, but does not remove the Pod’s identity or its RBAC grants. Removing a token from the prompt or masking environment variables in logs does not stop a process from reading credentials it was given. Use synthetic markers to verify that the runner sees none, and keep real secret values out of this browser-saved worksheet.

Accept a trial only with a positive control and a denied control

For each row in the worksheet, save a sanitized case ID, the exact runtime revision, attempted action, expected enforcing layer, observed error, and an external observation. For a read probe, use a team-controlled script that would emit a synthetic canary if reading succeeded; capture its result outside the runner and require a denied-open error or host audit event, with no canary emission. An unchanged checksum alone cannot prove that reading failed. For a write probe, compare the host-side checksum or target state. A network denial needs a destination log with no hit. A write through the adapter needs the target application’s audit record or state query. A resource limit needs termination evidence and a healthy neighboring job. One generic “permission denied” message is not enough if the intended work also fails.

Fail the trial if a forbidden probe succeeds, a positive task is blocked, the policy cannot be shown to apply to this runtime, or evidence of external state is missing. Fix the mount, policy, identity, adapter, or resource configuration and rerun both probes. Keep the results tied to image digest, launch settings, node or plugin, adapter version, and credential route so a later change has a clear retest trigger.

Example one: user-uploaded code runs against a sample file

A fictional data-service team lets a customer upload Python code to transform an invented CSV. The authorized run reads a read-only /task/input/sample.csv and writes /task/output/result.csv within a size and time limit. It does not need external network access or cloud credentials. The team launches a disposable runner with only those task paths mounted, a separate writable output area, resource caps, and no credential injection. Its Docker trial uses --network none; the team still inspects for sockets and unexpected mounts.

The team-controlled negative script tries to read a synthetic other-tenant marker, alter the input CSV, contact a test endpoint, and exceed a small process or time budget. If it reads the marker, it emits a synthetic success signal to a host-captured trace. Reviewers require an observed denied read with no success signal, an unchanged input checksum for the separate write attempt, no endpoint hit, and a recorded stop at the configured limit. The normal transform must still produce a valid result. If the marker is readable or the endpoint records a hit, the team blocks uploads and repairs the runtime. A prompt telling the model to ignore the extra path would not change this result.

Example two: an agent prepares a museum collection draft

A fictional museum assistant uses an approved snapshot of collection metadata to prepare an exhibit-label draft. The runner can read the snapshot and produce a proposal, but holds no collection-management credential and cannot call that service directly. A separate adapter accepts only an eligible collection ID, draft fields, the curator’s approved scope, and a matching record version. The adapter’s identity may create drafts; it cannot publish them through this route.

The team tests one valid synthetic draft and confirms a new unpublished draft in the collection system. It then changes the ID to another collection, changes the requested action to publish, and tries a direct request from the runner. The adapter must reject both altered proposals; the network path and token search must deny the direct route. The collection system’s audit record must show no unauthorized write. If the runner can read the adapter’s token or reach a write endpoint that bypasses the adapter, the draft feature stays disabled even if its ordinary demo looked correct.

Boundary

This sheet tests the execution environment and credential path for one named workload and runtime revision. It does not establish that a container alone withstands every hostile-code or host-kernel attack; choose stronger isolation when the threat model requires it. Action approval, tool semantics, third-party server admission, and downstream data rights still need their own reviews. Re-run this trial when mounts, image, runtime, network plugin, Pod labels, service identity, adapter, or task scope changes.