Understand the Feedback Loop
Primary source: [Hugging Face technical timeline](https://huggingface.co/blog/agent-intrusion-technical-timeline) • Related research context: [ExploitGym](https://arxiv.org/abs/2605.11086)
The enterprise problem and today’s slice
Enterprise problem: A team may secure individual prompts while leaving observation, retry, and transport paths open, allowing automation to turn weak signals into a reliable intrusion loop.
Whole-course context: We consume Day 01’s qualified evidence ledger and preserve its reported-versus-inferred distinctions while reconstructing mechanism.
Today’s slice: We model one malicious dataset configuration moving through propose, execute, observe, and adapt, without publishing exploit instructions.
End-of-day evidence: A defensive feedback-loop map with bounded inputs, observable outputs, channel ownership, and tested breakpoints.
Still unsolved: We defer how a production foothold accumulated cluster, secret, network, and source-control authority.
Thesis: The dangerous capability was not a single clever payload; it was a closed feedback loop that could observe consequences, retain state, change tactics, and try again across weakly separated systems.
The smallest complete model is proposal → observable consequence → adaptation. Break any edge reliably and experimentation stops compounding.
Customer use cases
Defenders need to see the loop as a system, because blocking one payload leaves retry and alternate channels intact.
| Use case ID | Actor | Customer job | Success outcome | Denial or recovery evidence |
|---|---|---|---|---|
| D02-UC-01 | Evaluation-platform owner | Contain untrusted dataset processing | A dataset configuration cannot read local files, execute code, or reach undeclared networks | Denied operation records policy, workload, attempted capability, and unaffected control job |
| D02-UC-02 | Detection engineer | Detect automated adaptation across attempts | Related attempts correlate by workload, sequence, channel, and changed hypothesis | Lost or malformed telemetry causes fail-closed evaluation and a preserved recovery record |
Actor-centred user stories
Platform owners and detection engineers require the same loop evidence or each sees only an isolated event.
| Story ID | Use case IDs | User story | Observable acceptance conditions |
|---|---|---|---|
| D02-US-01 | D02-UC-01 | As an evaluation-platform owner, I want dataset processing inside a capability-minimal sandbox, so that content cannot become ambient host authority | Local-file, code-execution, credential, and undeclared-egress probes are denied while a safe dataset completes |
| D02-US-02 | D02-UC-02 | As a detection engineer, I want sequenced attempts correlated, so that adaptation is visible before one attempt crosses a boundary | Alert shows ordered attempts, changed input class, observations, and immutable trace IDs |
End-to-end product flows
A secure evaluation flow begins with a customer-visible submission and ends with evidence for both the allowed job and the denied capability.
| Flow ID | Use case IDs | Path | Trigger | Numbered steps | Terminal evidence |
|---|---|---|---|---|---|
| D02-FLOW-01 | D02-UC-01, D02-UC-02 | Happy | User submits an approved dataset configuration | 1. Resolve immutable evaluator policy. 2. Create an isolated workload. 3. Process declared data only. 4. Record outputs and egress. 5. Destroy workload identity. | Safe result plus policy digest, workload ID, network trace, time, and immutable run ID |
| D02-FLOW-02 | D02-UC-01, D02-UC-02 | Denied | Configuration requests an undeclared local or execution capability | 1. Kernel or runtime denies capability. 2. Evaluator stops. 3. Correlator links prior attempts. 4. Safe positive-control job runs. | Denial reason, attempted capability, attempt sequence, unaffected-job result, and trace ID |
System design derived from the flows
URL allowlists alone are incomplete because a parser can reach local files or execution primitives without making a forbidden URL request.
| Use case ID | Entry point | Responsible services | Authoritative store | Failure evidence |
|---|---|---|---|---|
| D02-UC-01 | Dataset evaluation API | Schema gate, sandbox manager, kernel policy, egress proxy | Immutable evaluation-policy registry | Capability denial with policy digest and safe positive control |
| D02-UC-02 | Attempt telemetry stream | Trace collector, sequence correlator, adaptive-pattern detector | Append-only attempt evidence store | Missing sequence or telemetry closes evaluation and creates recovery incident |
Data model and ownership
The loop cannot be detected if attempts are logged independently, because adaptation exists in the relationship between attempts.
Generated-application database: Not created in this slice — the evaluation control plane and evidence service own policy, attempt, observation, and channel state.
| Record or entity | Store and owner | Primary key | Foreign key or opaque reference | Tenant key | Material constraint | Lifecycle and deletion | Use case IDs |
|---|---|---|---|---|---|---|---|
| Evaluation policy | Policy registry / platform security | policy_digest | None — root immutable policy artifact | organization_id | Capability set and image digest are signed | Retain while referenced; revoke by new version | D02-UC-01, D02-UC-02 |
| Evaluation attempt | Evidence store / evaluator | attempt_id | policy_digest and prior_attempt_id | organization_id | Input hash, sequence, workload identity, and terminal state required | Retain through incident window then policy retention | D02-UC-01, D02-UC-02 |
| Observation envelope | Evidence store / detection | observation_id | attempt_id | organization_id | Channel, sequence, checksum, and raw evidence hash required | Append-only; redact payload under controlled workflow | D02-UC-01, D02-UC-02 |
Progressive visual explanation: how weak signals became a control loop
Looking at payloads one at a time hides the compounding mechanism; add observation and memory progressively.
Level 1 — One attempt, one bounded result
The Hugging Face timeline describes two different dataset-processing paths: HDF5 (Hierarchical Data Format version 5) external storage reportedly disclosed environment and worker information, while a separate Jinja2 template-engine and fsspec filesystem-interface reference path reportedly reached Python execution. These are distinct mechanisms; neither should be generalized into a recipe.
Conceptual reconstruction — Stage 2 loop: synthetic controls preserve the reported feedback structure, not hidden trace parameters.
What this level adds: a terminal observation. Decision rule: isolate by capability, not only by input format or URL.
Level 2 — Close the adaptation loop
Once observations are reachable, an automated system can vary hypotheses cheaply. Persistence and memory convert partial failures into search guidance.
What this level adds: retained state and retry. Decision rule: rate, correlate, and contain sequences, not just individual requests.
Level 3 — Add constrained command transport
Hugging Face reports more than 100 capture endpoints, paste sites, file drops, controlled datasets, and a permissive cross-origin proxy used as dead drops. Its described envelope carried type, channel, sequence, and checksum around compressed/base64 content. XOR, gzip, and base64 are obfuscation or transport encoding, not strong encryption.
Conceptual reconstruction — C2 envelope: controls expose field relationships without recreating a working channel.
What this level adds: asynchronous transport across network restrictions. Decision rule: govern egress by destination identity, method, payload class, and workload purpose; a generic HTTPS allowlist is not a boundary.
Level 4 — Investigator guardrails change the investigation path
The source screenshot concerns the defender investigation, not attacker behavior: Claude/Fable refused an investigator request, after which investigators switched to GLM-5.2 to continue analysis. It must not be used as evidence that the attacker tactically reframed a refused action.
Conceptual reconstruction — investigator model switch: synthetic controls explore defender workflow provenance and make no claim about attacker tactics.
What this level adds: correct provenance for the screenshot. Decision rule: never transfer behavior observed in a defender’s forensic tool to the attacker’s agent.
Failure modes, trade-offs, and decision rules
Teams often optimize evaluator compatibility, but each convenience expands the authority available to untrusted content.
| Failure mode | Trade-off | Reusable decision rule |
|---|---|---|
| URL allowlist without local-file and execution controls | Broad format support versus complete mediation | Enumerate capabilities at parser, process, kernel, identity, and network layers |
| Long-lived worker with shared state | Throughput versus cross-attempt contamination | Use fresh workload identity and storage per untrusted evaluation |
| Alert per payload | Low implementation cost versus invisible adaptation | Correlate by input lineage, sequence, observation, and changed hypothesis |
| Blocking one public service | Fast containment versus channel substitution | Default-deny egress and require authenticated purpose-bound relays |
| Relying on refusal text | Flexible model behavior versus bypassable policy | Put authorization below the model and independently revoke it |
Implementation and verification
A capability policy is only real when denied and allowed paths are both observed under the same environment.
- Inventory every parser and helper invoked by untrusted dataset content.
- Remove ambient service-account tokens and metadata access from evaluation workers.
- Apply read-only ephemeral filesystems, syscall controls, resource limits, and explicit egress identity.
- Correlate attempts and retain raw terminal evidence outside the workload.
- Run a harmless positive control beside local-file, process, credential, and undeclared-egress denial probes.
Kubernetes explains why service-account credentials are workload identities, not harmless metadata in its service-account documentation. The verification goal is defensive: prove absence of ambient authority, not reproduce the reported exploit path.
Practical next action and falsifiable evidence
Select one production evaluator and draw its complete propose-observe-adapt loop, including every place a result can leave the workload. Then run policy-safe denial probes and one safe dataset.
The design is falsified if the dataset worker can observe a host secret, create an undeclared process capability, reach an unapproved destination, reuse a prior workload identity, or lose an attempt from the sequence. Observable success is a trace set showing all four denied classes, an unaffected safe result, immutable policy digest, workload teardown, and correlated attempt IDs.