Real World Atlas
Research note // 003
Category
Failure recovery /
Physical AI /
Autonomous systems /
Real-world execution
Status
Frontier research
Research origin
Milan, Italy
45.4642° N, 9.1900° E
Plan // valid
Execution // started
Expected state ≠ observed state
Realitydoesn't throwexceptions.
What happens
when the physical world
refuses the plan?
Software can retry. Physical systems collide with a harder problem.
- Objects move.
- Doors stay closed.
- Humans change their minds.
- Providers fail.
- Roads close.
- Prices change.
- Machines lose access.
- Sensors misunderstand.
Reality does not guarantee that the world still matches the plan. Nothing raises a structured error. The gap simply exists, and something has to notice it.
AI can plan.
Machines can act.
Reality gets the final vote.
The pipeline is a lie
The conventional model
Intent
↓ Plan
↓ Action
↓ Outcome
Insufficient // assumes the world holds still
Real-world execution
Intent
Plan
Act
Observe
Compare
Replan
Act again
Verify
Real-world execution
is not a pipeline.
It is a loop.
Digital workflows are often modelled as deterministic sequences: each step completes, the next begins. Physical environments are not deterministic. Between the moment a plan is formed and the moment it is executed, the world has already moved.
So the system must continuously compare what it expected with what actually happened. Only then can execution meaningfully continue.
// System observation 031
Reality is a moving target.
When the world changes
Intent // "Take me to the airport."
The system plans a pickup, assigns a vehicle, computes a route, commits to an arrival time. Every element of that plan is a claim about a world that has not happened yet.
The plan
Reality intervenes
- 01
Driver cancelled
- 02
Road closed
- 03
Vehicle unavailable
- 04
Passenger not at pickup
- 05
Flight time changed
- 06
Payment failed
Execution status
- 01Plan // valid
- 02Execution // degraded
- 03Reality change detected
- 04Recovery required
The intent
did not change.
The world did.
The plan was valid. The door was not open.
Atoms
remember.
Physical failure does not roll back. It leaves state behind.
Software failure vs physical failure
Digital failure
API timeout
→ Retry
Bad response
→ Regenerate
Process failed
→ Restart
Database state
→ Roll back
Physical failure
Glass dropped
→ Broken
Vehicle departed
→ Cannot rewind
Food prepared
→ Resources consumed
Door opened
→ Access already happened
Human left waiting
→ Time lost
Package delivered incorrectly
→ Physical recovery required
Bits can often
be restored.
Atoms remember what happened.
This is not a philosophical claim. It is an operational one: a physical action consumes time, energy, materials, access and human patience. Some of those can be repeated. None of them can be un-spent.
// Reality 036
Physical action has memory.
Failure recovery is intelligence
The real test
is not the plan.
It is the recovery.
When execution diverges, a physical system has to do a sequence of things that has nothing to do with the original task: detect the failure, understand what changed, estimate the remaining options, decide whether to retry, replan, switch executor, request human intervention or change the constraints — and then verify the new outcome.
Recovery architecture // conceptual
Failure detected
Expected state ≠ observed state
State reassessment
What is actually true now
Can the original plan continue?
Yes
→ Retry / adapt
No
→ Replan
Can the machine recover autonomously?
Yes
→ Execute new path
No
→ Escalate
Escalation target
Human / provider / alternative machine
New execution path
Same intended outcome
Verify
Confirm the outcome, not the attempt
// Thesis 039
Recovery is not an edge case.
In the real world,
it is part of execution.
The exception is the product
The demo
shows the happy path.
Deployment sells the exceptions.
A successful prototype proves that a machine can complete a task. That is a real and hard achievement. But an infrastructure system has to answer a second question that a demo never asks: what happens when it cannot?
- 01
Robot cannot grasp object
- 02
Autonomous vehicle cannot access location
- 03
Delivery robot finds closed door
- 04
Service provider does not answer
- 05
Payment authorization expires
- 06
Robot loses connectivity
- 07
Environment differs from world model
- 08
Human blocks the path
The wrong question
Did the executor fail?
The infrastructure question
Did the outcome fail?
Executor substitution
The executor can fail.
The intent doesn't have to.
Human intent
Primary executor // Robot
Failure01
Alternative robot
02
Autonomous machine
03
API
04
Human provider
05
Remote operator
06
Service provider
Outcome completed
A resilient real-world system should be able to preserve the intended outcome even when the original executor becomes unavailable. That requires the abstraction to sit above any single machine: the system commits to an outcome, and treats the executor as a replaceable resource.
Outcome first.
Executor second.
// Atlas thesis 043
A reliable robot is useful.
A reliable outcome is infrastructure.
Human-in-the-loop as recovery infrastructure
Research note 001 examined the human as the source of authority: the party who grants permission to act. There is a second role, and it appears only after something goes wrong.
Human as authority
- Approval
- Authorization
- Supervision
Human as recovery infrastructure
- Context interpretation
- Exception handling
- Alternative path creation
- Physical intervention
Recovery path // access
Recovery path // substitution
Humans may not disappear
from Physical AI.
They may move to the exception layer.
The world has no API contract
The physical world does not return clean API responses.
Software defines
- Input
- Output
- Schema
- Error
- Timeout
The real world provides
- Ambiguity
- Delay
- Weather
- Humans
- Friction
- Physics
- Uncertainty
- Non-determinism
The world
does not violate the specification.
The world is the specification.
Atlas thesis // interpretation, not a sourced scientific claim
Which means any infrastructure operating in reality has to be built around three things that software systems can often omit: observation, adaptation and verification.
Closed-loop execution
Open loop
- Plan
- → Action
No feedback // no knowledge of the result
Closed loop
- Plan
- ↓ Action
- ↓ Observe
- ↓ Compare
- ↓ Adapt
- ↺
↺ observe · compare · adapt
The difference
between action and execution
is feedback.
A physical system needs continuous state awareness, because the only evidence that something happened is an observation of the world after it happened.
- 01
Did the vehicle arrive?
- 02
Did the robot pick up the object?
- 03
Did the provider confirm?
- 04
Did the package reach the location?
- 05
Did the door actually open?
- 06
Did the human receive the service?
No verification.
No verified outcome.
Verification
When is
the task actually done?
- The agent sent the request.Not done.
- The provider accepted.Not done.
- The robot started moving.Not done.
- The autonomous vehicle arrived.Maybe.
- The intended human reached the destination.Done.
Activity
is not outcome.
Execution lifecycle
- Task initiatednot done
- Task acceptednot done
- Task executingnot done
- Task deliverednot done
- Outcome verified✓ done
What research is trying to solve
A growing body of robotics research treats the divergence between expected and observed state as the central problem rather than a nuisance. Below, each entry separates what a source demonstrates, what it does not claim, and our interpretation. Nothing outside the first column is evidence.
Failure explanation and correction
REFLECT — Columbia University, CoRL 2023 →Research finding
The framework converts multisensory robot experience into a hierarchical summary, uses an LLM to identify and explain a failure, and generates a corrective plan from that explanation.
Limitation
Demonstrated on a curated set of household-scale simulated and real tasks. It does not claim general recovery across arbitrary open-world deployments, nor coverage of non-robot failure sources such as providers, payments or access.
Our read
Failure explanation is being treated as a first-class capability rather than logging. That is the precondition for any recovery policy: a system cannot choose a response to a failure it cannot characterise.
Plan-execution misalignment
DoReMi — Tsinghua IIIS / Shanghai Qi Zhi, 2023 →Research finding
An LLM emits constraints alongside each plan step; a vision-language model monitors those constraints during execution and triggers recovery when low-level execution deviates from the plan.
Limitation
Constraint monitoring is only as good as the constraints generated and the perception behind them. The work targets misalignment detection in robot tasks, not organisational or multi-party execution failure.
Our read
This is the closest published analogue to the compare stage in our loop: the plan carries its own expected-state description, so divergence is detectable rather than inferred after the fact.
Closed-loop replanning
Inner Monologue — Google / Stanford, CoRL 2022 →Research finding
Injecting closed-loop language feedback — success detection, scene description, human responses — into an LLM planner improves performance on long-horizon embodied tasks without additional training.
Limitation
Feedback quality is bounded by the success detectors and describers supplying it. The paper studies planning behaviour, not deployment reliability or safety guarantees.
Our read
Notably, one of the feedback channels is a human. Human input is already being modelled as a runtime signal inside the execution loop, not only as an approval step before it.
Long-horizon closed-loop planning
NovaPlan — Robotics and AI Institute / CMU / Brown / UPenn →Research finding
Long-horizon manipulation is framed as closed-loop video-language planning, where the system continues to reason about physical outcomes across a task rather than committing to a single decomposition.
Limitation
A research system evaluated on manipulation benchmarks; it does not address executor substitution, human escalation or multi-party outcome verification.
Our read
The direction of travel matters more than the numbers: long-horizon work is converging on loops, not longer pipelines.
Autonomous system fallback
Waymo — safety case approach, 2023 →Research finding
A deployed autonomous-driving safety case is structured around layered fallback behaviour, operational readiness and continuous field monitoring, rather than an assumption of nominal operation.
Limitation
Domain-specific to driving, and a framework for arguing absence of unreasonable risk — not a general recovery architecture for arbitrary physical tasks.
Our read
The most mature real-world autonomy deployment organises itself around what happens when things go wrong. That is the pattern we expect to generalise across Physical AI.
Fallback definitions
SAE J3016 — driving automation taxonomy →Research finding
The standard formally defines the fallback role and the minimal risk condition: who or what is responsible when the automated system can no longer perform the task.
Limitation
Applies to driving automation only. There is no equivalent cross-domain standard defining fallback responsibility for robots, agents and service providers acting on human intent.
Our read
One physical domain has agreed language for failure responsibility. The broader Physical AI stack does not yet. That absence is itself a frontier.
The common thread: current work is strongest at detecting that expected state ≠ observed state, and weakest at deciding what the system owes the human once that is true.
The recovery stack
Real-world recovery layer // conceptual architecture
Intent
What the human wants
Expected outcome
The measurable definition of done
Plan
A claim about a future world
Execution
Matter changes state
Observation
What is actually true now
State difference detection
Expected ≠ observed
Failure classification
What kind of failure this is
TransientEnvironmentalExecutorAccessHumanCommercialRecovery policy
What the system owes the outcome
Recovery action
Replan / retry / switch executor / escalate
MachineHumanProviderAlternative machineNew execution
Same intent, different path
Verification
Confirm reality, not activity
Outcome
The intent survived
Recovery primitives
01
Retry
02
Replan
03
Reroute
04
Reassign
05
Escalate
06
Switch executor
07
Change constraint
08
Ask human
09
Abort safely
10
Verify again
Our read
// Real World Atlas — interpretation, not sourced findingPhysical AI is usually discussed in terms of capability. Can the robot walk? Can it grasp? Can the vehicle drive? Can the agent plan? Those questions matter, and the progress behind them is real.
But infrastructure becomes valuable at a different moment: when capability survives contact with reality.
Can the system preserve the outcome when the original plan no longer works?
A robot may fail.
A provider may fail.
An API may fail.
A human may fail.
The execution layer must not confuse
executor failure
with outcome failure.
The first trilogy
001 / Authority
Who gives AI permission to act?
002 / Orchestration
If atoms become bits, what becomes the operating system?
003 / Reality
What happens when the world refuses the plan?
May I act?
How do I act?
What do I do when action fails?
Authority.
Orchestration.
Recovery.
Three infrastructure problems between intelligence and reality.
// Atlas thesis 003
AI can plan.
Machines can act.
Reality gets the final vote.
The future of Physical AI may belong not to systems that never fail, but to systems that know what to do next.
Execution is not complete
when something acts.
It is complete
when the intended outcome survives reality.
// Real World Atlas
// Research note 003
// Milan, Italy
// Reality observed
Source architecture
Section 11 is source-derived. Every other section — the loop, the recovery stack, executor substitution and the Atlas theses — is our framing, and is not attributed to any of the researchers or organisations below. No claim on this page states that a cited work solves general real-world recovery.
Columbia University (Liu, Bahety, Song)
Peer-reviewed / CoRL 2023REFLECT: Summarizing Robot Experiences for Failure Explanation and Correction — Framework that builds a hierarchical summary of robot sensory experience and queries an LLM to detect, explain and generate a correction plan for failed executions.
Primary source →Tsinghua IIIS / Shanghai Qi Zhi (Guo, Wang, Zha, Jiang, Chen)
Research preprintDoReMi: Grounding Language Model by Detecting and Recovering from Plan-Execution Misalignment — Uses an LLM to emit constraints alongside the plan and a vision-language model to monitor them during execution, triggering recovery when execution and plan diverge.
Primary source →Google / Stanford (Huang, Xia, Xiao et al.)
Peer-reviewed / CoRL 2022Inner Monologue: Embodied Reasoning through Planning with Language Models — Closed-loop language feedback — success detection, scene description and human interaction — fed back to the planner so it can replan without additional training.
Primary source →Robotics and AI Institute / CMU / Brown / UPenn
Research preprintNovaPlan: Zero-Shot Long-Horizon Manipulation via Closed-Loop Video Language Planning — Long-horizon manipulation framed explicitly as closed-loop planning rather than one-shot plan generation.
Primary source →Waymo
Industry safety frameworkBuilding a Credible Case for Safety: Waymo's Approach for the Determination of Absence of Unreasonable Risk — Deployed autonomous-driving safety case structured around layered fallback, operational readiness and continuous field monitoring rather than assumed nominal operation.
Primary source →SAE International
StandardJ3016 — Taxonomy and Definitions for Terms Related to Driving Automation Systems — Defines the fallback role and the minimal risk condition: what an automated driving system, or a human, is expected to do when the driving task can no longer be performed.
Primary source →
Research finding
Traceable to the peer-reviewed or first-party material listed above.
Real World Atlas interpretation
Our question, our framing. Never presented as evidence.
Publication record
Real World Atlas Research · Published 2026-08-19 · Modified 2026-08-19 · Milan, Italy
Related research
Next signal // Research note 004
When does a robot become economically inevitable?
Cost curves, utilization, reliability, labor economics and Physical AI deployment.
[ Coming next ]
// Milan, Italy
// Real World Atlas
// Observation continues