Robots Need to Know Who You Are

A home robot walks into a house with four people in it and one account. It knows the coffee routine, the bedtime, the wifi password. What it does not know, because nothing in its memory layer can represent it, is that the bedtime belongs to the kid, the coffee order belongs to the grandmother who is visiting this week, and the wifi password is none of the guest's business. Face recognition will not save it. Faces tell you who is in the room. They do not tell you whose preferences govern which room, who may be told what, or what the family promised each other.

This is part five of the person-centered series, and it is the part where the argument stops being theoretical. Memory research has spent two years teaching agents how to act. Almost nothing teaches them whom to act for. For a robot, that gap is not a UX problem. It is the whole product.

What embodied memory research has already built

The progress is real, and worth stating plainly. Voyager (2023) runs an agent in Minecraft that writes its own skills as executable code and stores them in a library indexed by description, so mastered behaviors can be retrieved and composed; it collected 3.3x more unique items than prior methods largely by reusing what it had already learned. ExpeL (2023) distills cross-task insights from an agent's own successes and failures and lifted HotpotQA success from 28.0% to 39.0% without touching model weights. 3DLLM-Mem (2025) gives embodied agents a working-memory plus episodic-memory architecture for multi-room environments and beat the strongest baselines by 16.5% on the hardest in-the-wild tasks.

Read those three together and a pattern appears. Voyager remembers how. ExpeL remembers what worked. 3DLLM-Mem remembers where. All of it is about things, places, and procedures. None of it is about people. The subject is missing from the embodied stack exactly as it is missing from chat memory, and the fix is the same shape.

The multi-person problem with a body

What does a robot actually need to know about people? Start with the list a household implies:

• Who is who. Not identity in the abstract, but role and relationship: owner, kid, guest, caregiver. Each with different permissions.
• Whose preferences govern which context. The kitchen at 7am, the living room after 9pm, the thermostat never.
• What commitments exist. "Pick up my daughter at three." "Dad takes his medication with breakfast." "The guest leaves for a flight at six."
• Who said what, to whom. A request from the kid is not a request from the guest, and an instruction forwarded through a third party is not an instruction from the owner.

"Make coffee" is three different actions in one house: black, no sugar, for the owner; decaf for the grandmother who is visiting; and nothing at all for the guest who is already late. A skill library can store how to make coffee. Only a person-centered layer can store for whom, and that selector is what decides which skill runs. Get it wrong in chat and you get an awkward message. Get it wrong in a kitchen and someone drinks the wrong thing at the wrong time.

Provenance becomes a safety property

In a chatbot, a memory that cannot attribute is an embarrassment. In a robot, it is an attack surface.

The memory-laundering result we covered earlier in this series reported attack success up to 1.000 against weakly integrated memory: a claim about the user, injected by a third party, gets stored as fact and then quoted back with confidence. Now put that in a body. "Your owner said it is fine" is a sentence that opens doors, moves money, and changes schedules, and it works precisely when the system cannot check who actually said it. Attribution is not a nice-to-have for embodied agents. It is the safety property.

Embodiment generates the richest person data there is

Here is the part that should worry everyone building in this space, including us. A home robot perceives who is in the room, who is stressed, who is sleeping, who is visiting, who has not come home. That is the richest stream of person data any consumer device has ever had access to, and it is collected by something that moves.

Which makes governance the first feature, not the last. A person root in the schema. Provenance on every observation. Consent scoped per person and per context. Deletion that works at the level of the person, not the account. Those four things are the difference between a robot that serves a family and a surveillance device with good manners. We would not ship the second, and we do not think anyone else should either.

What we are building toward

Our own work in this direction is early, and we would rather say so than dress it up. The shape of it is the one the rest of this series argues for: a person-centered memory layer with a person root in the schema, three-valued provenance (model-inferred, human-confirmed, disputed), no auto-confirmation of anything about you without you, subject-scoped retrieval, and "unknown" as a first-class answer. It runs above whatever base model and whatever store you use. It is designed for a house with four people in it, not an account with one.

The test we keep coming back to is the one from part one: everyone is unique, and a generic memory cannot solve everyone's problems. A house is just the fastest way to notice.

References

1. Wang, G., Xie, Y., Jiang, Y., Mandlekar, A., Xiao, C., Zhu, Y., Fan, L., & Anandkumar, A. (2023). Voyager: An Open-Ended Embodied Agent with Large Language Models. arXiv:2305.16291. TMLR 2024.
2. Zhao, A., Huang, D., Xu, Q., Lin, M., Liu, Y., & Huang, G. (2024). ExpeL: LLM Agents Are Experiential Learners. arXiv:2308.10144. AAAI 2024.
3. Hu, W., Hong, Y., Wang, Y., Gao, L., Wei, Z., Yao, X., Peng, N., Bitton, Y., Szpektor, I., & Chang, K.-W. (2025). 3DLLM-Mem: Long-Term Spatial-Temporal Memory for Embodied 3D Large Language Model. arXiv:2505.22657.
4. Memory Provenance Laundering (2026). arXiv:2607.29167.