Know what your users will do before your agent builds it.
User stories checked against your data, turned into specs your coding agent can ship. Applied in Woku; simulated users are a separate research prototype.
P. Riquelme and D. Orrego · OnPuck
Illustration of a simulated run. Each person carries frustration, fatigue and motivation, and leaves under the simulated-user rule. Drawn, not recorded. Hover or tap a person.
- Friction
- Tired
- Done
- Leaving
- Voice
AI user research for coding agents: user stories checked against your data and specs tested on simulated users, for growing startups in San Francisco, CA, USA.
How OnPuck works
Building got cheap. Building the wrong thing didn’t.
OnPuck finds out what your users really need, and hands your coding agent a spec it can trust.
- 1
Plug in your stack
Read-only access to your repo and data. Revoke it anytime.
- 2
We talk to your users
Short interviews about one real moment, every detail checked against your data.
- 3
Your agent ships the right fix
A spec with the evidence attached, ready for your coding agent.
What changes for your team
| Today | With OnPuck |
|---|---|
| Guessing from a handful of interviews | Every story checked against your data |
| Vague tickets your agent misreads | Specs with the evidence attached |
| Building on what users say | Building on what they actually did |
| Research eating your team’s week | Run for you, free during the partnership |
Works with the tools you already use
GitHub
PostHog
Sentry
Axiom
CloudWatch
Postgres
MongoDB
Claude Code
Codex
Cursor
Copilot
OpenCode
What OnPuck is: AI user research for coding agents
OnPuck turns user stories into specs checked against code, analytics and logs. The founders applied this workflow in Woku and shipped changes. We are building a repeatable platform for coding agents, alongside a separate research prototype that tests preview builds with simulated users.
- 01Interviews explain why.
- 02Data proves how many.
- 03Trackers join the two.
- 04Simulation tests the change before it is built.
Figure 1The loop
- Step 1
Connect
Trackers verified against live data
Applied in Woku - Step 2
Listen
One concrete story per interview
Human-led in Woku - Step 3
Anchor
Each detail confirmed or contradicted
Applied in Woku - Step 4
Spec
A spec with the evidence attached
Applied in Woku - Step 5
Simulate
A predicted change, with an interval
Research prototype - Step 6
Ship
The change in production
Shipped in Woku; MCP planned - Step 7
Calibrate
An accuracy record
By hand in exp-01 to exp-04
1Why user interviews and product analytics disagree
Interviews tell you why. Analytics tell you how many. Neither tells you if the story is true.
Across 52,007 people, self-reports about software use rarely matched the logs [32]. So OnPuck checks every tracker: a detail your system could have recorded, like a time, a device, a feature or an error.
Figure 2One story, resolved
Interview with a synthetic example user
“Last night I tried to export the monthly report from my phone, and it never downloaded.”
| Tracker | User said | System recorded | Resolution |
|---|---|---|---|
| Time (anchor) | “last night” | export_clicked at 21:42 | confirmed |
| Device (dimension) | “my phone” | iOS Safari, 390 x 844 | confirmed |
| Feature (dimension) | “export the monthly report” | POST /reports/export returned 200 | confirmed |
| Outcome (dimension) | “it never downloaded” | File served, 1.2 MB; never opened | contradicted |
Resulting spec
fix Make the exported file findable on iOS Safari
The download worked. The user could not find the file.
4 trackers, 3 queries, 1 story
2How OnPuck turns user stories into specs for your coding agent
Seven steps, each backed by research or by our own experiments.
- Step 1
Connect
Applied in Woku - Step 2
Listen
Human-led in Woku - Step 3
Anchor
Applied in Woku - Step 4
Spec
Applied in Woku - Step 5
Simulate
Research prototype - Step 6
Ship
Shipped in Woku; MCP plannedThe change in production
Woku changes were implemented with agents. Repeatable delivery to your agent over MCP [27] is being built.
- Step 7
Calibrate
By hand in exp-01 to exp-04An accuracy record
We compare prediction and reality, then recalibrate. Model versions are pinned [13].
3From customer interviews to shipped changes in Woku
Before digital users, we ran the evidence workflow on Woku and shipped changes from its customer stories.
Woku is the feedback product the founders operate. Its customer interviews were human-led; agents helped trace stories through code and read-only data into specifications.
One investigation catalogued 79 trackers: 65 marked verified, 7 stale and 7 unverifiable. Account analyses retained confirmations, contradictions and gaps.
- Review reliability. Customers described blank pages and uncertain submissions. Code and error records shaped the specification. The implementation added retries, pending drafts and explicit submission outcomes; its engineering plan records a production rollout.
- Repeated account warnings. An interview about repeated inactivity emails was traced to duplicate jobs and account records. Warning and account deduplication were implemented; the batch plan records a production rollout.
- Reports by period. Customers assembled reports manually. Checking report scope exposed missing date controls. Period filters and reports were implemented; the batch plan records their production rollout.
This field work demonstrates execution and delivery. It does not establish causal uplift or simulator accuracy. Woku is the founders’ lab, not an independent OnPuck customer.
We are turning that agent-assisted process into a repeatable platform. Human review and source links remain part of the workflow.
4Simulated users that get tired, frustrated and quit
An LLM alone plays a user who is too efficient. We add the part that gets tired.
LLM agents take 2.5 times fewer actions than people [6], know too much [4] and flatten groups [15]. Each simulated user keeps an explicit state: frustration R, fatigue F, boredom B, progress P and motivation M.
RuleChance of quitting at each step
hazard = σ(h0 + 3R + 1.5F + 1.5B − 3M − 2P)
h0 is fitted on real sessions (exp-02).
Figure 3Simulated run
Registered prediction
People whose first action is writing, among those who start.
- Rating first7%
- Rating after the comment20%
This replay so far
- 0
- started
- –
- wrote first
- 0
- submitted
Result pending. We publish it either way.
5How accurate are simulated users? Our validation
Four experiments on woku, misses included, and one blind prediction waiting for its result.
| Experiment | Question | Predicted vs real | Verdict |
|---|---|---|---|
| exp-01BacktestPartial | Can a simulated user predict how submissions changed between two past versions of woku? 560 simulated vs 1,499 real sessions | PredictedΔ +0.028 [−0.016, +0.072]RealΔ −0.022 [−0.084, +0.040] | Inside the real interval; the generic prior was not. But "nothing changes" was too, and the model missed real abandonment. |
| exp-02RecalibrationMiss on Δ, bug found | Recalibrated on real sessions, does it reproduce abandonment out of sample? 280 simulated sessions | PredictedAbandonment after starting 0.24; Δ +0.061RealAbandonment after starting 0.20; Δ −0.022 [−0.084, +0.040] | Abandonment reproduced; the Δ missed. A simulated user found a real bug: tapping a tab submitted the form. Fixed in production. |
| exp-03VarianceHit | Same model, better estimator: was the exp-02 miss just noise? 120 paired simulated sessions | PredictedΔ −0.011 [−0.046, +0.025]RealΔ −0.022 [−0.084, +0.040] | Yes. Inside the real interval, 62% narrower, and it excludes the generic prior. It can’t yet beat "nothing changes". |
| exp-04Blind predictionRegistered, result pending | Blind, before a live A/B: what changes when the rating moves after the comment? 540 simulated; up to 600 real sessions per arm | PredictedWriting first goes from 7% to 20% (Δ +0.131 [+0.059, +0.202])RealResult pending | Registered (git tag and SHA-256) before any real data. It can clearly fail, and we will publish it either way. |
Figure 4Prediction vs reality
Real 95% interval Simulator 95% interval Generic prior “Nothing changes”
Best result so far: a simulated user found a real bug on woku’s review form (exp-02). It was fixed in production.
6The research behind AI interviews and simulated users
What the research found, and what OnPuck does with it.
In OnPuck todayIn the prototype
Our prototype uses sessions; interview-grounded digital users remain planned.
In OnPuck todayIn the prototype
A state layer decides when someone gets tired or quits.
Research
Simulations get the direction right and the size wrong [5].
In OnPuck todayIn the prototype
We report a direction and an interval, never exact magnitudes.
In OnPuck todayIn the prototype
The model proposes actions with probabilities; code samples them.
Research
Keystroke-level models predict task time [20].
In OnPuck todayIn the prototype
Time per action is fitted on real sessions.
In OnPuck todayIn the prototype
We run your real preview build instead.
In OnPuck todayRun by hand
Every prediction is scored against "nothing changes". Misses are published.
In OnPuck todayDesigned, not built
Human-led Woku interviews; automated interviewing is specified.
In OnPuck todayRun by hand
We checked Woku trackers against live data and resolved existing interview stories.
In OnPuck todayDesigned, not built
Specs carry their evidence, designed to reach your agent over MCP [27].
7What simulated users cannot predict
What we can’t promise, and what we do about it.
Exact magnitudes
Why not.Simulations overestimate effect sizes [5].
What we do.We give a direction and an interval, scored against "nothing changes".
Brand-new features
Why not.Hypothetical answers flatter [16]; twins transfer poorly to new behaviour [14].
What we do.We test changes to flows people already use.
One person’s exact session
Why not.Next-action accuracy is 11 to 21% [35, 36].
What we do.We predict rates across a population, never one person.
Replacing interviews
Why not.Without real input, simulated people are generic [15].
What we do.Interviews feed every step.
Stable results across models
Why not.Results shift with prompts and models [13].
What we do.Model versions are pinned and baselines re-run.
Transfer beyond Woku
Why not.The evidence workflow has been applied in the founders’ product.
What we do.Design partners test repeatability, cost and usefulness in other teams.
8Design partner program for startups in San Francisco
We’re building OnPuck with 5 teams. The founders run it for you while the platform is built.
- Length
- 3 months, renewable
- Check-in
- Every two weeks
- Cost
- Free
- Cohort
- 5 teams in San Francisco
What you get
- Trackers your system can verify
- Interviews with your users, checked against your data
- Specs your coding agent can ship
- Simulated runs before you build
- Every prediction scored against reality
What we ask
- Read-only access to your repo and one data source
- A check-in every two weeks
- Intros to a few of your users
- Honest feedback
A good fit if
- You ship weekly, ideally with Claude Code, Codex or Cursor
- You have users in production
- You run PostHog, Sentry, Axiom or CloudWatch
- Seed to Series A, in San Francisco
Apply to be a design partner or book a call
Apply in five fields, or just book a 30-minute call with the founders.
Design partner application
Questions about access, data, time and cost
What CTOs ask us first.
Read-only access to your repo and one data source: PostHog, Sentry, logs or a database replica. You grant it and can revoke it any time. Nothing we run writes to your systems.
Two weeks to connect, a check-in every two weeks, and introductions to a few users for 10-minute interviews.
It is a research prototype: four experiments on woku, misses published, one blind prediction pending. Every prediction we make for you comes with an interval and gets scored.
No. Interviews are the input. OnPuck checks each one against your data and turns it into a spec.
Interview tools don’t check stories against your data. Synthetic users simulate opinions and publish no accuracy. Analytics shows how many, never why. OnPuck joins the three on your stack.
Nothing during the partnership. After three months we decide together whether to renew. Either side can stop any time.
The founders, P. Riquelme and D. Orrego: two engineers who research and build the models behind OnPuck, and run woku, our lab. Write to paula@onpuck.com.
Human-led interviews, agent-assisted tracker discovery and story checks produced specifications and shipped changes. The field work is separate from simulator validation. Woku is the founders’ product, not an independent OnPuck customer.
References
Every claim on this page links to one of these papers or to our experiments.
- [1]Park et al. Generative Agents: Interactive Simulacra of Human Behavior. UIST, 2023. arXiv:2304.03442Twenty-five LLM agents with memory, reflection and planning produced believable individual and social behaviour in a sandbox town.
- [2]Park et al. LLM Agents Grounded in Self-Reports Enable General-Purpose Simulation of Individuals. arXiv preprint (first titled "Generative Agent Simulations of 1,000 People"), 2024. arXiv:2411.10109Agents grounded in interviews with real people predicted their answers better than agents built from demographics alone (83 to 86% vs 74%).
- [3]Argyle et al. Out of One, Many: Using Language Models to Simulate Human Samples. Political Analysis, 2023. arXiv:2209.06899Conditioned on real demographic backstories, a language model reproduced the response patterns of human subgroups ("silicon samples").
- [4]Aher, Arriaga, Kalai Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies. ICML, 2023. arXiv:2208.10264LLMs replicated several classic studies but showed a "hyper-accuracy distortion": simulated people knew too much.
- [5]Ashokkumar, Hewitt, Ghezae, Willer Large language models can predict results of social science experiments. Nature, 2026. gsb.stanford.eduAcross 70 preregistered experiments, LLM simulations predicted the direction of effects about as well as expert forecasters, but overestimated effect sizes.
- [6]Lu et al. AgentA/B: Automated and Scalable Web A/B Testing with Interactive LLM Agents. arXiv preprint, 2025. arXiv:2504.097231,000 LLM agents reproduced the direction of a large A/B test on Amazon, but took about 2.5 times fewer actions and 4.5 times fewer searches than humans.
- [7]Lu et al. UXAgent: An LLM Agent-Based Usability Testing Framework for Web Design. CHI Extended Abstracts, 2025. arXiv:2502.12561Simulated users ran usability tests on web designs and surfaced problems before real participants were recruited.
- [8]Ha, Schmidhuber World Models. arXiv preprint, 2018. arXiv:1803.10122An agent learned a compressed model of its environment and trained a policy inside that learned model.
- [9]Hafner et al. Mastering Diverse Domains through World Models (DreamerV3). Nature, 2025. arXiv:2301.04104One world-model algorithm with fixed settings learned more than 150 diverse tasks, including collecting diamonds in Minecraft from scratch.
- [10]Chae et al. Web Agents with World Models: Learning and Leveraging Environment Dynamics in Web Navigation. ICLR, 2025. arXiv:2410.13232LLM web agents struggle to predict the outcome of their own actions; a learned model of web transitions improved their choices.
- [11]Zheng et al. Code2World: A GUI World Model via Renderable Code Generation. arXiv preprint, 2026. arXiv:2602.09856A GUI world model predicts the next screen of an app by generating code that renders it.
- [12]Binz et al. Centaur: a foundation model of human cognition. Nature, 2025. arXiv:2410.20268A model fine-tuned on 160 psychology experiments predicted human choices in held-out experiments better than classic cognitive models.
- [13]Anthis et al. LLM Social Simulations Are a Promising Research Method. ICML, 2025. arXiv:2504.02234Simulation results shift with prompts and model versions; the paper lays out the challenges to solve before trusting them.
- [14]Peng et al. Digital Twins as Funhouse Mirrors: Five Key Distortions. arXiv preprint, 2025. arXiv:2509.19088Digital twins of real respondents show systematic distortions and transfer poorly to new behaviour.
- [15]Wang, Morgenstern, Dickerson Large language models that replace human participants can harmfully misportray and flatten identity groups. Nature Machine Intelligence, 2025. arXiv:2402.01908LLMs prompted with identity labels misportray demographic groups and flatten their diversity.
- [16]Nielsen Norman Group Synthetic Users. nngroup.com, 2024. nngroup.comSynthetic users give vague, overly favourable answers to hypothetical questions and miss what real users do.
- [17]Kirk et al. Understanding the Effects of RLHF on LLM Generalisation and Diversity. ICLR, 2024. arXiv:2310.06452RLHF improves generalisation but sharply reduces the diversity of model outputs.
- [18]Hu et al. SimBench: Benchmarking the Ability of Large Language Models to Simulate Human Behaviors. ICLR, 2026. arXiv:2510.17516Instruction tuning helps models simulate questions people agree on and hurts on questions where people disagree.
- [19]Zhang et al. Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM Diversity. arXiv preprint, 2025. arXiv:2510.01171Asking a model to state several responses with their probabilities restores diversity lost to mode collapse.
- [20]Card, Moran, Newell The Keystroke-Level Model for User Performance Time with Interactive Systems. Communications of the ACM 23(7), 1980. doi:10.1145/358886.358895Expert task time can be predicted as a sum of keystroke, pointing, homing and mental operators.
- [21]Jimenez et al. SWE-bench: Can Language Models Resolve Real-World GitHub Issues?. ICLR, 2024. arXiv:2310.06770Real GitHub issues from popular repositories need coordinated changes across files.
- [22]Yang et al. SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering. NeurIPS, 2024. arXiv:2405.15793The design of the agent-computer interface, not only the model, drives results on real repositories.
- [23]Lei et al. Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows. ICLR, 2025. arXiv:2411.07763An agent solved 21.3% of enterprise text-to-SQL tasks vs 91.2% on Spider 1.0; success needed metadata and the project’s codebase.
- [24]Li et al. Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs (BIRD). NeurIPS, 2023. arXiv:2305.03111On dirty, real database values, model execution accuracy was 40% vs 93% for humans.
- [25]Chen et al. Automatic Root Cause Analysis via Large Language Models for Cloud Incidents (RCACopilot). EuroSys, 2024. arXiv:2305.15778On a year of incidents at Microsoft, LLM root-cause accuracy reached up to 0.766.
- [26]Wang et al. RCAgent: Cloud Root Cause Analysis by Autonomous Agents with Tool-Augmented Large Language Models. arXiv preprint, 2023. arXiv:2310.16340Tool-using autonomous agents performed root-cause analysis on a real cloud platform’s incidents.
- [27]Model Context Protocol Model Context Protocol specification. modelcontextprotocol.io, 2025. modelcontextprotocol.ioAn open protocol that connects models to tools and data through explicit, scoped capabilities.
- [28]Xiao et al. Tell Me About Yourself: Using an AI-Powered Chatbot to Conduct Conversational Surveys with Open-ended Questions. ACM TOCHI, 2020. arXiv:1905.10700In a field study of about 600 people, a chatbot drew more informative, relevant, specific and clear answers than a Qualtrics survey.
- [29]Wuttke et al. AI Conversational Interviewing: Transforming Surveys with LLMs as Adaptive Interviewers. LaTeCH-CLfL, 2025. arXiv:2410.01824Randomised AI and human interviewers on the same guide reached comparable interview quality.
- [30]Geiecke, Jaravel Conversations at Scale: Robust AI-led Interviews. CEPR Discussion Paper 19705, 2024. papers.ssrn.comAn open-source AI interviewer, validated against human interviewers, ran qualitative interviews at scale.
- [31]Flanagan The critical incident technique. Psychological Bulletin 51(4), 327–358, 1954. doi:10.1037/h0061470Record specific behaviour in a concrete incident, not opinions or general estimates.
- [32]Parry et al. A systematic review and meta-analysis of discrepancies between logged and self-reported digital media use. Nature Human Behaviour 5, 1535–1547, 2021. doi:10.1038/s41562-021-01117-5Across 106 comparisons and 52,007 people, self-reported digital media use correlated only moderately with logs and was rarely accurate.
- [33]Belli et al. Event History Calendars and Question List Surveys: A Direct Comparison of Interviewing Methods. Public Opinion Quarterly 65(1), 45–74, 2001. doi:10.1086/320037Event history calendars gave more accurate retrospective reports than question lists.
- [34]OpenAI Introducing SWE-bench Verified. openai.com, 2024. openai.com93 developers screened 1,699 tasks and discarded those with underspecified issues or unfair tests; 500 were kept.
- [35]Wang et al. OPeRA: A Dataset of Observation, Persona, Rationale, and Action for Evaluating LLMs on Human Online Shopping Behavior Simulation. arXiv preprint, 2025. arXiv:2506.05606The closest public dataset joining what people did with why: 51 users, 692 sessions; the best model predicted the next action 21.5% of the time.
- [36]Lu et al. Can LLM Agents Simulate Multi-Turn Human Behavior? Evidence from Real Online Customer Behavior Data. arXiv preprint, 2025. arXiv:2503.20749On 31,865 real shopping sessions, prompted LLMs predicted the next action about 12% of the time; fine-tuning reached 17.3%.