How AR/VR Developers Use iPhone Notes to Capture Spatial Computing Insights

Augmented and virtual reality development operates at the frontier of what's technically possible and what's perceptually comfortable. You're simultaneously solving graphics programming challenges, designing novel interaction paradigms, and fighting against motion sickness, latency, and battery drain. The insights from a single day of user testing can reshape an entire design direction — and those insights are most valuable when captured immediately.

iPhone notes give AR/VR developers a capture layer for the observations that happen away from a development workstation: spatial design insights from walking through a physical space, interaction ideas sparked by using consumer AR apps, or performance hypotheses formed while wearing a headset.

Why AR/VR Developers Need Mobile Notes

AR/VR development involves a unique combination of rapid experimentation and constraint navigation. A rendering technique that works perfectly on a desktop GPU may be completely unusable in a battery-constrained headset. An interaction pattern that feels intuitive in a lab becomes disorienting in user testing. These discoveries need to be captured when they happen, not reconstructed from memory.

The field moves fast. New headsets, new platforms, new SDKs, new interaction paradigms — the developer who captures observations systematically builds institutional knowledge that stays current. The developer who relies on memory ends up rediscovering the same performance bottlenecks and UX anti-patterns repeatedly.

What AR/VR Developers Capture in iPhone Notes

Comfort and presence observations: When you notice what breaks presence or causes discomfort in an experience — whether using your own prototype or a commercial product — capture it immediately with specific details. "Hand tracking lag above 15ms causes noticeable visual-motor mismatch, immediately reduces comfort score" is worth capturing even when you're the user, not the developer.

Interaction design discoveries: New interaction paradigms are discovered by trying things, not by theorizing. When a gaze-and-dwell interaction works better than anticipated, or a gesture is consistently misread by users, note the specific conditions. "Pinch gesture works reliably when hand is within 30cm of face but fails at arm's length — needs proximity-aware threshold adjustment."

Performance constraints and solutions: AR/VR performance budgets are tight. When you find a rendering optimization, a culling strategy, or a batching pattern that meaningfully improves frame rate, capture the specific numbers. "Replacing dynamic shadows with baked lightmaps on static geometry reduced draw calls from 340 to 80, gained 12ms on target hardware."

SDK and platform quirks: Every AR/VR platform has undocumented behaviors, quirks, and workarounds. Note them when you discover them — they're valuable for yourself and for the broader developer community. "ARKit plane detection fails to merge coplanar surfaces when room lighting is below 50 lux — need fallback for low-light environments."

User testing observations: Live user testing produces observations faster than you can document them formally. Keep iPhone notes running during sessions to capture behavioral patterns: confusion moments, delight moments, and the spontaneous comments that don't make it into structured feedback forms.

Design space exploration: When walking through a physical space and imagining how AR might augment it, note the ideas immediately. "Warehouse picking workflow: anchoring bin locations to physical positions would eliminate need to look at handheld scanner — pure gaze + voice workflow possible." These contextual insights evaporate quickly.

The AR/VR Developer Observation Note

For performance findings: ``` Platform: [headset/OS/SDK version] Scenario: [what was rendering] Problem: [frame time / battery / thermal] Solution tried: [technique] Result: [ms saved / battery impact] Trade-off: [visual quality / development complexity] ```

For interaction design findings: ``` Interaction type: [gaze/gesture/voice/controller] Context: [what user was trying to do] What worked: [specific behavior] What failed: [specific failure mode] Hypothesis: [why it worked or failed] Design implication: [what to change] ```

For comfort observations: ``` Symptom: [discomfort / presence break / motion sickness] Trigger: [what caused it] Threshold: [at what intensity/duration it occurs] Mitigation: [what reduces it] Reference: [simulator sickness literature if applicable] ```

Connecting Notes to the Development Cycle

AR/VR development cycles involve rapid iteration between prototype and test. Notes captured during testing feed directly into the next iteration's design decisions. A systematic note practice creates a decision log that explains why certain design choices were made — invaluable when returning to a project after months away.

Nemos' organization system helps maintain separate streams for different project types. Notes for a room-scale VR experience have different concerns than notes for a mobile AR feature. Keep them searchable but separate, with cross-links when a finding applies to both contexts.

Staying Current in a Fast-Moving Field

AR/VR platforms ship major SDK updates frequently. Patterns that worked on an older SDK version may be deprecated or superseded. Notes that include platform version context remain useful — you can quickly assess whether a captured optimization is still applicable on the current SDK.

Tag notes with platform version and SDK version. When a new SDK ships, you can review your notes and identify which optimizations and workarounds need re-evaluation. The notes that say "workaround for SDK v2.3 bug" can be evaluated for removal. The notes that say "fundamental comfort principle" remain relevant across SDK versions.

FAQ

Q: Should I keep separate notes for AR and VR, or combine them? A: Separate notebooks make sense if you work across both regularly, since the constraints and design concerns are quite different. Use cross-links for findings that apply to both — comfort principles and interaction design fundamentals often transfer.

Q: How do I capture user testing observations quickly without disrupting the session? A: Use voice-to-text for rapid capture during sessions, then clean up the notes immediately after. The raw voice notes capture the urgency and specificity of live observations. A 5-minute cleanup session right after testing converts them into structured, searchable notes.

Q: What's the most valuable thing to note after discovering a performance optimization? A: The exact numbers before and after, plus the specific conditions (target hardware, scene complexity, rendering settings). Without specifics, optimization notes become vague platitudes that don't help when you return to the same problem on a different project.

Q: How should I note SDK bugs and workarounds? A: Always include the SDK version, the specific behavior, the workaround, and a note to check whether the bug was fixed in later versions. Bugs get fixed; your workarounds shouldn't persist after the underlying issue is resolved.

Q: Do I need a formal note structure for quick observations during testing? A: No. Even a one-line observation with a timestamp is better than nothing. "User consistently grabbed for object that wasn't there — affordance mismatch" takes 5 seconds to note and is infinitely more useful than relying on memory.

Q: How do AR/VR notes connect to broader product development notes? A: Use links to connect technical AR/VR notes to product requirements and user research notes. The technical constraint ("ray casting fails at distances above 5m on this hardware") and the product requirement ("users need to interact with objects across the room") need to be in dialogue with each other.

Related Reading

• /blog/ios-developer-notes-iphone
• /blog/android-developer-notes-iphone
• /blog/game-developer-notes-iphone
• /blog/flutter-developer-notes-iphone

Sources

• Apple Vision Pro Developer Documentation — https://developer.apple.com/visionos/
• Meta Quest Developer Center — https://developer.oculus.com/
• Microsoft Mixed Reality Documentation — https://learn.microsoft.com/en-us/windows/mixed-reality/