Building an FSD wheelchair with a post-trained model
A condensed version of the post first published on ishaikh.com. Photos and videos are from the original.
Over one weekend our team added full self-driving to an off-the-shelf electric wheelchair. The stack is an Arduino UNO Q on the chair, a private mesh network, the OmniVLA robotics model post-trained on data we curated with Google DeepMind's Gemini Robotics-ER 2, and a simple iPad interface aimed at elderly riders and people with disabilities. Chairs that drive themselves can cost $30,000 or more. Ours cost about 1% of that.
What it does
The rider types or speaks a destination on the iPad: an address, or something like "take me to the bathroom". The chair works indoors and outdoors in three modes.
A phone mounted on the chair streams the camera, gyro and GPS. The iPad shows the trip and allows manual override, and the original joystick still works as an emergency stop.
Talking to the chair
The joystick turned out not to send an analog voltage. The chair polls a small magnetic sensor under the stick about 400 times a second, using a timing quirk the Arduino's built-in hardware rejects. Each failed attempt made the chair beep at us. We reverse-engineered the protocol in C++ until the Arduino could pass as that sensor. Nothing was cut or soldered, so the conversion is non-invasive and easy to undo.




Everything else talks over a Tailscale mesh. The phone sends sensor data, a model picks where to go, a backend turns that into stick commands, and the iPad is the rider's screen.
Why the base model wasn't enough
We picked OmniVLA after testing robotics models on public benchmarks. It handles indoor and outdoor scenes, accepts plain-language goals and is small enough to be fast. On a rented A10 GPU through Modal it answers about four times a second, and each answer gives eight waypoints. We call it again every ~28 cm of travel. It still had three problems:
- Size. It was trained on small robots, but our camera sits 28 cm up on the chair's front-left corner with the rest of the chair to its right.
- Stopping point. It stopped as soon as the target was fully in view, not where a rider could get off.
- Memory. It sees one frame at a time, so after steering around a box it would turn straight back into it.
Three experiments
1. Let an LLM drive
We gave Claude the camera image, the chair's geometry, the full history of moves and a small set of actions. It understood the scenes better than OmniVLA but drove badly: 2–3 seconds per step made the ride stop-and-go. Haiku was a bit faster but no better at judging distance, and extended thinking didn't help.
2. Calibrate the camera view
Gemini Robotics-ER 2 is built to point at things in images. We taped parking-camera style guidelines on the floor to show where the chair's sides would pass, with marks at 0.5, 1 and 2 m, plus up to seven numbered paths from 60° left to 60° right. That made the chair's footprint and real depth part of what the model sees.
3. Let the model draw the route

Then we split the job. Gemini marks the goal and draws a route as a few points on the floor. The harness projects those points onto the floor using phone-camera depth and keeps following the route between model replies. Each reply is a small JSON object with a one-line scene description, the goal and route points as image coordinates, and a short note to its future self like "asked left 30°, gyro measured left 21°". The harness remembers the goal even when the camera is blocked.




This was the first driver that reliably reached a named object: the bottom of the stairs around an ottoman in 37 s, a person in 12 s and a rocking chair in 14 s. It was slow to think, though.


Seconds per decision, bars scaled to 13 s (warm bar shows the 4.9 s upper end). By the time Gemini answers, the picture it saw is already old.
How the harness drives
- Depth without LiDAR. Depth Anything V2 with a calibrated scale. At 0.84 m it's about 3 cm off when checked against tape on the floor.
- Pure pursuit. The chair steers along the arc through a point ahead on the route and slows on tight arcs so the rider isn't thrown sideways.
- Memory. A map of 15 cm-radius cells, where obstacles show up as clusters.
- Turns. The phone's gyro checks how far the chair actually turned compared with what was commanded.

Turning drives into training data
Physical data can't be generated by a script overnight. Every sample means moving the real chair, so we built the data loop first. Each run logs gyro, command, camera frame and believed versus actual position, along with a hand-curated "golden" solution. Every failure is labeled with a reason ("obstacle 0.4 m ahead", "gyro went quiet"), and runs with stale camera or gyro data are thrown out automatically.
How logged drive segments ended.
Post-training recipe
We fine-tuned OmniVLA with supervised learning on hand-driven runs and Gemini-driven runs, teaching it to predict where the chair will be over the next eight waypoints from one frame and a task. We used OmniVLA's own training script.


Annotated training samples: the task, plus in orange where the chair actually went over the next ~2.7 s.
- Adapter
- LoRA rank 32 (α 16) on every linear layer, with the action head and goal projector trained fully
- Loss
- Waypoint MSE plus a 0.1 smoothness term
- Optimizer
- AdamW, learning rate 1e-4
- Schedule
- 300 steps, batch size 8, one 80 GB GPU on Modal, about five minutes
Results
On held-out slices from the middle of each drive, error at the last waypoint dropped from 0.40 m to 0.11 m, 3.7× lower than the original. A model that always stands still scores 0.28 m, so the base model was worse than doing nothing. Heading error fell from 4.6° to 1.9°, and average error across all eight waypoints fell from 0.35 m to 0.09 m.
Path error at the last waypoint on held-out frames. Lower is better.
We also replayed 49 recorded frames through every driver, reduced each answer to its first action, and had a separate Claude Opus judge pick among them blind, with names and reasoning removed.
Blind-judged first actions on the same 49 frames, scale 0–1, with median seconds per decision.
The slow models were far more accurate, so we used distillation to move that judgment into the small, fast model.
Takeaways
- Harnesses matter as much for robotics models as for LLMs, especially for latency.
- Give the model concrete measurements as context instead of hoping it infers them.
- With physical data, fast iteration depends on building the collection and training loop early.