The painful gap between possible and supported
Using a webcam, tablet, open-source control tools, and a small amount of AI-assisted setup, I assembled a communication workflow that could be operated without a conventional mouse or voice. The technical prototype was achievable for hundreds rather than many thousands of dollars. The hard realization was that a prototype is not the same as dependable assistive care.
Start with the person, not the tracker
A useful AAC system must match vision, cognition, fatigue, positioning, motor control, language, caregiver support, and the environments where communication happens. An occupational therapist, speech-language pathologist, assistive-technology professional, or rehabilitation team should guide selection and safe use. AI can help document and adapt; it must not replace that assessment.
A responsible prototype path
- Write the communication needs in the person's own priorities: pain, comfort, yes/no, family, care, and urgent requests.
- Choose the lowest-effort reliable input available: touch, switch, head tracking, hand gesture, or professionally evaluated eye gaze.
- Test with a tiny high-contrast vocabulary before adding pages of options.
- Measure target accuracy, false activations, fatigue, setup time, and caregiver recovery steps.
- Provide a non-AI fallback board and a clear way to stop the system.
- Document installation, calibration, charging, privacy, and support contacts.
Where AI helped
An AI coding assistant connected existing tools, wrote small setup scripts, and helped reduce the time from research to a functioning trial. The correct success metric was not lines of code. It was whether the person could express an intended message with less effort and fewer misunderstandings.
Privacy and safety
Camera input and health-related communication are sensitive. Prefer local processing when possible, minimize stored data, obtain consent, protect API keys, and never route emergency communication through an experimental service without a reliable fallback.
Software, links, and setup
- TD Snap as one commercial AAC option to evaluate with professionals.
- OptiKey, an open-source on-screen keyboard and AAC project.
- Camera Mouse for webcam-based pointer research.
- MediaPipe Hand Landmarker for prototyping gesture input.
- OpenAI text-to-speech documentation for optional generated speech.
Not medical advice: evaluate any assistive setup with qualified clinicians and the person who will use it.
Prompt pack
Needs-discovery assistant
Help me organize questions for a qualified AAC or rehabilitation professional. Do not recommend a device or diagnose ability. Ask about communication priorities, languages, vision, fatigue, reliable movements, positioning, cognition, environment, caregiver support, privacy, fallbacks, and how success will be measured.
Accessibility test plan
Create a short supervised evaluation for this prototype. Include target size, contrast, dwell time, false activation, calibration drift, fatigue, message completion time, caregiver recovery, offline fallback, and stop conditions. Protect sensitive data and do not make clinical claims.
Maintenance guide
Turn these verified setup notes into a one-page caregiver checklist with start, calibrate, communicate, recover, charge, clean, privacy, and support steps. Use plain language and pictures where possible.
What to remember
- Begin with communication needs and reliable movement, not novelty.
- Prototype cost is not the same as lifetime support cost.
- Measure fatigue and recovery, not only technical accuracy.
- Clinical guidance, consent, privacy, and a non-AI fallback are essential.
Then test it, document it, and make the next version easier for someone else to reproduce.