Recently Google shared a project called Creatability to their Experiments page which explores how creative tools can be used in combination with technology such as AI to provide a more accessible experience to those with disabilities. The various experiments involved web pages that use a combination of webcam and keyboard or mouse input to create art and music. The webcam input allowed users to move their body or face as opposed to moving a mouse, making technology significantly more accessible to those with limited movement and dexterity. Body movements are captured using Posenet, a machine learning model that detects human movement in real time, and the experiments themselves are open source and complete with tutorials on how to create your own experiments with Posenet and P5.js.
All of this is very important in a time where assistive technology can still be very expensive; eye tracking technology can cost anywhere between $100-$10,000, and Creatability’s face tracking features could be just as effective in those that are still able to make minute head movements when they shift their gaze. And technology’s developments in accessibility don’t stop there- Max MSP programmers are making great progress in webcam-based eye tracking already. Furthermore, Drake Music, an organisation working in music, disability, and technology offer up a list of accessible music technology resources, ranging from high-end assistive technology to DIY instruments and computer programs.
Inspired by the Creatability project, I decided to have a go at recreating a few of their experiments with Max MSP. So far I have created a patch that uses head movements to draw lines, and another patch that uses head movements to play music. I managed to create them fairly easily and included a download link to the files so that people can develop on the basic features. They are simple patches, but I aim was not to create something groundbreaking but rather than to point out this: if this technology is so readily available to us and so easy and affordable to create and use, then why has it taken us this long to get here?