Skip to main content

Sight enhancement: Disability sparks invention


July 19, 2010

ASU students create a device to help people with visual impairments succeed in the classroom

David Hayden turned frustration with his visual impairment into motivation, and the result has earned an Arizona State University student research team a top prize in a major international competition for technological innovation.

Computer science student Hayden and team member Andrew Kelley recently returned from Warsaw, Poland, with a first-place trophy from the 2010 Imagine Cup Worldwide Finals organized by Microsoft, a global leader in computer software development and services.

Overall, more than 325,000 students from more than 100 countries took part in stages of the competition leading to the finals.

Some 50 student teams from around the world entered a special category of the competition that challenged them to find creative ways to use Microsoft Windows-based Tablet PCs to improve access to education.

Of those 50 teams, the ASU group was one of only two whose projects earned them invitations to Warsaw to compete at the Imagine Cup finals.

Hayden’s and Kelley’s presentation and demonstration of the Note-Taker, a system designed to aid the visually impaired, deeply impressed the panel of judges, said ASU research scientist John Black.

Necessity drove invention

Black mentored the team, which developed the Note-Taker system in the Center for Cognitive Ubiquitous Computing – known as CUbiC – in the School of Computing, Informatics, and Decision Systems Engineering, one of ASU’s Ira A. Fulton Schools of Engineering.

CUbiC’s research is aimed at developing computer-based assistive and rehabilitative technologies for people with perceptual or cognitive disabilities.

The Note-Taker team, led by Hayden, includes computer science undergraduate Kelley, computer science doctoral student Mike Rush, industrial design graduate student Liqing Zhou, electrical engineering undergraduate Michael Astrauskas, and post-doctoral research associate Gaurav Pradhan.

An Imagine Cup trophy wasn’t on Hayden’s mind when he began the Note-Taker project about two years ago. 

It started when he realized that his visual disability was jeopardizing his pursuit of a bachelor’s degree with a dual major in computer science and mathematics.
 
None of the commercially available assistive technologies allowed Hayden to keep up with classroom note-taking in advanced mathematics coursework, during which instructors often filled more than a dozen whiteboards with theorems and proofs in a mere 45 minutes. Left with little recourse, Hayden began working on a solution.

Multifunctional system

The Note-Taker consists of a portable, custom-designed video camera and a Tablet PC. The camera is able to tilt up and down, and sweep side to side, as well as zoom in on its target. The Tablet PC provides a split-screen display.

One half of the screen has a window that shows live video from the camera, while the other half has a window that is used for handwriting or typing notes.

This dual-window interface allows students with visual impairment to quickly glance back and forth between the live view of the classroom whiteboard and their notes, just like their sighted peers. The video window also allows the user to aim and zoom the camera by simply dragging, tapping or pinching within the video window.

As the development of the proof-of-concept prototype progressed, Black sought support from the National Science Foundation for further development, which brought a grant of about $400,000 over a two-year period.

As the Note-Taker’s development progressed, the team discovered Microsoft had added a category to the Imagine Cup  competition that challenged students to find ways to use Tablet PC technology to make education more accessible.  “It was as if that category was tailored specifically to our project,” Black says.

Overcoming limitations

Their success at the Imagine Cup finals also got Hayden and Kelley an invitation to the recent Microsoft Faculty Research Summit at its headquarters in Redmond, Wash.

At the event’s DemoFest they demonstrated the Note-Taker to hundreds of academics, government officials and Microsoft researchers.

Hayden says that the Note-Taker overcomes the limitations of many assistive technologies which “force students with disabilities to rely on a special classroom infrastructure, or on people who aren’t always available when the student needs assistance.”

The Note-Taker solves the problem by being portable, inexpensive, small enough to fit on a typical classroom desk, and easy to set up. Teachers don’t need to adapt their instructional methods. They are often unaware it is being used, Hayden says.

Work continues on improving the functionality and the aesthetics of the Note-Taker. The team wants to expand its capabilities by providing audio/video recording that allows synchronized playback of lectures, along with the corresponding handwritten or typed notes, so students can review lessons after classes.

Goals for the future

With these and other substantial improvements, the team plans to enter the Imagine Cup competition again next year in the larger and even more competitive Software Development category.

Hayden, who was recently awarded the National Science Foundation’s Graduate Research Fellowship, plans to pursue a Ph.D. in computer science. He says the experience with the Note-Taker has bolstered his commitment to achieving further advances in portable, wearable and prosthetic technologies that help people improve their capabilities in perception, cognition and mobility.

The Note-Taker team is currently seeking assistance from ASU students with visual disabilities who would like to be involved in the continuing development and testing of the system.  If interested, contact John Black at john.black@asu.edu.