Doctoral graduate pursues new technologies for visually impaired

Computer technologies are continually evolving to meet social, educational and health needs. When Shayok Chakraborty, working at IBM India, discovered ASU’s Center for Cognitive Ubiquitous Computing (CUbiC), he knew he had to get his doctorate here.

“CUbiC is doing cutting-edge research in the fields of machine learning, pattern recognition and computer vision,” he says. “The lab is developing applications that can empower physically challenged individuals with technologies that enrich their lives.”

“Study has shown that about 65 percent of the information during social interaction is conveyed through non-verbal cues, like eye-gaze and body mannerisms,” says Chakraborty. “So individuals who are visually challenged face fundamental limitations in their everyday interactions.”

What if blind individuals could “see” the facial expressions and mannerisms of a sighted person with whom they are conversing? It would lead to far more social interaction and less isolation for the visually impaired, says Chakraborty.

After joining ASU in 2007, Chakraborty worked with researchers at CUbiC to develop a Social Interaction Assistant system to enrich the interaction experience of visually challenged individuals with their sighted peers. The system consists of a camera mounted on the nose-bridge of a pair of glasses, which the blind user wears. The incoming video stream, analyzed in a handheld device like a PDA, provides valuable information to the user about a sighted partner, such as knowing whether they are smiling or frowning, nodding or using hand gestures.

Developing such a system poses significant challenges, including the staggering number of images that must be analyzed and processed. Machine learning algorithms developed by Chakraborty, called batch mode active learning techniques, efficiently analyze large amounts of video data, substantially reducing the human hours of work.

The active learning techniques also have very practical and cost-saving uses in medicine.

“Active learning can be judiciously used to select the most promising and informative tests that need to be performed,” he says. “For instance, to determine whether a patient has heart disease or not, active learning can identify the most useful medical tests to be performed for effective diagnosis.”

Chakraborty’s work on active learning techniques to analyze video data has led to three U.S. patents as well as national recognition in his field. He has published more than 15 peer-reviewed papers in preeminent conferences and journals, as well as two book chapters.

Prestigious venues, including the CVPR (Computer Vision and Pattern Recognition), NIPS (Neural Information Processing Systems), ACM-MM (ACM Multimedia Conference ) and AAAI (Association for Advancement of Artificial Intelligence)conferences, the Multimedia and Vision Meeting organized by IBM Research, the Machine Learning (ML) Symposium organized by the New York Academy of Sciences and the ML departments of Duke University and Carnegie Mellon University invited him to present his research. He was also selected for a research internship in the ML department at Microsoft Research, Redmond.

At ASU, in addition to serving as a Teaching Associate for several courses on computer science and programming, he also taught “Mathematical Foundations of Informatics” in the Computer Science department.

Chakraborty recently earned his doctorate in computer science with the School of Computing, Informatics and Decision Systems Engineering (SCIDSE), part of Ira A. Fulton Schools of Engineering, and his dissertation was nominated for the best thesis award in the department.

A native of India, Chakraborty earned his bachelor's degree in computer science and engineering from Jadavpur University in Kolkata, India. Although he was accepted at three other universities for his graduate studies, including Boston University, Illinois Institute of Technology (IIT), and University of California, Irvine (UCI), he chose ASU to join the research in the CUbiC lab.

Presently, he is seeking an opportunity in a renowned research lab such as Microsoft or IBM to continue his research into developing new technologies which can make a positive difference in society.

Discover more about Chakraborty’s research and interests on his website at: sites.google.com/a/asu.edu/shayok-chakraborty.