MIT Media Laboratory: The Human Speechome Project

Stepping into a Child’s Shoes

Dr. Deb Roy

When Deb Roy and his wife, Rupal Patel, learned of their impending bundle of joy, they did what many first-time parents do: They got a video camera. Actually, they bought 11 video cameras and 14 state-of-the-art microphones. Then they built a temperature-controlled data-storage room in their basement and loaded it with, among other gear, five Apple Xserves and a 4.4TB Xserve RAID, backup tape drives, and robotic tape changers. No, Roy and Patel hadn’t instantly become the world’s most doting parents; instead, they had hatched a plan to record practically every waking moment of their son’s first three years.

The high-powered academic couple—he directs of the Cognitive Machines Group at the Massachusetts Institute of Technology (MIT) Media Lab, and she directs the Communication Analysis and Design Laboratory at Northeastern University—scrambled to convert their suburban Boston home into a state-of-the-art research center that would host the most ambitious study ever conducted on how children acquire language. They named the linguistic data-mining odyssey the Human Speechome Project (HSP), a marriage between “Speech” and “Home.” The name also pays appropriate homage to the grand scale and scientific payoff of the Human Genome Project, which catalogued the complete genetic makeup of the human species. Following on the heels of the Human Genome Project, the HSP was recently singled out by the journal Science as an example of pioneering research. In addition to their roles as primary investigators in the study, Roy and Patel are, along with their now two-year-old son, the central research subjects.

“My ultimate goal is to understand how language works,” Roy explains. That’s a tall order, and the logical place to start, he maintains, is with children. Decades of inquiry involving video and audio recordings of children interacting with caregivers and psychologists in institutional “speech labs” have laid a foundation to begin answering questions about how children develop language skills. The day-in/day-out interactions between children and adults, Roy points out, are key to the way children grasp language. “But for all of the interest in how children learn language, there’s no comprehensive data of even a single child’s development,” Roy says. “Most researchers rely on speech recordings that cover less than 1.5 percent of a child’s complete linguistic experience.”

This visualization captures the interaction between caregiver and child.

And that simply isn’t a dense or broad enough data set to answer the kinds of deep questions that Roy thinks are necessary to uncover the steady process of language acquisition. Truly understanding how human beings acquire language requires “stepping into a child’s shoes.”

So, from the moment he arrived home from the hospital, Roy and Patel’s son has lived under the almost constant observation of the 14 microphones and 11 video cameras that are embedded in the ceiling over every major room of the house. “Somewhere around 80 percent of his waking hours at home are being recorded,” says Roy. For the other 20 percent, privacy considerations permit mom, dad and other caregivers to turn off the cameras or microphones using wall-mounted touch panels in each room. Roy also equipped each controller with an emergency “oops” switch, marked with a giant exclamation point, to erase any particularly embarrassing family moments.

“Truly understanding how human beings acquire language requires “stepping into the shoes of a child.”

As head of the Cognitive Machines Group, one of Roy’s motivations in understanding how children learn language is to learn how to give robots the gift of gab. His MIT lab already hosts a menagerie of robots with eerily human thinking and speaking abilities, if not looks. One prototype named Ripley looks something like the working end of an arcade “grab the prize out of the bin” game. When presented with objects scattered on a tabletop, Ripley obediently watches his graduate student caretaker and listens to his instructions to “pick up the green one” or “touch the one on my left.” Like a toddler, Ripley sometimes gets confused with the words and asks politely for clarification. The obvious technological benefits aside, Roy’s insists that his primary goal in building talking robots is to validate the principles he learns from observing and analyzing the way his son learns language. “To (best) understand how something works,” he says, “you need to build it.”

1 2 3 Next Page >