The team has developed a system that searches videos for specific types of objects and activities without any special labeling or annotations. “Think of it like Googling for video,” Siskind says.
Searching 10 Hollywood movies in the genre of Westerns, his team was able to find a variety of horse-related images and video segments.
“Our system has an object detector for people and for horses, and it has semantic definitions of concepts like riding a horse, leading a horse, approaching a horse,” Siskind says. “It has concepts like toward and away from, leftward and rightward, and quickly and slowly.”