1. Visual Question Answering Aaron Honculada, Aisha Urooj Khan, Dr
Visual Question Answering Aaron Honculada, Aisha Urooj Khan, Dr. Niels Lobo, Dr. Mubarak Shah
Results
Background
Architectures
Visual Question Answer (VQA) is at the intersection of Computer Vision and Natural Language Processing. Given an image or video and questions, VQA aims to design systems capable of answering questions about or related to the visual information.
Baseline Model
TVQA Prediction
A1) Soft drink
Our Prediction
A4) Beer
What was Raj drinking when he was sitting at the table with Leonard and Howard ?
A0) Lemonade
A2) Water
A3) Milk
Dataset
TVQA Prediction
A1) Chandler was eating a cheeseburger .
Our Prediction
A4) Chandler was smoking a cigarette.
What was Chandler doing when Phoebe turned around on her stool ?
A0) Chandler was looking at stars .
A2) Chandler was making a cake.
A3) Chandler was doing jumping jacks.
TVQA Dataset:
6 TV Series
460 hours of video
21,793 clips
152,545 Question and Answer Pairs
7 Questions per clip
5 Multiple choice answers
5 Question Families
Included
Subtitles
Video Features
Visual Concepts
Attention Based Model
TVQA Prediction
A0) 300
Our Prediction
What number did House write on the board after Taub suggested a MRA ?
A1) 10
A2) 1
A3) 60
A4) 12
Results
Model
Feature
Accuracy
LSTM
N/A
42.48%
VCPT
42.67%
CNN+LSTM
ResNet (FC)
42.73%
I3D (FC)
42.86%
I3D (Conv)
42.59%
Attention
TVQA
43.78%
TVQA Prediction
A3) Purple
Our Prediction
A2) red
What color is the shirt Castle is wearing when he says, “Ashley , I'm sorry.”?
A0) black
A1) green
A4) yellow
Acknowledgement
We would like to thank the National Science Foundation for funding this project. NSF IIS