To study how human evaluates the quality of machine-generated evidence in Multimedia Event Detection, I built this website for an effective survey.
Flask web framework is chosen for the back end. The reasons are simple: It is Python-based, super light-weight, and flexible. The content of the pages is generated dynamically in nature.
Generally, a user is presented by a multimedia event topic with detailed descriptions. Then the user needs to answer if a given video belongs to the topic as quickly as possible. Instead of allowing to play the whole video, the user is only presented by the "evidence" generated by algorithms that are devised to extract the segments of interest in the video. Apparently, to answer the question, the user needs to make predictions with the given evidence, thus we know if certain evidence is good enough.
We have enforced the following constraints in the survey design:
-
There are five types of evidence which are either images or video snippets (3 seconds each). For each video, a user is only presented by one of the types. No other types from the same video can be shown to the same user. This avoids the user memorizing a made decision.
-
All types of evidence of a video are ensured to be evaluated by different users.
-
The order of both topics and videos are random. For a single video, the type of evidence shown to a user is random too.
-
We randomly select a portion of the questions and distribute them to all the users. With this overlap, we are able to evaluate the validity of a user's answer or any of the personal bias.
-
A few sample answers of some marginal cases are designed to align the criteria and minimize personal bias. They are shown to a user before the start of the actual questions in each topic.
-
A user's answer (yes /no /not sure) along with the response time are recorded.
-
A user can leave at any moment and come back to continue the survey. In this case, he needs to read the topic description again at the question he left with.
-
A user can't go back to the question he already answered or override the answer in any way.
As the dataset is copyrighted. All the question media concerning the videos is removed. The code is shared for reference, not for reuse.