Overview
Highlights are specific to Ask Opinion questions and identify the “spread of opinion” or a selection of thoughts in which each participant is most likely to find a thought they strongly agree with.
Included in this Article
How it Works
Highlights is specific to our Ask Opinion and Branching questions. In these questions, participants submit their response in their own words then vote on other participants responses anonymously. We compute Highlights using the voting activities the participants complete after submitting their own responses.
- Agree/Disagree Voting Activity: The agree/disagree exercise presents the participant with a single response that was submitted by someone in the audience and prompts them to select if they agree or disagree with that response. This exercise gives us an absolute baseline of what a participant does or doesn’t agree with.
- Binary Choice Voting Activity: The binary choice exercise presents the participant with two responses that were submitted by individuals in the group and prompts them to select the response they prefer more. This exercise provides a relative signal of agreement or disagreement between the responses which assists in a relative ranking of responses.
The intention of this algorithm is to bring opinions that have significant factions of support to the foreground. This has advantages overtaking the top items of a ranked list, in which polarizing responses get buried in the middle. To find this set of responses, we use an iterative algorithm that maximizes the probability for each participant to find a thought they strongly agree within the remaining set.
Location
Highlights is located in the Live portion of the platform and can be found by navigating to the question.
Example
The usefulness of this approach can be demonstrated by examining a common, everyday scenario: Ordering food for a party. Imagine you have 10 guests on the guest list and you need to decide what food will most satisfy your guests.
Here are the food options:
- Spicy Thai Curry — 5 guests love this option but the other 5 can’t stand spicy food
- Sushi — Another 5 guests also love this option but the other 5 won’t eat raw fish
- Cheese Pizza — Mostly everyone is okay with this but not as their 1st choice
For the sake of the example, assume you are a sushi person.
If you were to order 1 single item for the whole group, your only option is cheese pizza. This is because the average agreement for sushi is 55%, Thai curry is 45% but cheese pizza is 100% agreeable, even though it’s not anyone’s favorite.
So a list ranked by Percent Agree (the other metric associated with Ask Opinion and Branching questions) would be:
- Cheese Pizza 100%
- Sushi 55%
- Spicy Thai Curry 45%
but the Highlights view would be:
- Sushi 6
- Spicy Thai Curry 5
You can see that ordering the top 2 items using the ranked list would deprive all 5 guests that love Spicy Thai Curry of their favorite item, forcing them to eat pizza. Highlights aim to remove this problem by directly providing the options that would be most agreed with by the most guests, thus, showing the spread of opinion.
How do we select the thoughts for Highlights?
To understand this, it’s important to first understand what we do with participant data in order to calculate the Percent Agree scores as this data is used to calculate Highlights.
Highlights is a utility maximization algorithm. Each participant is assigned to their highest utility response. We then remove the response with the lowest percent agree score from the set. Any participants whose highest utility thought was the one removed, are assigned to their next highest utility thought.
We continue iterating in this manner until we are left with the set of thoughts (typically 10) that have the highest utility, where every participant strongly agrees with at least one thought in the set.