Overview
Percent agree is an estimate of the percentage of participants that would agree with a particular response in an Ask Opinion or Branching question.
Included in this Article
How it Works
Percent Agree is specific to our Ask Opinion and Branching questions. In these questions, participants submit their response in their own words then vote on other participants responses anonymously. We compute Percent Agree scores using a machine learning model called collaborative filtering, which uses the voting activities the participants complete after submitting their own responses.
- Agree/Disagree Voting Activity: The agree/disagree exercise presents the participant with a single response that was submitted by someone in the audience and prompts them to select if they agree or disagree with that response. This exercise gives us an absolute baseline of what a participant does or doesn’t agree with.
- Binary Choice Voting Activity: The binary choice exercise presents the participant with two responses that were submitted by individuals in the group and prompts them to select the response they prefer more. This exercise provides a relative signal of agreement or disagreement between the responses which assists in a relative ranking of responses.
When a participant indicates that they agree with a response, we give the response a utility score for that participant that is greater than zero. When they disagree, we assign a score less than zero for that participant.
During binary voting, as participants indicate which responses they prefer over other responses, we move their agreement scores higher or lower based on which they choose. We’ve implemented a machine learning model into the platform called collaborative filtering which uses inference to fill in utility scores for all thoughts for all participants. Given that we can have hundreds of participants online at a time, it is impossible to collect voting data for each participant on all responses during the voting period. This is where collaborative filtering comes into play to rank and structure the responses – the same model adopted by Netflix or Amazon to make predictions for consumers.
To generate the percent agreement scores, we count how many participants had a utility score > 0 for each response and divide that by the total number of participants that answered the question. This gives us an estimate of the percent of the group that agreed with that particular response.
Locations and Best Practices
You can find Percent Agree in a number of places throughout the platform, you can find a few of the most commonly used below. Percent Agree is available for Ask Opinion and Branching questions.
Remesh Live
Percent Agree appears once a question has stopped collecting responses and will appear to the right of each response. You can also view Percent Agree by clicking on Analyze beneath the question. This will allow you to view Percent Agree scores cut by Segment.
- Best Practices: When conducting a Live Remesh Conversation, use the Percent Agree scores to quickly note the most agreed upon thoughts and use this information to craft follow up questions as needed.
Post Conversation Analysis: Results
When selecting an Ask Opinion or Branching question in Results, you can see the Percent Agree score associated with each response. You can compare Percent Agree scores across Segments by clicking the "Compare" drop down above the responses.
- Best Practices: Compare Percent Agree scores to identify areas in which there are consensus or disagreement between key demographics.
Auto Code
Auto Coded responses will also include the associated Percent Agree score to the right of the response.
- Best Practices: Sort by highest and lowest agreement to see what themes are showing up most frequently in most frequently in most and least agreed upon responses.
Frequently Asked Questions
- How long do participants have to submit their response in an Ask Opinion or Branching Question?
In Remesh Live participants have the first 80% of the Ask Opinion time to respond. If they don't submit their response in that time, their responses will be automatically submitted then they will move into voting exercises. Flex is self-paced, so participants will submit their response at their own pace. - Where can I read more about how this works?
We recommend taking a look at this document to learn more.