Auto Code: Automatically tag and organize themes from open-ended responses

Overview

Auto Code leverages a large language model (LLM) from OpenAI, to automatically tag or code your open-ended responses or upload your own set of tags. Tags are organized by category, allowing you to quickly visualize your data. See the big picture, or filter by tags to dig into deeper analysis.

LLM technology is used to power a few of Remesh's most powerful features, and does require an extra step to enable. Read more about this suite of features and how to enable them here.

Included in this article:

1. Where to Find AutoCode
2. Automated Tags and Tag Categories
3. Import Your Own Tags
4. Filtering Responses
5. Customizing Tags and Tag Categories
6. Downloading Data (CSV)
7. Getting a Sense of Spread
8. Frequently Asked Questions

Where to find AutoCode

1. From your Results page, click the Auto Code (BETA) tab.
   
   
   
2. Choose a question to code from the drop-down menu. 
   

Automated Tags and Tag Categories

Once you select the question you would like to analyze, tags and tag categories will populate.

On the left side of the screen, you’ll see Tags organized by Tag Category. You can collapse these categories to get a high-level view of your data. On the right side of the screen, you’ll see all responses (along with the % Agree scores for Ask and Branch Opinion questions and Sentiment labels). Each response will have about 1-5 tags that describe the content.

The “Uncategorized” category houses tags that don’t fit into any other category group.

Import Your Own Tags

Alternatively, you can import your own tags using the following steps.

• Click “Run Custom Tags” then “Start Over with a Custom List of Tags” 
  

• You will then be prompted to input your tags. Your list of tags should have one tag per line, and should not include commas, quotes or ordered numbers. We recommend adding a tag for “other” or “none” to catch any responses that may not fit into the tags you’ve outlined. 
  
• You can then choose if you would like this list of tags to be applied to all open-end questions. Then click "Next" when you’re ready to assign tags to your responses. You will have to wait for a brief period of time as the system assigns your tags.
• Once your tags have been assigned, they will all appear under the category “Uncategorized.” You can make adjustments and add additional categories, see Customizing Tags and Categories.

Filtering Responses

To filter responses by tag(s), select one or more tags by selecting tags under the category drop down menus on the left, or selecting tags directly under individual responses.

To filter responses by demographic segment click the Segment filter drop-down menu, located below the question near the top of the screen then select your desired Segment. Learn more about Segments and how to create them here.

Customizing Tags and Categories

As you are reviewing responses and assigned codes, you can make adjustments to many of the elements if you see other themes arise.

Adding a New Tag or Category

• Click Add Tag or Add Category, located at the top left of the Auto Code screen
• Type the name of the tag/category you’d like to create
• New tags will appear as “New Tag” with the option to edit the name and be assigned to the “Uncategorized” Category by default. You can move the tag to any category by following the directions below.
• New categories will appear as “New Category” with the option to edit the name.

Editing or Combining Tag or Category

• To edit, select the three dots next to the tag/category, then click edit.
• To combine, select the three dots next to the tag/category, choose "combine with" then select from the list of categories you would like to combine.

Deleting or Moving Tag or Category

• To delete a tag or category click the three dots next to the tag/category you’d like to delete then choose delete.
• To move a tag or category, click the three dots next to the tag you’d like to move then click "move to" and select the category you'd like to move it to from the drop down.

You can assign and unassign responses to tags, whether they were assigned manually by you or automatically by Auto Code.

• To assign responses to tags, select the responses you would like to assign then click the arrow to the right of the tag you'd like to assign responses to.
• To unassign responses, click the X next to the tag under the response that you would like to remove.

Downloading Data (CSV)

To download this data (CSV), click Export Codes. You’ll receive two CSV files:

• One file shows all the responses and the tags associated with each of them
• The second file shows the tags and their organization into categories

Getting a Sense of Spread

As you filter in specific tags and tag categories, you can always get a sense of which responses relative to the total are currently in view. 

Auto Code displays two cues to give you a sense of spread.

1. Response, Tag & Tag Category Count: Above the responses (right side of the screen), you’ll see some metadata that gives you a more quantifiable sense of spread for what’s filtered into view, for example:
   • 34 of 100 Responses, across 5 of 15 Tags, within 3 of 6 Tag Categories
     
2. Tag Category Dot Cues: Next to the categories (left side of the screen), you’ll see partially filled-in dot icons if some tags within the category have been filtered in; you’ll see a fully filled-in dot if all tags within the category have been filtered in. This helps you visually get a sense of which tag categories are currently being viewed when they are collapsed.

Frequently Asked Questions

• Why can’t I access the Auto Code (BETA) feature?
  In order to use this feature, a Workspace Admin must grant access by navigating to the Data Privacy Page in Workspace Settings. Read more about this here.
• What’s the difference between Auto Code, Common Topics, and Thematic Clusters?
  While Auto Code, Common Topics, and Thematic Clusters all categorize responses, they are implemented differently.
  • Auto Code is the only one that leverages Generative AI. It’s a descriptive and flexible way to categorize responses that are designed to enhance efficiencies and fit into your existing research process. Tags and Tag Categories will also be editable with a future update.
  • Common Topics surfaces important keywords that frequently appear. It identifies themes by leveraging the similarity of single keywords or short phrases.
  • Thematic Clusters organize responses into system-defined groups. It identifies themes by leveraging the similarity of full responses. You can compare differences, if any, between theme group responses and the demographics of the participants who supplied them.
• Can I search responses?
  You can use the search box on the top right of the screen to directly search for keywords.
• Does Auto Code work for Merged Conversation Results?
  Yes, you can use Auto Code for your Merged Conversation Results!
• How are tags and tag categories organized?
  Tag Categories are organized alphabetically, with the “Uncategorized” category at the very bottom. When you expand a category, you’ll see tags associated to that category organized by frequency; the tag that has the most responses will be listed first.
• How can I check the status of my tags?
  You can check the status of a tag or category in two places:
  
  • Next to the question in the top drop-down menu
  • In the left pane once you’ve selected a question
  You’ll see one of five different status indicators:
  • Not Yet Requested: A request has not yet been made to process this question; indicated by a clock icon
  • Queued: The question is going to be processed but hasn’t yet started
  • In Progress: The question is currently being coded
  • Completed: Tags and categories are available to view; indicated by a green check icon. Once complete all tags and categories will be visible.
  • Retry - there was an issue processing the question, click on this question & status to retry
• Can you tell me more about how this feature works?
  At Remesh, our R&D team found that a combination of individual response processing with GPT in tandem with clustering models and further GPT refinement produced the best results. This ensures that categories found with a single tag are included in larger categories, and that those larger categories found encompassing many tag ideas are divided into more focused categories.
• What conversations will Auto Code work on?
  Conversations run on Jan 1, 2023 and beyond.