{"id":111581,"date":"2019-08-01T15:49:00","date_gmt":"2019-08-01T20:49:00","guid":{"rendered":"https:\/\/beta.pewresearch.org\/pewresearch-org\/decoded\/\/\/interpreting-and-validating-topic-models\/"},"modified":"2024-04-14T04:10:40","modified_gmt":"2024-04-14T09:10:40","slug":"interpreting-and-validating-topic-models","status":"publish","type":"decoded","link":"https:\/\/beta.pewresearch.org\/pewresearch-org\/decoded\/2019\/08\/01\/interpreting-and-validating-topic-models\/","title":{"rendered":"Interpreting and validating topic models"},"content":{"rendered":"\n
\"\"<\/a><\/figure>\n\n\n\n

(Related posts: <\/em>An intro to topic models for text analysis<\/em><\/a>, <\/em>Making sense of topic models<\/em><\/a>, Overcoming the limitations of topic models with a semi-supervised approach<\/em><\/a>, <\/em>How keyword oversampling can help with text analysis<\/em><\/a> and <\/em>Are topic models reliable or useful?<\/em><\/a>)<\/em><\/p>\n\n\n\n

My previous post in this series<\/a> showed how a semi-supervised topic modeling approach can allow researchers to manually refine topic models to produce topics that are cleaner and more interpretable than those produced by completely unsupervised models. The particular algorithm we used was called CorEx, which provides users with the ability to expand particular topics with \u201canchor words\u201d that the model may have missed. Using this semi-supervised approach, we were able to train models on a collection of open-ended Pew Research Center survey responses about sources of meaning in life<\/a> and arrive at a set of topics that seemed to clearly map onto key themes and concepts in our data.<\/p>\n\n\n\n

Next, we faced the task of interpreting the topics and assessing whether our interpretations were valid. Here are four topics that seemed coherent after we improved them using our semi-supervised approach:<\/p>\n\n\n\n

\"\"<\/a><\/figure>\n\n\n\n

Interpreting topics from a model can be more difficult than it may initially seem. Understanding the exact meaning of a set of words requires an intimate understanding of how the words are used in your data and the meaning they\u2019re likely intended to convey in context.<\/p>\n\n\n\n

Topic 37 above appears to be a classic example of an \u201covercooked\u201d topic, consisting of little more than the words \u201chealth,\u201d \u201chealthy,\u201d and a collection of phrases that contain them. At first, we were unsure whether we\u2019d be able to use this topic to measure a useful concept. While we were hoping to use it to identify survey responses that mentioned the concept of health, we suspected that even our best attempts to brainstorm additional anchor words for the topic might still leave relevant terms missing. Depending on how common these missing terms were in our data, the topic model could seriously understate the number of people who mentioned health-related concepts.<\/p>\n\n\n\n

To investigate, we read through a sample of responses and looked for ones that mentioned our desired theme. To our surprise, we realized that this particular \u201covercooked\u201d topic was not a problem in the context of our particular dataset. Of the responses we read (many of which used a wide variety of terms related to health), we found very few that mentioned the theme of health without using a variant of the word itself. In fact, the overwhelming majority of responses that used the term appeared to refer specifically to the theme of being in good health<\/em>. Responses that mentioned health problems or poor health were far less frequent and typically used more specific terms like \u201cmedical condition,\u201d \u201cmedication,\u201d or \u201csurgery.\u201d<\/p>\n\n\n\n

Based on the specific nature of our documents and the context of the survey prompts we used to collect them, we decided that we could not only use this \u201covercooked\u201d topic, but we could also assign it a more specific interpretation \u2014 \u201cbeing in good health\u201d \u2014 than might otherwise have been possible with different data.<\/p>\n\n\n\n

However, this turned out to be a unique case. For other topics, separating out both positive and negative mentions was often impossible.<\/p>\n\n\n\n

For example, the language our survey responses used to describe both financial security and financial difficulties was so diverse and overlapping that we realized we wouldn\u2019t be able to develop separate positive and negative anchor lists and train a model with two separate topics related to finances. Instead, we had to group all of our money-related anchor words together into Topic 44, which we could only interpret as being about money or finances in general. We manually coded a sample of these responses and found that 77% mentioned money in a positive light, compared with 23% that brought it up in a neutral or negative manner. But even our manually-refined semi-supervised topic model couldn\u2019t be used to tell the difference.<\/p>\n\n\n\n

Clearly, context matters when using unsupervised (or even semi-supervised) methods. Depending on how they\u2019re used, words found in one set of survey responses can mean something entirely different in another set, and interpretations assigned to topics from a model trained on one set of data may not be transferable to another. Since algorithms like topic models don\u2019t understand the context of our documents \u2014 including how they were collected and what they mean \u2014 it falls on researchers to adjust how we interpret the output based on our own nuanced understanding of the language used.<\/p>\n\n\n\n

Between the two semi-supervised CorEx topic models that we trained, we identified 92 potentially interesting and interpretable topics. To test our ability to interpret them, we gave each one a short description, including some additional caveats based on what we knew about the context of each topic\u2019s words in our corpus:<\/p>\n\n\n\n

\"\"<\/a><\/figure>\n\n\n\n

For each topic, we first drew a small exploratory sample consisting of some responses that contained the topic\u2019s top words and others that didn\u2019t. A member of the research team then coded each response based on whether or not it matched the label we had given the topic. After coding all of the samples, we used Cohen\u2019s Kappa<\/a>, a common measure of inter-rater reliability, to test how well the topic models agreed with the descriptions that we had given the topics.<\/p>\n\n\n\n

Some topics resulted in particularly poor agreement between the model and our own interpretation, often because the words in the topic were used in so many different contexts that its definition would have to be expanded to the point that it would no longer be meaningful or useful for analysis.<\/p>\n\n\n\n

For example, one of the topics we had to abandon dealt with opinions about politics, society and the state of the world. Some of the words in this topic were straightforward: \u201cpolitics,\u201d \u201cgovernment,\u201d \u201cnews,\u201d \u201cmedia,\u201d etc. While there were a handful of false positives for these words \u2014 responses in which someone wrote about their career in government<\/em>, or recently receiving good news<\/em> \u2014 the vast majority of responses that mentioned these words contained opinions about the state of the world, in line with our interpretation. But a single word, \u201cworld\u201d itself wound up posing a critical problem that forced us to give up on refining this topic.<\/p>\n\n\n\n

In our particular dataset, there were a lot of respondents that opined about the state of the world using only general references to \u201cthe world\u201d \u2014 but there were also many respondents that used \u201cworld\u201d in a non-political, personal context, such as describing how they wanted to \u201cmake the world a better place\u201d or \u201ctravel the world.\u201d As a result, including \u201cworld\u201d as an anchor term for this topic produced numerous false positives, but excluding it produced many false negatives. Either way, our topic model would either be overstating or understating the topic\u2019s prevalence to an unacceptable degree.<\/p>\n\n\n\n

In this case, we were able to narrow our list of anchor terms to focus on the more specific concept of politics, but other topics presented similar challenges and we were forced to set some of them aside. As we continued reviewing our exploratory samples, we also noticed that some topics that initially seemed interesting \u2014 like \u201cspending time doing something\u201d and \u201cthinking about the future\u201d \u2014 turned out to be too abstract to be analytically useful, so we set these aside, too.<\/p>\n\n\n\n

From our initial 92 topics, we were left with 31 that seemed analytically interesting, could be given a clear and specific definition, and had encouraging levels of initial reliability, at least based on our non-random exploratory samples. Drawing on the insights we\u2019d gained from viewing these topics in context, we refined them further and added or removed words from our anchor lists where it seemed useful.<\/p>\n\n\n\n

For our final selection of topics, we drew new random samples of 100 documents for each topic, this time to be coded by two different researchers to determine whether our tentative topic labels were defined coherently enough to be understood and replicated by humans.<\/p>\n\n\n\n

Unfortunately, we found that seven of these 31 topics resulted in unacceptable inter-rater reliability. Though they had seemed clear on paper, our labels turned out to be too vague or confusing and we couldn\u2019t consistently agree on which responses mentioned the topics and which did not. Fortunately, we had acceptable rates of agreement on the remaining 24 topics \u2014 but for a few of the rarer ones, that wasn\u2019t good enough. In an upcoming post, I\u2019ll explain how we used a method called keyword oversampling to salvage these topics.<\/p>\n","protected":false},"excerpt":{"rendered":"

Understanding the exact meaning of a set of words requires an intimate understanding of how the words are used in your data and the meaning they\u2019re likely intended to convey in context.<\/p>\n","protected":false},"author":655,"featured_media":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"sub_headline":"","sub_title":"","_prc_public_revisions":[],"_ppp_expiration_hours":0,"_ppp_enabled":false,"ai_generated_summary":"","relatedPosts":[],"datacite_doi":"","datacite_doi_citation":"","_prc_seo_qr_attachment_id":0,"spoken_article_player_enabled":true,"displayBylines":true,"footnotes":"","prc_watchers":[],"_prc_fork_parent":0,"_prc_fork_status":"","_prc_active_fork":0},"categories":[353],"bylines":[927],"collection":[],"_post_visibility":[],"decoded-category":[530],"formats":[],"_fund_pool":[],"languages":[],"regions-countries":[],"research-teams":[524],"workflow-status":[],"class_list":["post-111581","decoded","type-decoded","status-publish","hentry","category-data-science","bylines-patrick-van-kessel","decoded-category-data-science","research-teams-decoded"],"label":"Decoded","post_parent":0,"word_count":1433,"canonical_url":"https:\/\/beta.pewresearch.org\/pewresearch-org\/decoded\/2019\/08\/01\/interpreting-and-validating-topic-models\/","art_direction":{"A1":{"id":125979,"rawUrl":"https:\/\/beta.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/08.01.2019_feature.png","url":"https:\/\/beta.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/08.01.2019_feature.png?w=564&h=317&crop=1","width":564,"height":317,"caption":"","chartArt":false},"A2":{"id":125979,"rawUrl":"https:\/\/beta.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/08.01.2019_feature.png","url":"https:\/\/beta.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/08.01.2019_feature.png?w=268&h=151&crop=1","width":268,"height":151,"caption":"","chartArt":false},"A3":{"id":125979,"rawUrl":"https:\/\/beta.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/08.01.2019_feature.png","url":"https:\/\/beta.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/08.01.2019_feature.png?w=194&h=110&crop=1","width":194,"height":110,"caption":"","chartArt":false},"A4":{"id":125979,"rawUrl":"https:\/\/beta.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/08.01.2019_feature.png","url":"https:\/\/beta.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/08.01.2019_feature.png?w=268&h=151&crop=1","width":268,"height":151,"caption":"","chartArt":false},"XL":{"id":125979,"rawUrl":"https:\/\/beta.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/08.01.2019_feature.png","url":"https:\/\/beta.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/08.01.2019_feature.png?w=720&h=405&crop=1","width":720,"height":405,"caption":"","chartArt":false},"social":{"id":125979,"rawUrl":"https:\/\/beta.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/08.01.2019_feature.png","url":"https:\/\/beta.pewresearch.org\/pewresearch-org\/wp-content\/uploads\/sites\/20\/2022\/08\/08.01.2019_feature.png?w=1200&h=628&crop=1","width":1200,"height":628,"caption":"","chartArt":false}},"_embeds":[],"watchers":[],"table_of_contents":[],"datacite_doi":"","prc_seo_data":{"title":"Interpreting and validating topic models","description":"Understanding the exact meaning of a set of words requires an intimate understanding of how the words are used in your data and the meaning they\u2019re likely intended to convey in context.","og_title":"Interpreting and validating topic models","og_description":"Understanding the exact meaning of a set of words requires an intimate understanding of how the words are used in your data and the meaning they\u2019re likely intended to convey in context.","schema_type":"Article","noindex":false,"canonical_url":"","primary_terms":{"category":1},"custom_schema":[],"og_image":125979,"indexnow_submitted_at":null,"gsc_index_status":null},"prepublish_checks":{},"jetpack_sharing_enabled":true,"relatedPostsOrdered":[],"bylinesOrdered":[{"key":"_3lp5q8to2","termId":927}],"acknowledgementsOrdered":[],"_links":{"self":[{"href":"https:\/\/beta.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/decoded\/111581","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/beta.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/decoded"}],"about":[{"href":"https:\/\/beta.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/types\/decoded"}],"author":[{"embeddable":true,"href":"https:\/\/beta.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/users\/655"}],"replies":[{"embeddable":true,"href":"https:\/\/beta.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/comments?post=111581"}],"version-history":[{"count":2,"href":"https:\/\/beta.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/decoded\/111581\/revisions"}],"predecessor-version":[{"id":138538,"href":"https:\/\/beta.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/decoded\/111581\/revisions\/138538"}],"wp:attachment":[{"href":"https:\/\/beta.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/media?parent=111581"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/beta.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/categories?post=111581"},{"taxonomy":"bylines","embeddable":true,"href":"https:\/\/beta.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/bylines?post=111581"},{"taxonomy":"collection","embeddable":true,"href":"https:\/\/beta.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/collection?post=111581"},{"taxonomy":"_post_visibility","embeddable":true,"href":"https:\/\/beta.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/_post_visibility?post=111581"},{"taxonomy":"decoded-category","embeddable":true,"href":"https:\/\/beta.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/decoded-category?post=111581"},{"taxonomy":"formats","embeddable":true,"href":"https:\/\/beta.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/formats?post=111581"},{"taxonomy":"_fund_pool","embeddable":true,"href":"https:\/\/beta.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/_fund_pool?post=111581"},{"taxonomy":"languages","embeddable":true,"href":"https:\/\/beta.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/languages?post=111581"},{"taxonomy":"regions-countries","embeddable":true,"href":"https:\/\/beta.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/regions-countries?post=111581"},{"taxonomy":"research-teams","embeddable":true,"href":"https:\/\/beta.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/research-teams?post=111581"},{"taxonomy":"workflow-status","embeddable":true,"href":"https:\/\/beta.pewresearch.org\/pewresearch-org\/wp-json\/wp\/v2\/workflow-status?post=111581"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}