This section describes the online coding process used to code the items preselected by the topic modeling. The goal of this step was to identify a certain amount of thematically relevant items. The graphic below gives an overview of the different steps:
In the coding process a human coder was presented with a preselected item. This coder assigned a relevance score using a 4-point scale (not relevant, uncertain not relevant, uncertain relevant, relevant) to this item. Items that were rated as thematically irrelevant were discarded. Items that were coded as either “uncertain not relevant” “uncertain relevant”. or relevant were subjected to a second coding decision: Here, another coder could rate the item as thematically irrelevant or relevant. If the coding decision of the second coder was consistent with the decision of the first coder the item was passed on to the pool of relevant items. If the coders disagreed in their coding decision the item was subjected to a third coder who served as a tiebreaker.
To facilitate this complex online coding process we designed, developed and implemented an online coding tool called the MedCon Relevance Coder (MRC).