In the coding process a human coder was presented with a preselected item. This coder assigned a relevance score using a 4-point scale (not relevant, uncertain not relevant, uncertain relevant, relevant) to this item. Items that were rated as thematically irrelevant were discarded. Items that were coded as either “uncertain not relevant” “uncertain relevant”. or relevant were subjected to a second coding decision: Here, another coder could rate the item as thematically irrelevant or relevant. If the coding decision of the second coder was consistent with the decision of the first coder the item was passed on to the pool of relevant items. If the coders disagreed in their coding decision the item was subjected to a third coder who served as a tiebreaker.

To facilitate this complex online coding process we designed, developed and implemented an online coding tool called the MedCon Relevance Coder (MRC).