Prior to October 11, 2021, reports included responses only from completed surveys. This allowed us to avoid including meaningless data, e.g., when the same individual accessed the survey several times and only completed the survey once. However, there are cases when an individual’s only response may be a partial one. Given what research tells us about selection bias, this is likely to exacerbate an existing problem where the individuals we most want to hear from are the least likely to finish the survey. With our October 11, 2021 update, reports include partial responses in those cases, so that participants are not entirely discarded from the data simply for not making it to the last page of the survey.
What might users who collected data prior to October 11, 2021 notice as a result of this change?
Users who compare reports generated on or before October 4, 2021 to reports generated on or after October 11, 2021 may notice the response rates in their reports increasing, by more than would be expected given the amount of data they’ve collected. This is because participants who only gave partial responses are now being included in the sample. Scores may shift as well for the same reason.