What is Observer Bias in Research

Observer bias, a form of detection bias, manifests in observational studies when the individuals conducting the research subtly influence the results based on their own expectations and beliefs. It is a type of cognitive bias that can affect the objectivity of an observation, leading to inaccuracies that may invalidate the research.

Observation is crucial for scientific study and activity, and as such, observer bias may be involved. When such biases occur, scientific research might result in an overestimation or underestimate of what is true and correct, threatening the validity of the study’s findings and results, even if all other designs and procedures were adequate.

Observer bias is more likely when the investigator or researcher has a vested interest in the result of the study or has strong assumptions. Together with confusing underlying data and a subjective grading procedure, these three characteristics have a significant impact on the incidence of observer bias.

Cognitive biases that are common in observers include:

Effects on Study Quality

Researchers aim for objectivity, but when observer bias occurs, it can lead to experimenter bias, where expectations influence outcomes. In studies involving human judgments, such as observational research, this can lead to skewed data. Effects on study quality include misclassification of outcomes and distorted effect size, attributing false causality where none exists.

In behavioral science, psychologists must be cautious of confirmation bias — searching for or interpreting information in a way that confirms one’s preconceptions. This was evidenced in a review by Richard Spano highlighting observer bias in police observational data, where researchers noted how pre-set themes could direct observers’ attention, potentially influencing recorded behaviors and the data collected.

In medicine, the diagnostic process can be subjective, relying on a clinician’s interpretation of clinical signs. A study might exhibit observer bias if a clinician’s expectations affect their diagnosis or treatment efficacy assessment. One analysis of observer ratings found evidence of bias across various medical assessments.

Case Studies and Historical Examples

One of the most well-known examples of observer bias can be found in the publications and contributions of Cyril Burt, an English psychologist and geneticist who claimed IQ heredity. Burt felt, and thus established through his research due to observer bias, that children from low-income homes were more likely to have lower levels of cognitive capacities than children from high-income families.

Throughout the 1960s, this body of work had a significant impact on England’s educational system, with middle- and upper-class children attending top schools and children from poorer socioeconomic backgrounds attending schools with less desirable characteristics. Following Burt’s death, more study revealed that the data in Burt’s experiments was faked, which was assumed to be due to his observer bias and the conclusions he hoped to discover through his studies.

The Hawthorne Effect

The Hawthorne Effect is a classic example where the mere attention to study subjects altered their behavior. This phenomenon was named after a series of studies conducted at the Hawthorne Works of Western Electric in the 1920s and 1930s, which found that workers’ productivity seemed to improve when changes were made to their work environment, but also when those changes were later removed.

The key takeaway is that subjects may change their behavior simply because they are being observed, thus potentially skewing study results. However, the social psychologist Richard Nisbett has referred to the Hawthorne effect as “a glorified anecdote,” saying that “once you have got the anecdote, you can throw away the data.” There are many other criticisms of the original study and its use as an example of observer effect.

The prevalence and plausibility of the observer effect in theory have prompted researchers to hypothesize that it may occur on a second level, despite the possibility that the phenomenon first observed in the Hawthorne experiments was misidentified. Therefore, scholars have put forth the notion that the outcomes of their scientific investigations may be influenced by a secondary observer effect when utilizing secondary data sources, including survey data or various indicators.

The idea is that rather than having an impact on the subjects (as with the primary observer effect), the researchers are likely to have their own quirks that influence how they handle the data and even what data they obtain from secondary sources.

Rosenthal Effect

Often also referred to as the experimenter expectancy effect, the Rosenthal Effect was explored by psychologist Robert Rosenthal, who demonstrated that a researcher’s expectations could actually affect the performance of study participants. For instance, Rosenthal and others have shown in educational settings that teacher expectations can influence students’ academic performance.

This is particularly relevant when considering the controversial work of psychologist Cyril Burt on the heritability of IQ, which was later suggested to have been influenced by observer bias and even allegations of data fabrication.

Clever Hans Phenomenon

“Clever Hans” constituted one of the earliest documented instances of apparent observer bias, which occurred in 1904. Wilhem von Olson, the owner of a horse named Clever Hans, asserted that the animal was capable of solving arithmetic equations. Von Olson would present Clever Hans with a succession of inquiries pertaining to arithmetic functions; in response, the horse would exhibit hoof tapping in accordance with the corresponding numbered answer.

The situation was investigated by psychologist Oskar Pfungst, who discovered that when the horse approached the correct number of taps, the owner would instinctively react in a specific way, signaling Clever Hans to stop tapping. This only worked if the owner knew the answer to the question. It is an example of observer bias since von Olson’s expectations influenced Clever Hans’ behaviors and behaviours, resulting in incorrect results.

Strategies to Minimize Observer Bias

Bias is an inescapable issue in epidemiological and clinical research. However, there are several potential ways and solutions that have been suggested for reducing observer bias, particularly in scientific investigations and medical research.

Blinded protocols and double-blind research can serve as a corrective filter for lowering observer bias, increasing the reliability and quality of data acquired. Blind trials are frequently necessary for regulatory approval of medical devices and pharmaceuticals, but they are not widely used in empirical studies, despite research confirming their usefulness.

Double-blinding ensures that both the tester and research participants are unaware of any information that could influence their behavior, whereas single-blind describes experiments in which information is withheld from participants that could otherwise skew the results or introduce bias, but the experimenter is fully aware of and in possession of those facts.

Standardized Procedures

Adopting standardized procedures for data collection is vital for reducing variability that might arise from subjective judgment. Clear, detailed protocols should be established and adhered to, specifying every step of the observation process.

This includes consistent approaches to noting behaviors or scores, which contributes to the reliability of the study outcomes. Furthermore, identifying any potential conflicts of interest among observers prior to the start of the research is critical in reducing research bias.

Triangulation

Finally, triangulation is a useful method for increasing the validity and reliability of findings. In research, triangulation refers to the utilization of multiple observers or data sources to gain a more comprehensive and accurate understanding of the topic at hand.

Triangulation greatly increases confidence in a study. There are several methods for triangulation, including the use of numerous observers, which is a type of reliability known as interobserver reliability, which is assessed by the percentage of times the observers agree.

References:

Breznau N (May 3, 2016). Secondary observer effects: idiosyncratic errors in small-N secondary data analysis. International Journal of Social Research Methodology. 19 (3): 301–318. doi:10.1080/13645579.2014.1001221
Gillie, Oliver (1977). Did Sir Cyril Burt Fake His Research on Heritability of Intelligence? Part I. The Phi Delta Kappan. 58 (6): 469–471.
Goldstein, E. B., & Vanhorn, D. (2008). Cognitive psychology: Connecting mind, research, and everyday experience (Vol. 59). Belmont, CA: Thomson Wadsworth
Hoyt, W. T., & Kerns, M.-D. (1999). Magnitude and moderators of bias in observer ratings: A meta-analysis. Psychological Methods, 4(4), 403–424
Jones, S. R. (1992). Was there a Hawthorne effect? American Journal of Sociology, 98(3), 451-468.
Pronin, Emily (2007-01-01). Perception and misperception of bias in human judgment. Trends in Cognitive Sciences. 11 (1): 37–43. doi:10.1016/j.tics.2006.11.001
Rosenthal, Robert; Fode, Kermit L. (1963). Psychology of the Scientist: V. Three Experiments in Experimenter Bias. Psychological Reports. 12 (2): 491. doi:10.2466/pr0.1963.12.2.491
Samhita, Laasya; Gross, Hans J (2013-11-09). The ‘Clever Hans Phenomenon’ revisited. Communicative & Integrative Biology. 6 (6): e27122. doi:10.4161/cib.27122
Spano, Richard. (2005). Potential Sources of Observer Bias in Police Observational Data. Social Science Research. 32. 591-617. 10.1016/j.ssresearch.2004.05.003