Home > Department of Media and Communications > Research > EU Kids Online > Best Practice Guide > FAQ 5: How should qualitative research be evaluated?

 

FAQ 5: How should qualitative research be evaluated?

What's the issue?

Qualitative research is usually differently evaluated than quantitative research, especially by ethnographers. As the data collection is often of a nature that is harder to be repeated (like surveys or experiments for instance), qualitative researchers came up with different a set of quality measures, such as credibility, dependability, transferability, confirmability (Guba & Lincoln, 1989), member checking and others.

Common practice

  • The researcher himself usually demonstrates credibility, in the form of properly used scientific methods, his or her training, experience and beliefs.
  • Dependability, a as criterion of consistency, is achieved by auditing - the procedure, where the research process and researcher's work has been closely examined and evaluated by other experts in the field
  • Transferability assumes that research methods, analytic categories, and characteristics of phenomena and groups are each identified sufficiently explicitly that comparisons can be made between interviews or fieldworks, for example.
  • Confirmability is also checked by auditing. Auditors (i.e. experts in the field) focus on how interpretations are grounded in the data and whether they are formulated in ways consistent with the available data.
  • In member checking, the researcher checks the findings and interpretation with the original respondents. This could take place either at the end of research, providing participants with information that ensures their views have been properly captured, or during the research process - here participants can help design questionnaires or interview guidelines, thus being seen as co-researchers (Kellett, 2005).

Questions to consider

Which data quality standard is the most sensible to approach our qualitative data with? Are children old enough to go through member checking?

Pitfalls to avoid

A common mistake in qualitative methods is look for 'quantitatively' denoted validity and reliability as the only indication of objectivity. Qualitative methods are often semi-structured or unstructured and even informal which makes it difficult to determine in advance what we want to 'measure'. It is also literally impossible to replicate an observation, a focus groups or an interview to the extent we can replicate surveys.

Another mistake derives from the assumption that since we are dealing with participants' own accounts of social reality, or observing and participating in several social situations, we have access to social "reality itself" (easily assumed since we are looking at "natural settings" for social interaction rather than "second-hand" accounts). Yet all accounts (and observations) of social reality are mediated by participants, in one way or another and, thus, all research situations are to some extent "artificial".

Further resources

Guba, E. G. & Lincoln, Y. S. (1989). Fourth generation evaluation. Newbury Park.: Sage.
Kvale, S. (1996). Inter Views: An Introduction to Qualitative Research Interviewing. London: Sage.

A researcher's experience: qualitative insight into quantitative results

Qualitative research is used either to ground a research project or explore further the insights from quantitative research. In my study of digital divides, I examined ordinary people's discourses through interviewing after I had surveyed a representative sample of the study population. Qualitative research allowed social discourses and meaning constructions to emerge in context and to be appropriately interpreted. This in turn enabled me to go beyond the quantitative measurement of individual perceptions, evaluations, attitudes and behaviours and so to depart from 'quantified' causal relationships and explore the 'quality' - the exploration of the essential character of the object of research (Kvale, 1996). Finally, I dismissed the rule of thumb that qualitative usually constitutes 'a source of ideas for quantitative testing'. In the context of my work, qualitative research aimed to give more depth and exploratory power to the quantitative findings obtained in the previous phases of the research. (Panayiota Tsatsou, UK)

Share:Facebook|Twitter|LinkedIn|