Home > Department of Media and Communications > Research > EU Kids Online > Best Practice Guide > FAQ 27: How do we maximise the reliability and validity of children's answers?

 

FAQ 27: How do we maximise the reliability and validity of children's answers?

What's the issue?

It is commonly supposed that children are unreliable informants. While designing and conducting research with children takes care, so does research with adults. Parents, for example, are subject to considerable biases (social desirability, third person bias, etc) when reporting on their children's media use; teachers also may provide a partial and overly positive account of children's activities in class.
Every effort must be made to address the possible circumstances that might undermine children's responses in research (as reiterated throughout this guide). But the notion of children as unreliable must be traded against the benefits of direct questions to children. Who else can report on what a child does with media when alone, or in their bedroom, or how they feel about violent content, or what pressure they feel from their friends? A useful principle, therefore, is to assume that each child is capable of providing valid and insightful information, provided that s/he is approached appropriately and that the data are interpreted carefully.

Common practice

In qualitative interviews, one has the chance to address inconsistencies and contradictions in what children might say. Thus one should check for misunderstandings, verify interpretations and explore contradictions in what children say, to check if this indicates experienced ambiguities and ambivalences.
In surveys, piloting the questionnaire is vital to ensure reliability, as is taking care to understand the reasons for lots of missing values on a question, or comments scribbled or muttered during the interview, or peculiar results that suggest a misunderstanding has arisen.
There are four main problems survey respondents face - s/he doesn't understand the question, s/he doesn't know the answer, s/he cannot recall it, and s/he doesn't want to report the answer. Therefore, good research practice should anticipate and seek to eliminate these problems to increase validity:

  • Understanding the question: if the question includes difficult or complex terminology and is not well understood, then we have to simplify complex terms and give definitions of those terms if needed, especially when it comes to very young children. Also, children have to be given the chance to write in more detail about their experiences regarding the questions asked (i.e. the question needs to include a category of answer where the respondent can give his/her own answer in detail)
     
  • Lack of knowledge: If the child doesn't know the answer, a researcher can either change the questions so as to ask for information that is less detailed and easier to recall, or help the child to estimate the answer or, finally, change or drop the questions.
     
  • Can't recall: to increase recall, a researcher needs to have in mind that small events of less impact are more likely to be forgotten than more important events, while recent events can be recalled relatively easily. It may help to use words that provide a clear time frame.
     
  • Unwillingness/social desirability: this is mostly in cases where questions on sensitive personal data are asked. In this case, we need to put a lot of effort into minimising the sense of judgment and maximising the importance of accuracy (vocabulary and introduction need particular attention in this respect).
    To increase the validity of more subjective questions, the researcher could rephrase questions to ensure that they will mean the same thing to all respondents, or ask multiple questions with different question forms that measure the same subjective state.

    Since even trivial changes in the questionnaire design (e.g. wording, number of alternatives/ordinal scales, and position of a question) can make an important difference in how children answer, for subjective questions, answers often cannot be interpreted directly. In other words, it may not be meaningful to report that 73% of children like the internet; but it would be meaningful to interpret the same answers comparatively (e.g. more boys than girls reported liking the internet; or, parents of users report more positive attitudes to the internet compared with parents of non-users).

Pitfalls to avoid

Forgetting to pilot all research materials. Failing to use the interview situation to clarify possible interpretations of what children say, or to clarify whether inconsistencies and contradictions are the result of methodological confusions or the genuine ambiguities and ambivalences in their lifeworlds.

Further resources

Section in D4.1 (Lobe et al., 2007) on reliability.

Examples of good practice

Zaman (Zaman, 2005) combines observations of children playing electronic games in natural environments with observations in controlled settings (in the usability lab), allowing her to get a more accurate picture of children's actual gaming behaviour. She argues that children must not only be observed while exploring and playing a game, but they must also be given the chance to express their opinions and perceptions. In order to fulfil these two objectives, Zaman employs different techniques that allow her to evaluate the usability of the game being tested. These include (1) the "think aloud" method, in which children are asked to provide a running commentary as they play a game (taking into account non-verbal responses also, if possible); (2) the "active intervention" method, in which the researcher 'actively intervenes' by asking relevant questions during the task performance (but only after children have explored the game at their own pace first); and (3) the "laddering" method, in which the researcher asks users why they like or dislike something; when the user answers, the researcher asks 'why' again; this process results in a list of connected elements: 'a ladder', at the end of which the personal value(s) of the user will be revealed.
(Veronica Donoso, Belgium)

In our research, asking children to write an essay proved to be reliable - as evidenced by the wide range of viewpoints on sensitive political issues, instances of political incorrectness and the use of slang, all of which can be interpreted as a sign of pupils' frankness. What children produce may provide answers to questions not foreseen by researchers at the beginning of the study. The same strengths, and even greater possibilities, obviously characterise what children produce online as a data source.
(Veronika Kalmus, Estonia)

Share:Facebook|Twitter|LinkedIn|