Home > Department of Media and Communications > Research > EU Kids Online > Best Practice Guide > FAQ 4: How should quantitative research be evaluated?

 

FAQ 4: How should quantitative research be evaluated?

What's the issue?

Quantitative data as well as qualitative data has to be evaluated on the basis of its ability to reach the objectives of the research. Besides more complicated discussions related to epistemological value of quantitative approaches, the decision to use quantitative data should therefore be the answer to the question what kind of data is needed to analyse a particular problem. If the main aim of the study is generalizability then this is usually best achieved through quantitative methods such as surveys. This doesn't exclude however the possibility of combining, in different sorts of ways, quantitative and qualitative approaches.

Common practice

The decision to use quantitative methods is usually based on the desire to achieve a certain level of generalizability. More specifically, the goal is to achieve reliable and accurate measurement for one or both of the following:

  • Point estimates. This is the desire to be able to state, for example, how many children use the internet and what they do online.
  • Relationship between two or more variables - for example, if girls are more likely than boys to go online or vice versa.

As the goal of quantitative studies is to get results which then can be said to apply generally, the main issue in these studies is to limit both random and systematic errors.

  • Random errors are controlled by using the appropriate statistical tests and, as a rule of thumb, the bigger the sample the smaller the random error.
  • Systematic errors are controlled through the research design and through strict control over the research process. One of the most effective way to limit systematic error is to use simple random sampling and achieve a high response rate.

Questions to consider

The first goal of making point estimates puts strong demands on the data especially in terms of systematic errors. This means for example that if children with certain social status are more likely than others to be interviewed in a study that weakens the data as a basis for estimating how many children use the internet. However, even with systematic error present in the data it might be possible to make quite accurate estimates for relationships between variables.

Quantitative methods are popular because they allow you to make generalizations. But there are also some limitations. The number of questions is always limited, not to mention their scope. Some subjects may be difficult to translate into "closed questions", especially if dealing with sensitive subjects or when we are searching for meaning and understanding.

Pitfalls to avoid

Because quantitative methods rely on comparability and generalisation, the ability to measure exactly the same thing each time is crucial. This of course poses problems of reliability (that is, measuring something the same way each time, without introducing any changes) and of validity (that is, finding a way of measuring exactly what is intended in a particular piece of research). The last problem is more difficult to solve than the first, since it depends on the ability to translate accurately in to a specific set of questions a particular research problem. In other words, the problem of validity is directly connected with how well the concept or a construct is translated into a set of indicators to measure what we want to know.

Many quantitative studies are based on questionnaires. But questionnaires can be tricky and the validity of the results depends to a large extent on the assumption that all respondents have understood the questions in the same way and in the same way as the researcher. It is especially important to evaluate critically this assumption when working with children.

Further resources

Bryman, A. (2004). Social research methods (2nd ed.). Oxford: Oxford University Press.

Share:Facebook|Twitter|LinkedIn|