Home > Department of Media and Communications > Research > EU Kids Online > Best Practice Guide > FAQ 7: In comparative research, how do I choose which countries to compare?

 

FAQ 7: In comparative research, how do I choose which countries to compare?

What's the issue?

Little formal attention is paid to the question of country selection, these decisions often being somewhat ad hoc, convenient or serendipitous, not necessarily best meeting the research aims but depending instead on practicalities of contacts and funding. Yet, depending on the countries compared, findings will centre more on similarities or on differences.

Hence, a research project which spans continents, comparing vastly different countries, may have difficulty identifying the fine-grain differences that research on similar countries will reveal. Conversely, comparing similar countries, perhaps from the same geographic region, may miss the bigger picture of transnational differences. The lens one chooses to apply depends on the research question being asked.

Common practice

A helpful analysis developed by Kohn (Kohn, 1989) identifies four distinct approaches to cross-national comparison within social science according to its primary purpose (see below).

Approaches to Cross-National Comparison

Country as ... Object of study [1] Context of study [2] Unit of analysis [3] Part of larger system [4]
Primary purpose Idiographic - understand each country in own terms Test abstract hypothesis or dimension across countries Seek relations among dimensions on which nations vary  Interpret each country as part of transnational system
Country selection Compare any, all or similar countries Maximise diversity on one dimension Diversity within a common framework Maximise diversity on all dimensions

Source: abridged version of "Models of Comparative Cross-National Research" from Methodological report of EU Kids Online research network (Lobe et al., 2007).

If one is treating each nation as the object of study, comparing fairly similar countries may prove most useful, particularly to inform regionally-based (e.g. EC) policy (Hantrais, 1999; Teune, 1990).

If one is studying the generality of a finding across nations (country as context of study), selecting countries so as to maximise diversity along the dimension in question allows one to explore the scope or universality of a phenomenon.

For model three, one would select countries to capture diversity within a common framework. Since the use of multiple dimensions invites a conception of their inter-relations, this should support theory-building by developing a common framework based on a pan-national conception of relations among the dimensions.

Lastly, projects which conceptualise the nations to be compared as components of a transnational system will select countries by seeking to maximise range and diversity globally.

Questions to consider

While policy development, especially at a European level, provides a significant impetus towards comparison based on standardisation, with substantial funding being used to generate multi-national quantitative data sets, the academic trend is increasingly 'away from universalistic culture-free approaches to culture-boundedness, which has placed the theory and practice of contextualization at the nexus of cross-national comparative studies' (Hantrais, 1999: 93).

This is, arguably, a particular problem for qualitative research. As Mangen (1999: 110) observes, 'the strengths of qualitative approaches lie in attempts to reconcile complexity, detail and context' - all dimensions that are particularly difficult to convey when translating across languages and research cultures, and when undertaking the exercises in standardisation or data reduction that making comparisons seems to demand. Yet such concerns apply also to quantitative research, where the ease of producing neat tables of statistics may beguile the researcher into neglecting crucial differences in the meaning of terms or the contexts within which they apply.

Pitfalls to avoid

Many comparative researchers address the challenge of comparison by standardising their methodology and research tools, devoting considerable attention to strict equivalence in measurement procedures through such techniques as the back-translation of survey instruments, as well as ensuring transparency by including questionnaires and coding schedules in the final publications. The difficulties of comparative research, on this view, stem from the challenging task of ensuring equivalence of terms, comparability of measures and in applying standardised forms of analysis. It must be acknowledged, however, that many (perhaps all) key concepts change their meaning on translation.

In practice, quantitative research usually makes an effort to keep the exact wording in different national surveys (although variation can still be introduced in the process of translation and in terms of whether a concept means the same thing in different countries/cultures). In qualitative interviews, the difficulties are compounded by the fact that researchers can agree on a general interview schedule, but then in 'conversations' with the participants the exact wording often varies, depending on the particular interview context, on the researcher's disciplinary training and on the cultural or national research context.

Example of good practice: Mediappro research project

The Mediappro project illustrates the first approach, for it sought to identify the specific cultural contexts within which children in different countries use the internet and, in consequence, use it differently. While findings from one country were used to stimulate questions for another, with findings from each country reported side by side, few direct comparisons are drawn, possibly because these seem to violate the cultural integrity of each nation.

About 9000 young people aged 12-18 (7400 in Europe and 1350 in Quebec), participated in the Mediappro survey. For practical reasons, each national team selected the participants from their schools with the consent of school principals and parents. In order to construct a relevant sample at the international level, schools were selected according to their geographical location and their social, economic and cultural setting. Three school grades, representing three age groups, were defined: 12-14 (beginning of secondary school), 15-16 (middle of secondary school), and 17-18 (end of secondary school). Using this method we were able to obtain a varied sample representing the diversity of young people's life contexts, reflecting national differences that exist across Europe. We collected the data through two means. The project team elaborated a common questionnaire including 63 items and distributed it to the whole sample during school time, from September to October 2005. Based on the results of this quantitative phase, 240 young people (24 in each country) were selected according to their different levels of internet usage, ages and gender, for individual interviews. [...] Aside from the statistical analysis of the questionnaires, Mediappro teams conducted each phase of the survey themselves in order to guarantee a coherent process and high quality analysis.

(Mediappro - A European research project: the appropriation of new media by youth)

Example of good practice: SAFT project

The SAFT project (Staksrud, 2005) illustrates the second approach, for it examines how differences in age, gender, parent-child relations, etc are fairly constant across (Northern) European countries, as regards children's use of the internet and their contact with its risks. In other words, SAFT treated each country as a distinct context precisely in order to test whether the same finding (such as parents underestimate risks online compared with children) applies in those different contexts; only if the similarity holds is the finding considered robust.

Example of good practice: Children and their Changing Media Environment project

The Children and their Changing Media Environment project (Livingstone & Bovill, 2001) exemplifies the third approach, for it sought to understand how systematic differences in education, wealth, parenting, etc were associated with differences across countries in children's media use, including adoption of new media. Thus it examined the correlations between national wealth (e.g. GDP), or degree of ICT diffusion, and the dependent variables of children's media use; this model expects to find neither similarities nor differences, simply, but rather to find a model that applies across all nations that explains the differences observed among them, as explained to us by the authors of Chapter 1 regarding the choice of research contexts for comparison:

In what follows, we examine first the contexts for children's lives across Europe and, second, we map media environments across Europe, focusing on the electronic screen. In both cases, our aim is to identify key dimensions that discriminate among countries, or groups of countries, in order to facilitate the thematic cross-national comparisons that form the substantive chapters of this volume. We caution, however, that there's no easy way to place boundaries around "context". Our research involves countries that are broadly comparable in degree of modernization and global positioning; however, we can only provide a brief and necessarily selective overview of the key dimensions along which the 12 countries vary, and we include nation-by-nation tables only when cross-national differences are marked (Livingstone, d'Haenens, & Hasebrink, 2001).

Example of good practice: Special Eurobarometer 250

Though lacking an explicit theoretical framework, the recent Eurobarometer surveys of internet use at home illustrate the last approach, for the policy context assumes a global process of transition into the Information Society, with countries further advanced (earlier adoption, greater diffusion, more broadband, etc) showing signs of both benefits and risks for children. The implication is that all countries in the research are experiencing the same phenomenon, albeit at different points in the process (so that what is already evident in one country - regarding, for example, online risks for children - may be anticipated in the near future for the next).

In this report we present the findings from a survey about Safer Internet that was carried out in the 25 Member States of the European Union, in the two acceding countries [Bulgaria and Romania] and the two candidate countries [Croatia and Turkey] between 7 December 2005 and 11 January 2006. The survey is part of the European Union's Safer Internet Programme. This programme has been running since 1999, and aims to equip parents and teachers with the knowledge and tools they need to ensure internet safety. (Safer Internet Report from Special Eurobarometer n°250)

In these countries, the survey covers the national population of the respective nationalities and the population of citizens of all the European Union Member States who are residents in those countries and have a sufficient knowledge of one of the respective national language(s) to answer the questionnaire. A multi-stage random sample design was carried out in all countries, according to the distribution of the resident population of the respective nationalities in terms of metropolitan, urban and rural areas, thus being representative of all regions (According to Eurostat "administrative regional units" (NUTS II or equivalent). See Technical Note from Special Eurobarometer n°250)).

Rules of thumb

If one is treating each nation as the object of study, comparing fairly similar countries may prove most useful, particularly to inform regionally-based (e.g. EC) policy (Hantrais, 1999; Teune, 1990).

If one is studying the generality of a finding across nations (country as context of study), selecting countries so as to maximise diversity along the dimension in question allows one to explore the scope or universality of a phenomenon.

For model three, one would select countries to capture diversity within a common framework: since the use of multiple dimensions invites a conception of the relations among them, this tends to support theory-building through the development of a common framework based on a pan-national conception of the dimensions themselves.

Lastly, projects which conceptualise the nations to be compared as components of a transnational system will select countries by seeking to maximise range and diversity globally.

Further resources

Kohn, M. L. (1989). Introduction, in M. L. Kohn (Ed.), Cross-National Research in Sociology. Newbury Park: Sage.
Livingstone, S. (2003). On the challenges of cross-national comparative media research. European Journal of Communication, 18 (4), 477-500.

Share:Facebook|Twitter|LinkedIn|