Rory Fitzgerald (City University London)
One survey in 30 countries: the challenges of achieving equivalence in the European Social Survey now and in the future
Abstract:
There are considerable challenges to successfully implementing a high quality survey in a single nation, mono language context in terms of achieving a representative sample and ensuring equivalence of meaning through a standardized questionnaire. However it is even harder to achieve these outcomes in a cross-national survey due to differences in culture, language, institutional and organizational structures and context across the different participating nations.
This presentation will outline how the European Social Survey aims to address these cross-national challenges to provide a dataset that is as representative and comparable as possible. Efforts to ensure this as key stages of the data life cycle will be discussed such as during sampling, questionnaire design, fieldwork and data production. The presentation will also touch upon recent developments in survey research and discuss how these are pose specific challenges for cross-national surveys.
Jouni Kuha (LSE)
Measurement invariance in cross-national survey research: definitions, questions and complications
Abstract:
The core purpose of cross-national surveys is cross-national comparison – and a necessary condition for this is comparability of the survey data across countries. An important element of this comparability is measurement invariance (also known as equivalence of measurement) of the survey questions, i.e. the requirement that a question should measure the same construct and in the same way in every country in the survey. When we consider multiple-item batteries of questions for constructs such as attitudes, which are analysed with statistical latent variable models, 'multigroup’ extensions of these models give us powerful tools for assessing whether items are invariant, and even allowing for some lack of invariance in the final analyses. Such models are fairly widely used in factor analysis, and increasingly well-known also for other latent variable models such as latent class and latent trait models for categorical items. Even when these tools are used, however, they still often leave us with a fundamental open question: If the analysis suggests that our survey measures do not possess full measurement invariance, what should we do and which results should we use for final conclusions from the data? The answers to this question are not obvious, and not delivered by the models alone.
In this talk we discuss a number of issues related to this question. These include the sensitivity of conclusions of interest to ignoring any non-invariance of measurement, the meaning of different types of invariance, and the implications of 'partial’ invariance models where some but not all of the items are taken to be invariant.
Lars Lyberg (University of Stockholm)
Critical steps in the design and implementation of cross-national surveys
Abstract:
Problems associated with the design and implementations in cross-national surveys are magnified compared with those that we face in national surveys. Experience shows that there is a need for a strong central coordination effort when surveys are implemented in more than one country. Otherwise countries will deviate intentionally or unintentionally from prescribed procedures, which will affect comparability. Problems associated with translation are particularly crucial. Other issues include the conflict between national and comparative interests, a lack of methodological capacity in some local survey organizations, a general variability in general survey conditions across countries, and strong media interests in rankings of countries. We will present examples of problems experienced in surveys of literacy (IALS), adult competence (PIAAC), student achievement (PISA), and social research (ESS) and how they might be mitigated.
Bart Meuleman (Katholieke Universiteit Leuven)
Changing attitudes towards immigration in times of economic crisis: a cross-national and over-time comparison
Abstract:
While anti-immigration attitudes in Europe have been investigated quite extensively, research that focuses on the evolution of attitudes towards immigration remains relatively scarce. The existing studies show that, since the mid-90s, evolutions in anti-immigration sentiment are not monotonous and vary widely across European countries. Furthermore, the observed trends can be interpreted in terms of realistic group conflict theory: Unemployment rates and immigration flows partly explain the cross-country variation in attitude trends.
The increasing availability of cross-national repeated cross-sections provides new opportunities to put the group conflict theory to the test. Such data allows to shed light on how anti-immigration attitudes are evolving during the economic crisis different European countries are currently experiencing. This relatively novel strategy compares survey data across countries and time points simultaneously. Although this seems to be a promising approach, the combination of cross-national and over-time comparisons brings along conceptual issues as well as challenges for statistical analysis that merit further study.
Joint work with Elada Davidov and Jaak Billiet
Irini Moustaki (LSE)
Modelling missing values in cross-national surveys: a latent variable approach
Abstract:
In survey research, the aim is often to measure some underlying trait(s) of the respondents through their responses to a set of questions. In the paper, we focus on cross-national surveys, where the main research objective is to compare the distribution of the latent variables across countries. We focus on the modelling of item non-response in such surveys, and studying its effects on cross-national comparisons. We consider models which are extensions of standard multigroup latent variable models, extended in such as a way as to model the missing data mechanism together with the latent constructs and their measurement. The model for the missing data mechanism will serve two purposes: first to characterize the item non-response as ignorable or non-ignorable and consequently to study the patterns of missingness and characteristics of non-respondents across countries, but also to study the effect that a misspecified model for the missing data mechanism might have on the substantively interesting parts of the model, including the cross-national comparisons.
Joint work with Myrsini Katsikatsou and Jouni Kuha
Daniel Oberski (Tilburg University)
Valid cross-country comparisons using the EPC-interest
Abstract:
Cross-national surveys are done to compare countries. But findings of substantive differences are always threatened by the alternative explanation that the differences found are merely due to differences in measurement. For example, a country ranking on satisfaction may change substantially once cross-country differences in response behavior are accounted for. Therefore, the size and statistical significance of measurement differences are often evaluated before any substantive comparisons are done: that is, “measurement invariance testing” is performed to rule out this threatening alternative explanation.
Unfortunately, the current practice of measurement invariance does not guarantee that the threat is truly ruled out. The reason for this worrying problem is that measurement invariance testing evaluates the size and significance of cross-country measurement differences without considering their effect on the parameters of interest. Because this effect may be large even when measurement differences appear small and vice versa, current practice fails to achieve the goal for which it was intended.
In this presentation I discuss a complementary measure that directly evaluates the impact of cross-country measurement differences, the recently introduced “EPC-interest”. This measure aims to rule out the threat of measurement differences to substantive conclusions by directly examining their effect on the results of interest. Using examples from the European Social Survey and the World Values Survey, I demonstrate that using this measure allows for more robust cross-country comparisons and solves some problems encountered in the published literature.
Return to main workshop page