Research

Current research projects

ÉpStan (Épreuves Standardisées): The Luxembourg School Monitoring Programme

The ÉpStan assess students’ academic competencies, learning motivation and attitudes towards school at the beginning of each learning cycle of students’ compulsory education (i.e., at the beginning of cycle 2.1, cycle 3.1, cycle 4.1, VIIe/7e and Ve/9e; or in other words, at the beginning of grade levels 1, 3, 5, 7 and 9). Each year, the entire student population in each of these grade levels participates in the ÉpStan. While the assessments for all elementary schools are currently paper-based, for secondary schools the ÉpStan are completely computer‐ and web‐based (on LUCET’s in-house Online Assessment System; OASYS). Since 2009, the ÉpStan have been implemented in grade levels 3 and 9. Since 2014, the ÉpStan were successfully extended to grade level 1. The two missing grade levels, 5 and 7, will most likely follow in 2017 and 2018, respectively. Through the ÉpStan, LUCET is assembling a unique and incredibly rich longitudinal database. By bridging together future panels—entire cohorts—of students, LUCET will investigate and answer important societal and scientific questions to provide timely and policy-relevant information to the primary Luxembourgish stakeholders and the larger educational field in general.

For additional information, please read about the LUCET's core mission, visit the ÉpStan website, contact ÉpStan coordinator Pascale Esch and/or contact the LUCET management.

PISA: The OECD Programme for International Student Assessment

The Programme for International Student Assessment (PISA) initiated by the Organisation for Economic Co-operation and Development (OECD) measures the extent to which students, at age 15, have acquired the knowledge and skills that are essential for participating in the labour market and society. Starting from PISA 2000, it is conducted every 3 years, with a primary focus on one area—reading, mathematics or science literacy—for each cycle. Data from each PISA cycle (a) inform the wider public, (b) inform evidence-based policy making, and (c) are exploited in empirical research. PISA is considered to be among the most advanced international assessments to date, capturing roughly nine tenths of the world economy. Together with the Luxembourg Ministry of Education’s Service de Coordination de la Recherche et de l’Innovation pédagogiques et technologiques (SCRIPT), LUCET is responsible for the national PISA analysis and reporting.

For additional information, please visit the OECD's PISA website and/or contact the LUCET management.

OASYS: LUCET’s Online Assessment System

OASYS is LUCET’s in-house computer- and web-based (mobile) testing and assessment platform. OASYS is built around state-of-the-art web-technologies (i.e., HTML5, CSS3, PHP 5.3, JavaScript, AJAX, MySQL 5.6, jQuery, Dojo) and supports all modern browsers. The platform has been carefully designed around business processes, following a human-computer interaction (HCI) approach. To assure secure data collections and best-possible service, OASYS offers powerful live monitoring for test administrators. Every response is communicated and logged instantaneously on LUCET servers, which minimizes the loss of data even during client side computer crashes. OASYS collects timestamps of every action, thus making the platform an ideal choice for behavioural analysis and data mining. A load balancing solution allows for around 20k simultaneous client connections in the current server setup. Although OASYS has been primarily designed as an online assessment system, the client can also run in an app with a server installation on a laptop and a mobile Wi-Fi access point. Since 2010, OASYS is successfully used in the Luxembourg school monitoring programme ÉpStan. Moreover, since 2014, OASYS is also the platform of choice for the FLSHASE Student Course Evaluation (CE).

For additional information, please contact the LUCET management. For additional information on the HCI approach, please visit the University's HCI website and/or contact Vincent Koenig.

CE: The FLSHASE Student Course Evaluation

The Faculty of Language and Literature, Humanities, Arts and Education (FLSHASE) at the University of Luxembourg is committed to teaching quality assurance. The aim of the Student Course Evaluation (CE) is to develop and maintain high quality learning experiences for all students, and the CE procedure is designed and implemented to help the various categories of staff involved in teaching and related activities (i.e., instructors, course directors, the Dean and Vice-Dean) achieve this goal. Since its inception in 2009, the FLSHASE CE procedure has been empirically tested and scientifically validated over a period of years and produces verifiably stable results. Since 2015, the CE is administered electronically via LUCET’s in-house Online Assessment System (OASYS).

For additional information, please visit the FLSHASE's CE website and/or contact Salvador Rivas.

BScE (Bachelor in Educational Sciences) Admission Testing

Committed to high quality standards, the BScE is only recruiting a limited number of new students per year. Consequently, all BScE candidates have to undergo standardized admission testing in content and competency domains relevant for a future career as a schoolteacher (i.e., reading in Luxembourgish, German, French and English, mathematics and science literacy). LUCET is participating in the development, administration and analysis of the BScE admission assessments.

For additional information, please visit the BScE's admission website and/or contact Salvador Rivas.

The Genetics Lab (GL): A Computer-based Test to Assess Students’ Complex Problem Solving Abilities

FNR

General cognitive ability (“Intelligence”) is among the most important and useful psychological constructs capable of predicting health and longevity, academic success, as well as success on the job. Nevertheless, a major shortcoming of typical intelligence measures is that these paper-pencil instruments use static problem formats with a relatively limited amount of complexity and that they generally do not provide information on test takers’ problem representations. In the COGSIM-project the EMACS (Educational Measurement and Applied Cognitive Science) research unit—a precursor structure of the LUCET—aimed to overcome these severe limitations of typical intelligence tests by using computer-based complex problem solving scenarios as an alternative assessment instrument of students’ general cognitive ability. To this end, EMACS developed and validated a new computer-based assessment instrument, which is available in a German, French, and English translation: the Genetics Lab (GL). The GL is distributed under the terms of the GNU General Public License and LUCET remains interested in scientific cooperation around the GL.

For additional information, please visit the Genetics Lab website and/or contact Philipp Sonnleitner.

LEARN: Learning Expertise and Research Network

The Learning Expertise and Research Network (LEARN) is a group of—partly LUCET—scientists and practitioners who have an interest in cognitive learning processes and learning difficulties. The primary aim of this network is to stimulate the exchange of actors in different fields (e.g., research, school, therapy, parents…) who are involved in learning. The network’s most relevant results in relation to learning and learning difficulties are presented on a regular basis in the LEARN newsletter.

For additional information, please visit the LEARN website.

Current PhD theses

Development and Behavioural Validation of a Tablet-based Mathematics Training Instrument for Kindergarten Children

The research project consists of two phases. The first part will focus on the design, further development and enhancements of a framework that includes extensible and reusable libraries in addition to a touch based application for presenting and processing cognitive tests/puzzles, which involves real-time processing of user input and the verification of user provided solutions. More specifically, the application itself will be in the form of a tablet based iOS application that provides an environment for supporting mathematics training instruments in the areas of number processing and visual spatial puzzles. Moreover, the framework will provide extensible libraries that enable and facilitate the addition of new types of puzzles as well as solution verifiers. The design will also focus on the design of generic algorithms that can be extended and reused for the verification of a wide range of puzzles. The framework as a whole will provide the following capabilities: (1) A visual editor that can allow non-developers to firstly design and create new puzzles and secondly define verification patterns or rules that can allow the system to dynamically verify user solutions. (2) A comprehensive logging system for recording the behaviour of all students throughout their activities. (3) Finally, a synchronisation mechanism to store and update the progress of users between devices. The second part of the research will mainly deal with data mining and data analysis of the user activity data that is logged by the developed application to deduct patterns of the user’s behaviour. These will then be used to find correlations between different variables and to predict user behaviour and decision making strategies. One of the goals in this phase would be to apply state-of-the-art techniques from data mining and machine learning to discover behavioural patterns and implement prediction schemes.

For additional information, please contact PhD candidate Tahereh Pazouki.

GeoGebraTAO: Validation of an Adaptive Learning Environment

For additional information, please contact PhD candidate Carole Dording.

SELFASSESS: The Validation of a Student Self-Assessment-Instrument and its Usefulness for Teacher-Feedback

FNR

In a competency-based teaching and learning approach, it is essential for students to understand, what performances are expected from them and to receive qualitative feedback that goes beyond simple grades and scores. Ideally, teachers seek feedback provided by their students to assess students’ individual performance and to decide what steps to take next in the learning/teaching process (often neglected in classroom practice). Research review on the accuracy, validity and utility of student self-assessment shows that it is added value to teacher assessment. The SELFASSESS project aims to validate a student self-assessment-instrument (elementary school cycle 3: grades 3 and 4, theoretical age 9 and 10) measuring perceived academic competencies (mathematics, reading), study skills (e.g., metacognitive skills), and need for cognition. Students’ self-assessment will be compared to standardised assessment (school monitoring/Épreuves Standardisées-results) and teacher assessments (report cards/students’ grades). To make sure students have the right representation of a specific competence and to avoid bias due to possible differences in literacy abilities, a computer-based test with pictures and simple computer-animations illustrating competencies, will be developed. The relation between study skills (source: self-assessment) and need for cognition (source: self-assessment) and learning outcomes (sources: standardised assessment and teacher assessment) will be highlighted. In a further step, the test will be repeated and participating teachers will have access to their students’ self-assessment. The perceived usefulness and impact of student self-assessment for teacher-feedback will be explored via the results of a teacher-questionnaire and via comparison of the students’ test results before and after teachers’ inclusion in the student self-assessment process.

For additional information, please contact PhD candidate Denise Villányi.

SPATHMATH: Tablet-based Visuo-spatially Enhanced Early Mathematics Training

The main aim of the SPATHMATH project is to implement and evaluate a tablet-based intervention focusing on the training of visuo-spatial and numerical skills in kindergarten children. Children with language deficits are a vulnerable group for low mathematics performance, going often along with lower academic achievement throughout schooling. This is of a special interest for the Luxembourgish school environment, due to its high amount of pupils with migratory background. Especially children, who are not proficient in the language of schooling, are at risk for lagging behind their peers in mathematics (e.g., Alt et al., 2014). To overcome the language load of early mathematics instruction, a tablet-based intervention tool emphasizing visualization will be used. An aspect, which will be specifically considered in this project, is the impact of visual-spatial abilities on early mathematics skills in kindergarteners. As recently suggested, training visual-spatial abilities should enhance numerical skills (e.g., Cheng & Mix, 2014). We thus aim at investigating, in a first step, the effects of a visual-spatial training on visual-spatial and numerical skills in kindergarteners and thus contribute to this emerging field of research. In a second step, we will extend the intervention to visual-spatial numerical training. We aim at implementing this training in a sample of language-minority children in Luxembourgish kindergartens. Through the emphasis on visualization, language load should be reduced. This approach is line with the suggestions of different research groups on designing early math interventions for children with language difficulties(Alt et al., 2014; Mononen et al., 2014). When proving effective, this project will provide important information for early instruction in Luxembourg and other countries with a high rate of children from different language backgrounds. Furthermore, the interrelationship of visuo-spatial skills and early math abilities, as well as the importance of visuo-spatial abilities for later math learning, will be investigated.

For additional information, please contact PhD candidate Véronique Cornu.

Completed research projects

LANGNUM: The Effect of Language on Mathematics Performance in Bilinguals

FNR

Multilingualism is a major attribute of the current Luxembourgish educational system for preparing the future generations to face a knowledge society. However, the strong emphasis on language learning in education may also affect the learning of other key school-related skills, such as numerical processing (e.g., arithmetic). Numerical processing is an important learning outcome in itself and it is a necessary prerequisite for the acquisition of many more complex mathematical abilities, abilities that are particularly relevant for students’ success on the future labour market. What do we know from previous research on how linguistic factors affect numerical processing? Developmental and cross-linguistic studies with monolinguals have shown that language is vital for the performance on numerical tasks. Moreover, studies with bilinguals demonstrate that performance on numerical tasks depends on: (1) training language, (2) task language, and (3) language switching. These results point to important implications for the multilingual educational setting in Luxembourg, where the training language switches in grade 7 (i.e., the transition from primary school to secondary school) from German to French. Given its importance, the overarching objective of the present research project is to scrutinize how linguistic factors affect numerical processing. Specifically, we will use rigorous experimental designs (e.g., dualtask paradigms) to collect behavioural data (e.g., reaction times, correct responses) from the target population of adolescent students who are proficient in German and French to disentangle the linguistic influences on number processing. In doing so, we will tackle three research questions: (1) Which training language (German or French) do students use to perform such tasks? (2) How do number syntaxes of number words in the two languages such as the inversion of the decade and unit in German compared to French number words (e.g., 24: vierundzwanzig in German and vingt-quatre in French) influence numerical processing? (3) How does language switching and language proficiency interact with the performance on numerical tasks? The outcome of this research has therefore great potential to improve the national educational system by adapting teaching methods in mathematics to bilingual students (e.g., the inclusion of explicit number syntax instruction).

For additional information, please contact Sonja Ugen.

PELEDU: Personality in Learning and Education

FNR

Individuals’ personality is highly significant for their own but also for economic and societal development. A widely accepted taxonomy of personality is the “Big Five” model.

The Big Five are conscientiousness (reflecting dependability and will to achieve), agreeableness (likability and friendliness), openness (broad-mindedness and imaginativeness), extraversion (sociability and assertiveness), and emotional stability (adjustment vs. anxiety). Further, a higher-order personality dimension has recently been proposed: the Core Self-Evaluations (CSE) that cover the emotional stability dimension of the Big Five as well as general self-efficacy, self-esteem, and locus of control. To date, very brief, economical measures of these personality characteristics have been developed—the 10-item Big Five Inventory (BFI-10) and the 12-item Core Self Evaluation Scale (CSES).

Assessment of personality characteristics is key, because personality characteristics have been empirically proven to show impressive predictive capabilities for a large variety of life-outcomes, involving educational attainment, career success, and health. Further, personality processes that affect educational attainment are of major importance as these effects may cumulate to indirectly affect many key life-outcomes later in life. Crucially, although considerable knowledge exists on personality assessment and personality processes, the empirical basis of this body of knowledge is less strong for adolescent students. The overarching goal of the present project is therefore scrutinize adolescents’ personality in learning and education.

Drawing on a representative longitudinal sample of Luxembourgish persons (i.e., the MAGRIP sample, N = 745), Study 1 examines how personality characteristics (measured by teachers and self-reports) at the age of 12 affect life-outcomes, involving educational attainment, health (including mortality), and career success (i.e., job performance and job satisfaction) at the age of 52. A key aspect of Study 1 is to investigate how the effects of personality characteristics on health and career success are mediated via educational attainment and whether these effects remain when intelligence and socio-economic family background at the age of 12 are controlled for.

Study 2 analyzes the measurement equivalence of a German and newly developed French version of the BFI-10. Further, this study examines how the personality scales of the BFI-10 are related to learning outcomes—students’ achievement and achievement motivation. Data stem from over 4000 adolescent students who participated in two national extensions of the PISA 2009 cycle in Luxembourg.

Study 3 analyzes the measurement equivalence of a German and newly developed French version of the CSES. Moreover, Study 3 also examines how the CSES is associated with students’ achievement and achievement motivation. Data stem from the year 2010 cycle of the national school monitoring program Épreuves Standardisées (ÉpStan) where all seventh and ninth graders (total N > 9000) completed the CSES either in German or French.

To conclude, the empirical insights gained in this project inform on how students’ personality is related to learning outcomes as well as the mechanisms how Luxembourg’s educational system capitalizes on students’ personality in achieving career success and good health. Further, this project examines how to economically assess students’ personality and informs on the distribution of vital personality characteristics in today’s student population in Luxembourg. Thus, the outcomes of this project have profound significance for evidence-based educational policies that target students’ personality development, which in turn may promote educational attainment, career success and health in Luxembourg.

For additional information, please contact Romain Martin.

Completed PhD theses

ACHMO: Students’ Affect and Motivation: Assessment, Structure, and Development

FNR

Students’ affect and motivation are key determinants of academic effort, academic choices, and academic success. The present dissertation scrutinizes students’ affect and motivation with respect to (a) possibilities of economic assessment, (b) structure, and (c) development. To this end, the present dissertation focusses on three central affective-motivational constructs that have a long tradition in educational science and are not only important with respect to students’ learning, but are also considered to be vital learning outcomes themselves: academic self-concept, academic interest and academic anxiety. This dissertation includes three studies that are based on large-scale data sets.

In the first study, we examined the feasibility of short scales to assess affective-motivational constructs. This is an important research question, as testing time in educational research is typically scarce, which makes the use of long scales problematic. Specifically, we developed three-item and single-item scales for general and subject-specific (i.e., mathematics, German, French) academic anxieties and academic self-concepts and evaluated their psychometric properties by systematic comparison with corresponding long scales. Our results showed that (1) all three-item scales showed satisfactory reliabilities and substantial correlations with long scales, (2) the reliabilities and correlations of single-item measures were somewhat lower. Importantly, however, (3) the correlational patterns of the three-item as well as single-item scales with important students’ characteristics (e.g., gender, school satisfaction, achievement) were similar to those obtained with the corresponding long scales. We concluded therefore that when a study design requires short measures, three-item scales and perhaps even single items may be used as reasonable alternatives for assessing academic anxiety and academic self-concept.

The second study tackled the question of structural models of students’ affect and motivation. With regard to academic self-concept, much research has been devoted to the structural conceptualization of this construct. Current structural models consider academic self-concept to be not only subject-specific by nature but also hierarchically organized with general academic self-concept operating at the apex of the hierarchy. Although theoretical considerations and consistent correlational patterns of academic interest and academic anxiety measures indicate that these constructs show similar structural characteristics to academic self-concept, structural models that can account for and test these characteristics are missing. Therefore, first, we specified and examined structural models of academic self-concept, academic interest, and academic anxiety, separately. Our results underscored empirically the structural similarities between the constructs. Furthermore, theoretical predictions and empirical results indicate interrelations between the different affective-motivational constructs. In order to properly examine the constructs’ interrelations, the multidimensional and hierarchical organization of the constructs needs to be taken into account. Therefore, in the next step, we developed an integrative model which provides a comprehensive formal psychometric representation to capture and analyze the complex interplay of general and subject-specific (i.e., mathematics, French, and German) components across academic self-concept, academic interest, and academic anxiety. Finally, we validated the integrative model with respect to indicators of students’ achievement.

In the third study we investigated the developmental dynamics of students’ affect and motivation from Grade 7 to 9. Importantly, in previous developmental research the multidimensional and hierarchical organization of the constructs was rarely taken into account. Consequently, little is known about the manifold developmental dynamics of general and subject-specific components of academic self-concept, academic interest, and academic anxiety. Therefore, we applied longitudinal models that capture the hierarchical and subject-specific structure of these constructs to contribute to a fuller and more nuanced understanding of their developmental processes. The investigated constructs showed moderate differential stabilities at the general and subject-specific levels. Further, the development of academic self-concept, academic interest, and academic anxiety seems to be characterized neither by top-down nor bottom-up developmental processes. Rather, general and subject-specific components of the constructs in Grade 9 were shown to be primarily a function of the corresponding components in Grade 7. However, there proved to be several negative ipsative developmental processes across different school subjects.

For additional information, please contact Katarzyna Gogol.

GUIDE: Towards Consolidated Methods for the Design and Evaluation of User Experience

FNR

In the “third wave” of Human-Computer Interaction (HCI), the emergence of User Experience (UX) as a key concept has opened up both exciting perspectives and hard challenges. The conceptual shift to a more comprehensive and emotional view of human-computer interactions has been accompanied by the development of numerous methods and tools for the design and evaluation of interactive systems. UX research has thus been mainly driven by novelty and innovation and to date a majority of the developed tools lack validation and consolidation. UX research undoubtedly raises new concerns and challenges common conceptual and methodological practice. Thus the primary objective of this thesis is to contribute to UX consolidation.

We addressed this objective by relying on a mixed-methods approach for the empirical part of this thesis, involving comparatively large and representative samples. This part encompasses six studies, representing a variety of perspectives related to UX research consolidation. More specifically, this dissertation includes a replication study (Paper A, N = 758), the translation and validation of a tool (Paper B, N = 381), the development of new design and evaluation methods (Paper C and D, N = 137 and 33), the empirical assessment of the relevance of established HCI methods for the evaluation of UX (Paper E, N = 103) and finally an investigation on how to bridge UX research and practice through a design approach (Paper F).

The contributions of this thesis to UX research and practice regard both UX as a concept and its methodologies. Main findings inform about the benefits, challenges, and limitations of UX consolidation strategies as derived from our respective studies (papers A to F). Each study provides advances to both research and practice, while the combination of our studies pushes forward consolidation of UX. This is an essential step with regards to an emerging concept and informs an overarching research agenda aiming at a continuous interdisciplinary fostering of the UX field.

For additional information, please contact Carine Lallemand.

How does Usability Improve Computer-based Knowledge Assessment?

There has been a major shift from paper-and-pencil towards computer-based assessments (CBAs). CBA has the potential to overcome various limitations imposed by traditional assessment approaches, mostly because CBA allows for the easier and more effective measurement of complex knowledge concepts via the use of dynamic items. Innovative item formats in CBA (dynamic or interactive multimedia items) allow the assessment of complex skills (e.g., complex problem solving), but they also tend to increase the complexity of CBA instruments because of their augmented interactivity. If the CBA is not user-friendly, the test-taker might spend more time and effort trying to understand how to interact with the system instead of focusing on the assessment task itself. The research field of human-computer interaction (HCI) shows that the usability or user-friendliness of the system affects the interaction. Usability addresses how appropriate (for a particular use) or how user-friendly a CBA instrument is. Thus, to the extent that there is any detrimental technical bias in CBA affecting the user-friendliness of a CBA, it will affect the user’s interaction with the instrument and consequently the instrument’s psychometrics.

There are certain guidelines (e.g., International Test Commission, 2005) that require usability testing to safeguard against the CBA instrument measuring skills or competencies other than those that are supposed to be measured by the CBA instrument. However, surprisingly little research has been conducted to investigate whether usability has been acknowledged in CBAs and whether researchers are aware of the impact usability might have on the assessment results. To answer these and further research questions, three studies were conducted within this present Ph.D. project. These studies focused on a specific CBA instrument: concept maps.

A concept map is a graphical illustration of a knowledge concept and can be described as a “knowledge net.” Research has shown that the concept map is a valid and reliable instrument for measuring conceptual knowledge. Concept map instruments are a good example of the increasing popularity of CBA, as technology allows test-takers to correct and improve their concept maps with ease. Test-takers and their examiners receive immediate feedback on the correctness of the concept map. Furthermore, it is a good example of new approaches to knowledge assessment, which is currently a major trend. All of this taken together makes concept maps a good proxy for new ways of assessing knowledge. Our research builds on three studies (Study I – literature-based study, Study II – laboratory study, Study III – school study) that investigated the impact of usability on computer-based knowledge assessment using concept maps.

Study I, a literature-based study, scrutinized the existing literature and answered the fundamental research question of whether and how usability has been acknowledged in CBA concept map studies. As no such literature review existed prior to the current project, a systematic literature review was conducted to shed light on the representation and relevance of usability displayed in CBA concept map studies. The literature review indicated that only 24 of 119 journal articles that assessed computer-based concept maps actually discussed the usability of the applied instrument in some way and only three of those 24 journal articles explicitly mentioned that they evaluated the usability of the applied instruments. The literature review illustrated that usability is rarely acknowledged and reported in CBA concept map studies. Our review brings to light the idea that the impact of usability, although well established in the field of HCI, has received insufficient attention in the field of educational assessment.

Study II, a laboratory study, addressed the main research question of how HCI methods can be introduced to the field of educational assessment to improve the usability of a CBA concept map instrument. With a user-centered design and development approach, which makes the user the key reference, usability testing and heuristic analyses were conducted in the usability laboratory. The approved HCI methods of usability testing and heuristic analyses were combined to evaluate and further improve a CBA concept map instrument. We applied three iterative design and re-engineering cycles that were based on the results we received from the usability testing and heuristic analyses. To verify the improvements in the usability of the CBA concept map instrument, three independent and randomly assigned groups of 30 students underwent concept map assessments using the three iteratively developed instruments (baseline (V1), further developed: V2 and V3). The results of this independent design sample study showed that the HCI methods allowed us to design and develop demonstrably usability-improved concept map instruments; they furthermore revealed that the usability improvements significantly improved the assessment outcomes.

Study III, a school study, empirically verified the impact that usability has on the psychometrics of the applied CBA instruments and on the satisfaction and performance of the test-takers. The experimental study was conducted at school, and 542 students were randomly assigned to one of the three CBA concept map instruments that showed consecutively improved usability. The performance of the test-takers who worked with the usability-improved instruments significantly increased in comparison with the baseline version. The test-takers indicated that they were more satisfied with the usability-improved instruments. Moreover, the internal consistency of the items from the baseline instrument in comparison with usability-improved instruments increased from a Cronbach’s alpha of .62 to .84.

To summarize, Study I showed that the impact of usability is rarely evaluated and discussed in CBA concept map studies. Studies II and III clearly showed that usability has a positive impact on the test-takers’ interaction with the CBA concept map instrument. Specifically, Study II illustrated and demonstrated how HCI methods could be used to achieve usability-improved CBA instruments, which in turn allow for better assessment outcomes. In addition, Study III illustrated that the psychometrics were also affected by the usability of the instrument; specifically, the reliability (measured as the internal consistency of the applied items) increased when the usability-improved instruments were used. Thus, the continuing trend towards CBAs calls for more systematic usability research to help ensure satisfactory psychometric test properties and test-takers’ satisfaction with the instrument. The studies confirmed the hypothesis that if usability is not taken into account, the assessment results may severely compromise the quality of individual diagnostics as well as educational policies and educational decisions.

For additional information, please contact Katja Weinerth.

Changing Commuter Behaviour through Gamification

FNR

This thesis explores how the dynamic context of mobility, more specifically the commute to and from work in the region of Luxembourg, can be changed through gamified mobile applications. The goal is to get a better understanding of the innovative application area of gamified mobility and its potential, as well as to describe its implications for research and practice. This applied research is inspired by a participatory design approach, where information is gained by adopting a user perspective and through the process of conceptualising and applying methods in empirical studies. The four empirical studies described in this thesis employed a mixed- methodology approach consisting of focus group interviews, questionnaires and mobile applications. Within these studies four prototypes were developed and tested, namely Coffee Games, Driver Diaries, Commutastic and Leave Now. The studies show concrete possibilities and difficulties in the interdisciplinary field of gamifying mobility behaviour.

This dissertation is composed of seven chapters: Chapter I introduces the topics mobility, games and behaviour; Chapter II presents a proof of concept study (Using Gamification and Metaphor to Design a Mobility Platform for Commuters); Chapter III explains the development and validation of a mobility research tool (Driver Diaries: a Multimodal Mobility Behaviour Logging Methodology); Chapter IV describes the development of a new gamified mobility application and its evaluation (Studying Commuter Behaviour for Gamifying Mobility); Chapter V provides an empirical assessment of the relevance of gamification and incentives for the evaluation of a mobile application (Changing Mobility Behaviour through Recommendations) and Chapter VI is a summary on how to change mobility behaviour through a multilevel design approach (Using Gamification to change Mobility Behaviour: Lessons Learned from two Approaches).

The four prototypes help to address the primary goal of this thesis, which is to contribute to new approaches to urban mobility by exploring gamified mobility applications.

Coffee games is a proof of concept, low-fidelity implementation of a real-life game that tests gamification elements and incentives for changing indoor-mobility behaviour. The findings of two iterations with a total of 19 participants show the adaptability of the concept to different contexts. The approach to change indoor-mobility behaviour with this mock-up game was successful.

Driver Diaries is a methodology to assess mobility behaviour in Luxembourg. The aim with this mobile, digital travel diary is to study features of cross-border commuter mobility and activities in Luxembourg in order to identify suitable elements (activities etc.) for a gamified mobility application, such as Commutastic. After two rounds of data collection (Android and iOS) the records of 187 participants were analysed and the results illustrate the mobility habits of the target audience.

Commutastic is a mobility game application that motivates users to avoid peak-hour traffic by proposing alternative after work activities. Analysing the data of 90 participants, we find that the timely offer of an activity in the proximity along with gamification elements involves users and motivates a third of them to engage in alternative mobility behaviours.

Leave Now is a gamified recommendation application, which rewards users for leaving their workplace outside of their usual schedule and explores the role of specific gamification elements on user motivation. The study, which was conducted with 19 participants, shows differences between an individual play and a group play condition regarding leaving time changes.

The contributions of this thesis to gamification and mobility research and practice span from mobility participations as a game and integral part of our everyday life to methodologies of its successful implementation in the Luxemburgish context. The results show the advantages, disadvantages, and restrictions of gamification in urban mobility contexts. This is an important step towards gamifying mobility behaviour change and therefore towards research aiming at a wellbeing in a better urban life.

For additional information, please contact Martin Kracheel.

SPOTASSESS: Evaluation of Cognitive Ability in Heterogeneous Student Populations: Development of the Test of Cognitive Potential (TCP) 

FNR

Evaluation of cognitive potential of individuals with diverse backgrounds often presents a challenge. Individuals’ imperfect mastery of the language of instruction in particular presents a major threat to validity of results and prohibits comparison to performance with better language skills. Eliminating language from test instruction may be a solution to this problem. The present thesis presents the development and validation of the innovative Test of Cognitive Potential (TCP), a language-free, tablet-based group-assessment tool designed to measure fluid intelligence within children in grade 4 (regular age between 9 and 10). Using dynamic visual instruction videos, task requirements are presented visually, without adding any verbal clarification. In order to sample a wide array of cognitive processes on different contents (domain-general, quantitative and semantic-syntactic reasoning as well as visual processing), the TCP includes a variety of subtests. Using confirmatory factor analysis, the overall structure of the test is evaluated within student samples from Luxembourg, Germany and Brazil. In a 2x2 design, measurement invariance of a latent TCP g-factor is further shown to largely hold across combinations of verbal vs. visual dynamic instruction and tablet-based vs. paper & pencil assessment. For the target TCP version (tablet and dynamic visual instruction), only one subtest is found to present increased residual variance and thus needs revision. Overall, a strong association of the TCP with criterion measures (educational achievement and a well-established test of cognitive ability) is found across samples. Measurement of a latent TCP factor could further be shown to be invariant across students with differential opportunity to learn. No strong evidence of biased measurement is found for students with different language backgrounds and socio-economic status.

The Test of Cognitive Potential proofs to be a valid and reliable tool for the measurement of general cognitive ability. It will be particularly useful for the evaluation of children with limited understanding of the local language. Avoiding language bias, the TCP may contribute to the reduction of inequalities in educational opportunities that are produced by structural characteristics of many school systems.

For additional information, please contact Claire Muller.