Measurement of Non-cognitive Constructs in School Contexts: Implications from International and Large-scale Testing Programs

Transcription

1 Measurement of Non-cognitive Constructs in School Contexts: Implications from International and Large-scale Testing Programs Jihyun Lee (Ph.D.) School of Education, University of New South Wales (UNSW) Sydney, Australia Presented at the 2015 Assessment in Schools Conference UNSW Global - EAA, 2nd Floor, Training Room 3 October 12, 2015

2 Outline of My Talk 1. Part 1. Research on what to measure in non-cognitive domain (80%) My own and others research Large-scale, international assessments 2. Part 2. Measurement/assessment of noncognitive constructs (20%) Any substantial breakthrough? 3. Conclusion. Three take-home messages

3

4 WOW! THIS IS THE BEST CLASS THIS WEEK! KIDS ARE ENJOYING SO MUCH!

5

6

7

8

9

10

11 Part 1. Non-cognitive What skills or dispositions should we measure (know about)?

12 Non-cognitive Skills? Defining non-cognitive any skills, behaviors, habits, tendencies, traits, and disposition that are typically not measured by <cognitive, achievement> tests. In Education: motivational, attitudinal, emotional, interpersonal characteristics and personality (Farkas, 2003, Annu. Rev. Sociology, Vol. 29, p.542)

13 Challenges in Research in Non-cognitive Skills Some argue that non-cognitive traits are hardly non-cognitive. Non-cognitive qualities cannot be developed without proper cognition. Cognition cannot be triggered without non-cognitive qualities. Cognitive Noncognitive Cognitive Noncognitive Construct /Domain Independence Boundary of non-cognitive domains (Farkas, 2003, Annu. Rev. Sociology, Vol. 29, p.542)

14 Challenges in Research in Non-cognitive Skills Flexibility Critical thinking Creativity Openmindedness Selfmanagement Motivation Problem-solving Collaboration Metacognition

15 National Council on Measurement in Education (2012) Based on the commonality across different frameworks Constructs with relatively mature research base: C3M2 Critical Thinking Creativity Collaboration Metacognition Motivation

16 Lee & Shute (2010) adaptability, attitude, affect, anxiety, belief, basic skills, classroom behaviors, control strategies, cooperation, curiosity, discipline, effort, elaboration strategies, emotion, engagement, extracurricular activity, habit, help-seeking, homework, independence, interest, leadership, Any particular learning strategies, learning style, liking, memorization, metacognition, motivation, non-cognitive, note-taking, organizational skills, parenting, parental involvement, non-cognitive peer, peer-teaching, perseverance, principal leadership, resilience, school climate, school leadership, school organization, sense of belonging, self-confidence, self-concept, selfcontrol, self-direction, self-discipline, self-efficacy, self-regulation, qualities of a learner? social competence, social context, social development, social relation, study habits, student-teacher relationship, teacher efficacy, teacher support, teacher relation, teamwork, test anxiety, test-taking strategies, thinking skills, time management, time spent on tasks, and value Lee, J., & Shute, V. (2010). Personal and social-contextual factors in K-12 academic performance: An integrative perspective on student learning. Educational Psychologist, 45,

17 Lee & Shute (2010) Student-Personal Factor Social-Contextual Factor Student Engagement Learning Strategies School Climate Social-Family Influences Behavior Cognition Academic emphasis Parents Cognition Metacognition Teacher Peers Affect Behavior Leadership

18 Lee & Shute (2010) Student-Personal Factor Student Engagement Learning Strategies Behavior Cognition Affect Cognition Metacognition Behavior Attendance Intrinsic motivation Interest, curiosity Summarize Monitor Selfmanagement Punctuality Effort Sense of belonging Understand Regulate Strategies (e.g., notetaking) Participation Selfbeliefs Emotion Elaborate Evaluate Help-seeking

19 Lee & Shute (2010) Social-Contextual Factor School Climate Social-Family Influences Academic Teacher Leadership Parental Involvement Peer Influences Value in Academic Work Collective efficacy Collegiality Attitude Support Expectation Teacher empowerment High morale Behaviour Norms Positive response Sense of affiliation Goalsetting Style Attitude & Behavior

20 Hattie s (2009) Visible Learning Effect sizes of Non-cognitive Variables

21 Hattie (2009) Engagement (d =.48) Self-concept (d =.43) Student-level Non-cognitive Variables Anxiety (d =.40) Attitude to mathematics /science (d =.36) Personality (d =.19)

22 Hattie (2009) Teacher clarity (d =.75) Teacher-student relationships (d =.72) Not labeling students (d =.61) Teacher-level Non-cognitive Variables Teacher expectation (d =.43)

23 Classroom behavior (d =.80) Hattie (2009) Peer influences (d =.53) Classroom cohesion (d =.53) Classroom management (d =.52) School-level Non-cognitive Variables Implementation of social-skills Decreasing disruptive behavior (d =.34) (d =.40)

24 The Programme for International Student Assessment (PISA) conducted by Organization for Economic Co-operation and Development (OECD) What to measure? How to measure?

25 Overview of PISA PISA countries PISA countries PISA countries PISA countries Out of 65, 32 OECD countries. Plus 10 additional countries Further 10 countries (e.g., Costa Rica, India, Georgia, Malaysia) carried out the same assessment in PISA countries Out of 65, 33 OECD countries.

26

27 PISA A 30-minute selfcompleted school questionnaire administered to school principals Two hours of direct students' skill assessment through reading, mathematics and science tests. A 30-minute selfcompleted contextual questionnaire administered to students to collect background information to help understand the factors contributing to student achievement. A 10-minute section on Information Technology administered to students.

28 Overview of PISA 2000: Reading 2003: Mathematics 2006: Science 2009: Reading 2012: Mathematics 2015: Science Surveys of 15-year-olds in the industrialised countries. Assessing to what extent students near the end of compulsory education have acquired some of the knowledge and skills essential for full participation in society. Every 3 year cycle, each year has a focal subject area: Reading, Mathematics, or Science Every 9 years, repeat of the focal subject Each cycle, three domains are all measured.

29 Selection of the PISA variables OECD Development Team: Research Experts Education Psychology Sociology Economics Measurement Statistics National representatives Educational Psychology Cross-cultural Psychology Organizational Psychology Social Psychology

30 Lee & Stankov (2013) PISA 2003 Noncognitive Variables: 1. Mathematics self-efficacy 2. Mathematics anxiety 3. Mathematics self-concept 4. Interest in mathematics 5. Instrumental motivation in mathematics 6. Control strategies 7. Elaboration strategies 8. Memorization strategies 9. Competitive learning 10. Co-operative learning 11. Attitudes towards school 12. Student-teacher relations at school 13. Sense of belonging to school 14. Teacher support in math lessons 15. Disciplinary climate in math class Self-beliefs Motivation Learning Strategies Attitudes toward School Lee, J., & Stankov, L. (2013). A higher-order factorial structure among fifteen major educationalpsychological constructs. Learning and Individual Differences, 26,

31 Lee & Stankov (2013)

32 Lee (2014) PISA 2009 Noncognitive Variables: 1. Enjoyment of Reading 2. Attitude towards school 3. Teacher student Relations 4. Diversity Reading 5. Online Reading 6. Metacognition: Summarising 7. Metacognition: Understanding 8. Control 9. Memorisation 10. Elaboration Attitudinal Variables Reading Habits Reading Strategies Learning Strategies Lee, J. (2014). Universal factors of student achievement in high-performing Eastern and Western countries. Journal of Educational Psychology, 106,

33 Lee (2014): Predictability of Non-cognitive Skills/Dispositions based on PISA 2009 (r =.30 to.40) Enjoyment Summarizing Understanding Personal enjoyment Text-related cognitive, metacognitive strategies (r =.20) Control Strategies Online Reading Modes of reading General learning strategies Attitudes towards School Student-teacher relationship Diversity of Reading Memorization (r = nearly zero) Elaboration Strategies General attitudes towards school and teachers

34 Stankov & Lee (2014): Beyond Educational Psychology Constructs Educational Psychology: Motivation, Learning strategies, Self-concept, Self-efficacy, Anxiety Clinical Psychology: Depression, Wellbeing, Toughness, Modesty Social Psychology: Extraversion, Neuroticism, Agreeableness, Conscientiousness, Openness to experience Cognitive Psychology: Self-assessment of intelligence, Confidence

35 Stankov & Lee (2014) Educational Psychology Cognitive Psychology r = greater than.45 Clinical/ Educational Psychology r = nearly zero Social Psychology r =.20 r = up to.35 Openness to Experience Domain-specific selfconcept r = up to.45 Self-efficacy Anxiety Self-assessment Confidence Depression Wellbeing Agreeableness Conscientiousness Self-assessment of intelligence Toughness Modesty Motivation Learning strategies Domain Task Item Extraversion Neuroticism Well-being, Personality Self-beliefs

36 Lee (in preparation): Trends in International Mathematics and Science Study (TIMSS) TIMSS data: Pan-cultural Correlations TIMSS Math Test Scores TIMSS Science Test Scores HOME POSSESSION AMOUNT OF BOOKS IN YOUR HOME SES-type PARENTS EDUCATION LEVEL variables MOTHER S EDUCATION LEVEL FATHER S EDUCATION LEVEL STUDENT SELF-EXPECTATION OF EDUCATION LEVEL Self-beliefs CONFIDENCE WITH MATH School safety FEELING SAFE IN SCHOOL

37 Trends in International Mathematics and Science Study (TIMSS) TIMSS 2003 (N = 237,833) TIMSS 2007 (N = 246,112) TIMSS 2011 (N = 287,395) Conducted by the International Association for the Evaluation of Educational Achievement s (IEA)

38 Trends in International Mathematics and Science Study (TIMSS) The TIMSS variables were analyzed in this study are: home possession; amount of books at home; parents highest education level; mother s highest education level; father s highest education level; student s self-expectation of educational level; confidence in mathematics; positive affect to mathematics; valuing mathematics; student engagement in mathematics lessons; weekly time spent on mathematics homework; time spent on extra lessons on mathematics; confidence in science; positive affect to science; valuing science; student engagement in science lessons; weekly time spent on science homework; time spent on extra lessons on science; feeling safe in school; attitude towards school; parental involvement; self-activity and homework of mathematics; mathematics-specific learning strategies; traditional learning strategies of mathematics; self-activity and homework of science; traditional learning strategies of science; science-specific learning strategies.

39 Non-cognitive Variables PISA ANXMAT: Anxiety about Mathematics 2. ATSCHL: Attitude towards School 3. BELONG: Sense of Belonging at School 4. INSTMOT: Instrumental Motivation 5. INTMAT: Interest in Mathematics 6. SCMAT: Self-concept in Math 7. MATHEFF: Self-efficacy in Math 8. SUBNORM: Subjective Norm 9. MATINTFC: Mathematics Intentions 10. MATWKETH: Math Work Ethics 11. PERSEV: Perseverance 12. OPENPS: Openness for Problem Solving All together, 21 non-cognitive variables

40 Non-cognitive Variables PISA COGACT: Cognitive Activation in Mathematics 14. CLSMAN: Teacher's Mathematics Classroom Management 15. MTSUP: Mathematics Teacher's Support 16. DISCLIMA: Disciplinary Climate 17. STUDREL: Student-teacher relationship 18. TEACHSUP: Teacher support 19. TCHBEHFA: Formative Assessment 20. TCHBEHSO: Student Orientation 21. TCHBEHTD: Teacher-directed Instruction All together, 21 non-cognitive variables

41 Lee (in preparation): PISA 2012 Predictability of Math Achievement from PISA 2012 Mathematics Self-Efficacy (r =.46) Index of economic, social and cultural status (r =.40) Math Anxiety (r =.37)

42 Lee (in preparation): PISA 2012 Predictability of Math Achievement from PISA 2012 Mathematics Self-Concept (r =.26) Wealth (r =.26) Disciplinary Climate (r =.21)

43 Lee (in preparation): PISA 2012 OPENPS Openness for Problem Solving 0.17 PERSEV Perseverance 0.11 MATINTFC Mathematics Intentions 0.08 ATSCHL Attitude towards School: Learning Outcomes 0.05 CLSMAN Mathematics Teacher's Classroom Management 0.05 BELONG Sense of Belonging to School 0.04 INSTMOT Instrumental Motivation for Mathematics 0.03 MATWKETH Mathematics Work Ethic 0.02 INTMAT Mathematics Interest 0.01 COGACT Cognitive Activation in Mathematics Lessons STUDREL Teacher Student Relations MTSUP Mathematics Teacher's Support TEACHSUP Teacher Support SUBNORM Subjective Norms in Mathematics TCHBEHTD Teacher Behaviour: Teacher-directed Instruction TCHBEHFA Teacher Behaviour: Formative Assessment TCHBEHSO Teacher Behaviour: Student Orientation -0.33

44 PISA 2012: Countries Rankings on Non-cognitive Outcomes Country ANXMAT _mean Country ATSCHL_ mean Country BELONG_ mean Country CLSMAN_ mean Country COGACT_ mean Country DISCLIMA _mean Country INSTMOT _mean Country INTMAT_ mean Country SCMAT_ mean Country MATHEFF _mean Country SUBNOR M_mean Country MATINTFC _mean MATWKE Country TH_mean Country PERSEV_ mean Country OPENPS_ mean Country MTSUP_ mean Country STUDREL _mean Country TEACHSUP_ mean Country TCHBEHFA _mean Country TCHBEHSO _mean Country TCHBEHTD _mean NETHERLAN ALBANIA LIECHTENS KAZAKHST JORDAN KAZAKHSTA PERU ALBANIA ARA BEMIR SHANGHA ALBANIA SWEDEN JORDAN KAZAKHST MONTENE ALBANIA KAZAKHST ALBANIA KAZAKHST QATAR ALBANIA SWEDEN KAZAKHSTAN AUSTRIA ALBANIA BULGARIA JAPAN MALAYSIA MALAYSIA 0.91 JORDAN LIECHTENS MALAYSIA DENMARK ARA BEMIR ALBANIA JORDAN KAZAKHST ALBANIA THAILAND BULGARIA JORDAN KAZAKHST ICELAND COSTA RICA ISRAEL COSTA RIC QATAR SHANGHAI ALBANIA KAZAKHST KAZAKHST SINGAPOR KAZAKHST VIETNAM COSTA RIC BULGARIA ALBANIA JORDAN COSTA RIC MEXICO ALBANIA ALBANIA RUSSIA DENMARK LITHUANIA COSTA RIC LITHUANIA ARA BEMIR ALBANIA MEXICO SINGAPOR QATAR GERMANY JORDAN NORWAY QATAR RUSSIA KAZAKHST RUSSIA SHANGHA KAZAKHST QATAR KAZAKHST JORDAN LIECHTENSTE COLOMBIA KAZAKHST RUSSIA UNITED ST VIETNAM JORDAN INDONESIA ISRAEL PORTUGAL INDONESIA PERU 0.25 INDONESIA COSTA RIC SERBIA SHANGHA MEXICO COSTA RIC JORDAN ARA BEMIR THAILAND GERMANY MEXICO ALBANIA JORDAN PERU RUSSIA KAZAKHST JORDAN UNITED ST SLOVENIA ARA BEMIR ISRAEL ALBANIA TURKEY QATAR SINGAPOR COLOMBIA COLOMBIA ARA BEMIR THAILAND ARA BEMIR FINLAND CHILE SWITZERLA LIECHTENS PORTUGAL HONGKONG BRAZIL THAILAND ICELAND HONGKON THAILAND LIECHTENS TUNISIA VIETNAM BULGARIA INDONESIA LITHUANIA ICELAND THAILAND COLOMBIA SHANGHA AUSTRIA INDONESIA ICELAND MEXICO KAZAKHST LITHUANIA SINGAPOR PERU CANADA HUNGARY SINGAPOR JORDAN KAZAKHST COLOMBIA POLAND ARA BEMIR INDONESIA BRAZIL RUSSIA INDONESIA BULGARIA SWITZERLAN AUSTRIA SPAIN COLOMBIA CANADA ISRAEL TUNISIA ARA BEMIR DENMARK UNITED ST PERU TURKEY SINGAPOR ARA BEMIR ARA BEMIR TUNISIA PERU JORDAN MALAYSIA BULGARIA QATAR ESTONIA SPAIN GERMANY SINGAPOR ALBANIA LIECHTENST THAILAND VIETNAM INDONESIA SWITZERLA QATAR ICELAND MALAYSIA UNITED ST ISRAEL MALAYSIA ROMANIA PORTUGAL COLOMBIA MALAYSIA COLOMBIA UNITED STAT BRAZIL COLOMBIA ROMANIA ARGENTIN AUSTRIA INDONESIA MEXICO ALBANIA TAIPEI TUNISIA MALAYSIA ISRAEL ISRAEL TUNISIA LITHUANIA SINGAPOR PERU PERU PERU INDONESIA LUXEMBOUR PERU LUXEMBOU SHANGHA UNITED KIN SINGAPORE VIETNAM QATAR SINGAPOR CZECH ISRAEL KAZAKHST SHANGHA PERU GREECE THAILAND ARA BEMIR CHILE ROMANIA TUNISIA TURKEY HUNGARY PORTUGAL CHILE ARA BEMIR 0.22 SINGAPOR KOREA COLOMBIA TUNISIA PERU LUXEMBOU MEXICO QATAR MEXICO MONTENE CHILE TURKEY JORDAN ARA BEMIR INDONESIA RUSSIA UNITED ST UNITED KING SCOTLAND URUGUAY VIETNAM COLOMBIA ESTONIA ARA BEMIR COLOMBIA COLOMBIA MACAO-CH ICELAND COSTA RIC COLOMBIA PORTUGAL ROMANIA COSTA RIC PORTUGAL INDONESIA PORTUGAL MEXICO BRAZIL LITHUANIA RUSSIA LITHUANIA MONTENE ISRAEL UNITED KIN UNITED KIN ROMANIA SWEDEN POLAND NEW ZEAL SWITZERLA UNITED ST MEXICO UNITED ST VIETNAM CANADA SINGAPOR UNITED ST BRAZIL CHILE CZECH TUNISIA TURKEY UNITED ST SCOTLAND INDONESIA ICELAND BRAZIL UNITED KIN ISRAEL CANADA THAILAND BRAZIL JORDAN COSTA RIC BRAZIL THAILAND SHANGHA BRAZIL SWEDEN MEXICO ISRAEL VIETNAM CROATIA ISRAEL CHILE IRELAND CHILE TURKEY SWITZERLA SPAIN DENMARK COLOMBIA ICELAND ESTONIA TURKEY COLOMBIA BRAZIL QATAR SINGAPOR ROMANIA PERU POLAND LIECHTENSTE HUNGARY BULGARIA ROMANIA UNITED STA QATAR SHANGHA COSTA RIC KAZAKHST UNITED KIN NETHERLA PERU SINGAPOR BRAZIL BULGARIA MALAYSIA ARGENTIN UNITED KIN COSTA RIC VIETNAM CANADA UNITED KING MEXICO MALAYSIA URUGUAY CZECH COSTA RIC DENMARK GERMANY SCOTLAND CHILE ESTONIA UNITED KIN CHILE HUNGARY MEXICO BULGARIA URUGUAY CANADA ARGENTIN MONTENE SCOTLAND HUNGARY NORWAY UNITED KIN RUSSIA MACAO-CH ISRAEL COSTA RIC MALAYSIA SLOVAKIA AUSTRALIA MEXICO THAILAND QATAR NORWAY QATAR ICELAND UNITED KIN SCOTLAND TURKEY PORTUGAL BELGIUM IRELAND SERBIA TURKEY 0.18 NEW ZEAL SCOTLAND NEW ZEAL HONGKON RUSSIA AUSTRIA TURKEY ARA BEMIR TURKEY INDONESIA PERU PERU UNITED ST MALAYSIA MONTENE CHILE SERBIA SLOVAKIA CROATIA PORTUGAL INDONESIA MEXICO HUNGARY CANADA CHILE GREECE CANADA COSTA RIC LATVIA SCOTLAND URUGUAY GERMANY TAIPEI CHILE SCOTLAND SERBIA UNITED ST ROMANIA PORTUGAL FRANCE INDONESIA PERU SHANGHA LATVIA LITHUANIA RUSSIA LIECHTENS CROATIA COLOMBIA SINGAPOR CHILE SHANGHA COLOMBIA CANADA TURKEY AUSTRALIA SHANGHA ICELAND CANADA NORWAY TURKEY ARA BEMIR CANADA LIECHTENS BELGIUM DENMARK URUGUAY SCOTLAND ICELAND UNITED ST URUGUAY CANADA THAILAND PORTUGAL UNITED ST 0.28 URUGUAY CANADA CHILE VIETNAM LATVIA LATVIA UNITED STAT JORDAN ICELAND CZECH THAILAND SCOTLAND BULGARIA NETHERLA LITHUANIA BRAZIL CHILE PORTUGAL MALAYSIA CANADA URUGUAY MONTENE RUSSIA NEW ZEAL SERBIA ISRAEL SHANGHAI NEW ZEALAN IRELAND LATVIA IRELAND POLAND PORTUGAL GREECE TUNISIA SWEDEN SCOTLAND SHANGHA DENMARK SERBIA SWEDEN PORTUGAL UNITED KIN SWEDEN TURKEY PORTUGAL ARGENTIN KAZAKHSTA AUSTRALIA MONTENE TUNISIA AUSTRALIA MEXICO AUSTRALIA ISRAEL LUXEMBOU ALBANIA LITHUANIA LUXEMBOU MONTENE CANADA INDONESIA SCOTLAND LATVIA VIETNAM AUSTRALIA DENMARK GREECE AUSTRALIA ARGENTINA DENMARK ESTONIA INDONESIA SWITZERLA URUGUAY ARGENTIN AUSTRALIA FRANCE SWEDEN AUSTRALIA AUSTRIA LATVIA ESTONIA UNITED KIN SCOTLAND BULGARIA TUNISIA NORWAY SINGAPOR NEW ZEALAN SWITZERLAN UNITED KIN SCOTLAND THAILAND COSTA RICA NORWAY ICELAND MEXICO IRELAND LIECHTENS ARGENTIN LIECHTENS MACAO-CH RUSSIA LATVIA AUSTRALIA UNITED ST ISRAEL URUGUAY TUNISIA SLOVENIA FINLAND PERU IRELAND LATVIA PORTUGAL SWEDEN MACAO-CH AUSTRIA AUSTRALIA NORWAY AUSTRIA SPAIN LITHUANIA 0.15 SLOVENIA ICELAND RUSSIA FINLAND SLOVAKIA MONTENE MALAYSIA RUSSIA MONTENEGR 0.05 SWEDEN JAPAN TUNISIA ROMANIA ARGENTIN UNITED KIN ROMANIA ESTONIA ROMANIA PORTUGAL BULGARIA BRAZIL SHANGHA ROMANIA ARGENTIN TURKEY ITALY LATVIA LIECHTENS HONGKONG ICELAND BELGIUM SWITZERLA LITHUANIA MONTENEG IRELAND SWEDEN FINLAND NORWAY BULGARIA UNITED ST BELGIUM IRELAND LUXEMBOU CHILE DENMARK NEW ZEAL LATVIA ISRAEL LITHUANIA GREECE CANADA NETHERLA THAILAND SWITZERLA TAIPEI UNITED ST 0.12 PORTUGAL ESTONIA TURKEY IRELAND UNITED KIN IRELAND TUNISIA ICELAND ISRAEL NEW ZEAL DENMARK MEXICO LITHUANIA COSTA RIC IRELAND LATVIA SLOVENIA BELGIUM GREECE ARA BEMIR LATVIA NEW ZEAL NEW ZEAL ARA BEMIR SHANGHA FRANCE ROMANIA HONGKON AUSTRIA ARGENTIN SWEDEN TAIPEI SWEDEN LIECHTENS UNITED KIN CROATIA URUGUAY UNITED ST BRAZIL BRAZIL GERMANY LIECHTENS LITHUANIA ITALY JORDAN URUGUAY JAPAN URUGUAY SPAIN DENMARK AUSTRALIA SERBIA ROMANIA NORWAY MACAO-CH URUGUAY ALBANIA ESTONIA THAILAND PORTUGAL POLAND CANADA TURKEY LIECHTENS URUGUAY UNITED KIN PORTUGAL INDONESIA MACAO-CH SCOTLAND LIECHTENS NEW ZEAL QATAR IRELAND CROATIA CANADA SLOVENIA SINGAPORE THAILAND CANADA AUSTRIA MONTENE LUXEMBOU FINLAND AUSTRALIA LITHUANIA ITALY RUSSIA CROATIA LUXEMBOU CROATIA SPAIN HONGKON ISRAEL MONTENE LIECHTENS SWITZERLA CZECH SPAIN ISRAEL SCOTLAND MACAO-CH SPAIN ICELAND GREECE UNITED ST BELGIUM BULGARIA MONTENE FINLAND NORWAY UNITED KIN URUGUAY ITALY SWITZERLA LITHUANIA AUSTRIA SINGAPOR CROATIA MACAO-CHI MALAYSIA BULGARIA URUGUAY GERMANY SPAIN SHANGHA SCOTLAND TURKEY LATVIA SWITZERLA ALBANIA ITALY SLOVENIA SINGAPOR GREECE LIECHTENS LATVIA ARGENTIN NEW ZEAL AUSTRALIA MONTENEGR ITALY FRANCE CZECH MALAYSIA COLOMBIA ESTONIA TAIPEI SHANGHA CHILE ARGENTIN BELGIUM ARGENTIN AUSTRALIA SWITZERLA MONTENE TAIPEI HONGKON SLOVENIA SLOVAKIA SCOTLAND SERBIA ROMANIA NEW ZEAL SERBIA DENMARK PERU BULGARIA CANADA IRELAND RUSSIA HONGKON BRAZIL LITHUANIA ARGENTIN IRELAND MACAO-CH IRELAND ISRAEL VIETNAM AUSTRALIA HUNGARY ARA BEMIRA ARA BEMIRA GREECE CHILE SERBIA SLOVENIA HUNGARY IRELAND SLOVENIA DENMARK FINLAND GREECE GERMANY ROMANIA SCOTLAND SERBIA HONGKON SPAIN LITHUANIA FINLAND SWITZERLA VIETNAM SLOVENIA TUNISIA SLOVENIA FINLAND ITALY SPAIN MONTENE BRAZIL BELGIUM SPAIN LITHUANIA VIETNAM ITALY CROATIA IRELAND VIETNAM MACAO-CH HUNGARY TAIPEI SWEDEN BULGARIA SINGAPORE MALAYSIA AUSTRALIA TURKEY DENMARK SWITZERLA SWITZERLA THAILAND ROMANIA LATVIA CZECH HONGKON POLAND UNITED KIN ESTONIA TUNISIA NORWAY COSTA RIC CZECH MACAO-CH TURKEY GERMANY SINGAPOR LUXEMBOU SLOVENIA TURKEY SERBIA ESTONIA BULGARIA QATAR VIETNAM SCOTLAND SWEDEN HUNGARY FINLAND CROATIA HUNGARY SWITZERLA GREECE NETHERLA GERMANY QATAR LUXEMBOUR JAPAN HONGKON ESTONIA NORWAY RUSSIA FRANCE LATVIA NEW ZEAL GREECE GERMANY SWITZERLA NEW ZEAL LATVIA SPAIN MACAO-CH LIECHTENS POLAND GERMANY FRANCE ITALY BULGARIA BRAZIL QATAR FRANCE CROATIA GERMANY ITALY SPAIN GREECE GERMANY POLAND SERBIA GERMANY ITALY FINLAND FINLAND SERBIA SWITZERLA ITALY SLOVAKIA FRANCE DENMARK FINLAND GERMANY HUNGARY SLOVAKIA POLAND LATVIA POLAND MEXICO FRANCE CANADA AUSTRALIA FINLAND AUSTRALIA DENMARK LUXEMBOU JAPAN DENMARK POLAND FINLAND INDONESIA BELGIUM AUSTRALIA CROATIA AUSTRIA SERBIA FRANCE GERMANY PORTUGAL NETHERLANDS LUXEMBOURG NEW ZEALAND SLOVENIA AUSTRIA MEXICO SWEDEN ESTONIA TUNISIA NETHERLANDS UNITED KINGDOM ICELAND KOREA JAPAN RUSSIA HUNGARY LUXEMBOURG URUGUAY CZECH FINLAND ARGENTINA SERBIA ESTONIA TUNISIA GREECE DENMARK NETHERLANDS NORWAY SPAIN ESTONIA ESTONIA JAPAN TAIPEI TAIPEI SWEDEN TAIPEI ITALY ITALY AUSTRALIA ITALY NORWAY MONTENEGRO PERU KOREA ITALY FRANCE LUXEMBOURG ARGENTINA SWITZERLAND AUSTRIA GREECE IRELAND SCOTLAND NEW ZEALAND PERU GREECE ARGENTINA SLOVAKIA CROATIA NETHERLANDS SLOVENIA POLAND NORWAY THAILAND SLOVENIA IRELAND HUNGARY TAIPEI CZECH HUNGARY KOREA BELGIUM GERMANY SPAIN IRELAND COLOMBIA SERBIA ITALY SPAIN COSTA RICA BULGARIA HONGKONG CZECH HUNGARY INDONESIA ITALY SPAIN RUSSIA CZECH LITHUANIA SLOVENIA BELGIUM SLOVAKIA TAIPEI ESTONIA AUSTRIA CHILE SHANGHAI LATVIA TAIPEI TAIPEI MALAYSIA CROATIA LUXEMBOURG CHILE MALAYSIA AUSTRIA SLOVENIA NEW ZEALAND KOREA FRANCE JAPAN NORWAY HUNGARY URUGUAY GREECE NETHERLANDS JAPAN CZECH QATAR NEW ZEALAND ICELAND CHILE MONTENEGRO SPAIN CZECH FINLAND TAIPEI RUSSIA NETHERLANDS GREECE NEW ZEALAND SLOVAKIA GREECE ITALY SPAIN KOREA BELGIUM URUGUAY TAIPEI VIETNAM NORWAY BELGIUM SWEDEN MACAO-CHINA SERBIA SERBIA URUGUAY BELGIUM MACAO-CHINA ESTONIA LIECHTENSTEIN MALAYSIA POLAND CROATIA FRANCE ICELAND SHANGHAI LUXEMBOURG ROMANIA SLOVAKIA ROMANIA DENMARK SLOVAKIA NEW ZEALAND LUXEMBOURG HUNGARY HONGKONG VIETNAM POLAND SLOVAKIA POLAND ICELAND HONGKONG FRANCE NETHERLANDS POLAND FRANCE SLOVENIA ESTONIA MALAYSIA NORWAY SLOVAKIA FRANCE NORWAY GREECE TAIPEI SLOVAKIA SLOVAKIA MONTENEGRO SERBIA TAIPEI JAPAN NETHERLANDS BELGIUM CZECH JAPAN KOREA BELGIUM LUXEMBOURG ITALY COSTA RICA KOREA SHANGHAI SWEDEN HONGKONG JORDAN SLOVAKIA KOREA FRANCE TUNISIA HUNGARY KOREA CROATIA SWITZERLAND THAILAND BELGIUM FRANCE CZECH FINLAND AUSTRIA SPAIN MEXICO JORDAN KOREA POLAND SWEDEN FRANCE BELGIUM JAPAN MACAO-CHINA COSTA RICA MACAO-CHINA BULGARIA TAIPEI SWEDEN SLOVAKIA LIECHTENSTEIN ITALY GERMANY LUXEMBOURG BELGIUM NORWAY THAILAND NETHERLANDS POLAND FINLAND NETHERLANDS FINLAND NETHERLANDS SLOVENIA VIETNAM ARGENTINA NETHERLANDS HUNGARY LATVIA BELGIUM TAIPEI KOREA SLOVAKIA CROATIA HONGKONG HONGKONG POLAND ARGENTINA QATAR ESTONIA ARGENTINA MACAO-CHINA QATAR KOREA BELGIUM CROATIA KOREA SLOVAKIA HONGKONG FINLAND NORWAY MACAO-CHINA LUXEMBOURG CZECH SLOVENIA CZECH CROATIA DENMARK JORDAN HONGKONG CZECH GREECE VIETNAM BRAZIL AUSTRIA CROATIA KOREA JAPAN CROATIA ROMANIA CZECH FRANCE KOREA GERMANY SLOVENIA LUXEMBOURG MACAO-CHINA FRANCE JAPAN BRAZIL POLAND HONGKONG KOREA JAPAN TUNISIA JAPAN AUSTRIA TAIPEI COLOMBIA CZECH SERBIA SLOVAKIA SLOVAKIA VIETNAM NETHERLANDS GERMANY AUSTRIA JAPAN HUNGARY KOREA TUNISIA MACAO-CHINA MACAO-CHINA NETHERLANDS KOREA ARGENTINA ROMANIA NETHERLANDS JAPAN BRAZIL JAPAN MONTENEGRO KOREA JAPAN JAPAN AUSTRIA POLAND NETHERLANDS KOREA IRELAND HONGKONG

45 PISA 2012: Countries Rankings on Non-cognitive Outcomes Country ANXMAT _mean Country ATSCHL_ mean Country BELONG_ mean Country CLSMAN_ mean Country COGACT_ mean Country DISCLIMA _mean Country INSTMOT _mean Country INTMAT_ mean Country SCMAT_ mean Country MATHEFF _mean Country SUBNOR M_mean Country MATINTFC _mean MATWKE Country TH_mean Country PERSEV_ mean Country OPENPS_ mean Country MTSUP_ mean Country STUDREL _mean Country TEACHSUP_ mean Country TCHBEHFA _mean Country TCHBEHSO _mean Country TCHBEHTD _mean NETHERLAN ALBANIA LIECHTENS KAZAKHST JORDAN KAZAKHSTA PERU ALBANIA ARA BEMIR SHANGHA ALBANIA SWEDEN JORDAN KAZAKHST MONTENE ALBANIA KAZAKHST ALBANIA KAZAKHST QATAR ALBANIA SWEDEN KAZAKHSTAN AUSTRIA ALBANIA BULGARIA JAPAN MALAYSIA MALAYSIA 0.91 JORDAN LIECHTENS MALAYSIA DENMARK ARA BEMIR ALBANIA JORDAN KAZAKHST ALBANIA THAILAND BULGARIA JORDAN KAZAKHST ICELAND COSTA RICA ISRAEL COSTA RIC QATAR SHANGHAI ALBANIA KAZAKHST KAZAKHST SINGAPOR KAZAKHST VIETNAM COSTA RIC BULGARIA ALBANIA JORDAN COSTA RIC MEXICO ALBANIA ALBANIA RUSSIA DENMARK LITHUANIA COSTA RIC LITHUANIA ARA BEMIR ALBANIA MEXICO SINGAPOR QATAR GERMANY JORDAN NORWAY QATAR RUSSIA KAZAKHST RUSSIA SHANGHA KAZAKHST QATAR KAZAKHST JORDAN LIECHTENSTE COLOMBIA KAZAKHST RUSSIA UNITED ST VIETNAM JORDAN INDONESIA ISRAEL PORTUGAL INDONESIA PERU 0.25 INDONESIA COSTA RIC SERBIA SHANGHA MEXICO COSTA RIC JORDAN ARA BEMIR THAILAND GERMANY MEXICO ALBANIA JORDAN PERU RUSSIA KAZAKHST JORDAN UNITED ST SLOVENIA ARA BEMIR ISRAEL ALBANIA TURKEY QATAR SINGAPOR COLOMBIA COLOMBIA ARA BEMIR THAILAND ARA BEMIR FINLAND CHILE SWITZERLA LIECHTENS PORTUGAL HONGKONG BRAZIL THAILAND ICELAND HONGKON THAILAND LIECHTENS TUNISIA VIETNAM BULGARIA INDONESIA LITHUANIA ICELAND THAILAND COLOMBIA SHANGHA AUSTRIA INDONESIA ICELAND MEXICO KAZAKHST LITHUANIA SINGAPOR PERU CANADA HUNGARY SINGAPOR JORDAN KAZAKHST COLOMBIA POLAND ARA BEMIR INDONESIA BRAZIL RUSSIA INDONESIA BULGARIA SWITZERLAN AUSTRIA SPAIN COLOMBIA CANADA ISRAEL TUNISIA ARA BEMIR DENMARK UNITED ST PERU TURKEY SINGAPOR ARA BEMIR ARA BEMIR TUNISIA PERU JORDAN MALAYSIA BULGARIA QATAR ESTONIA SPAIN GERMANY SINGAPOR ALBANIA LIECHTENST THAILAND VIETNAM INDONESIA SWITZERLA QATAR ICELAND MALAYSIA UNITED ST ISRAEL MALAYSIA ROMANIA PORTUGAL COLOMBIA MALAYSIA COLOMBIA UNITED STAT BRAZIL COLOMBIA ROMANIA ARGENTIN AUSTRIA INDONESIA MEXICO ALBANIA TAIPEI TUNISIA MALAYSIA ISRAEL ISRAEL TUNISIA LITHUANIA SINGAPOR PERU PERU PERU INDONESIA LUXEMBOUR PERU LUXEMBOU SHANGHA UNITED KIN SINGAPORE VIETNAM QATAR SINGAPOR CZECH ISRAEL KAZAKHST SHANGHA PERU GREECE THAILAND ARA BEMIR CHILE ROMANIA TUNISIA TURKEY HUNGARY PORTUGAL CHILE ARA BEMIR 0.22 SINGAPOR KOREA COLOMBIA TUNISIA PERU LUXEMBOU MEXICO QATAR MEXICO MONTENE CHILE TURKEY JORDAN ARA BEMIR INDONESIA RUSSIA UNITED ST UNITED KING SCOTLAND URUGUAY VIETNAM COLOMBIA ESTONIA ARA BEMIR COLOMBIA COLOMBIA MACAO-CH ICELAND COSTA RIC COLOMBIA PORTUGAL ROMANIA COSTA RIC PORTUGAL INDONESIA PORTUGAL MEXICO BRAZIL LITHUANIA RUSSIA LITHUANIA MONTENE ISRAEL UNITED KIN UNITED KIN ROMANIA SWEDEN POLAND NEW ZEAL SWITZERLA UNITED ST MEXICO UNITED ST VIETNAM CANADA SINGAPOR UNITED ST BRAZIL CHILE CZECH TUNISIA TURKEY UNITED ST SCOTLAND INDONESIA ICELAND BRAZIL UNITED KIN ISRAEL CANADA THAILAND BRAZIL JORDAN COSTA RIC BRAZIL THAILAND SHANGHA BRAZIL SWEDEN MEXICO ISRAEL VIETNAM CROATIA ISRAEL CHILE IRELAND CHILE TURKEY SWITZERLA SPAIN DENMARK COLOMBIA ICELAND ESTONIA TURKEY COLOMBIA BRAZIL QATAR SINGAPOR ROMANIA PERU POLAND LIECHTENSTE HUNGARY BULGARIA ROMANIA UNITED STA QATAR SHANGHA COSTA RIC KAZAKHST UNITED KIN NETHERLA PERU SINGAPOR BRAZIL BULGARIA MALAYSIA ARGENTIN UNITED KIN COSTA RIC VIETNAM CANADA UNITED KING MEXICO MALAYSIA URUGUAY CZECH COSTA RIC DENMARK GERMANY SCOTLAND CHILE ESTONIA UNITED KIN CHILE HUNGARY MEXICO BULGARIA URUGUAY CANADA ARGENTIN MONTENE SCOTLAND HUNGARY NORWAY UNITED KIN RUSSIA MACAO-CH ISRAEL COSTA RIC MALAYSIA SLOVAKIA AUSTRALIA MEXICO THAILAND QATAR NORWAY QATAR ICELAND UNITED KIN SCOTLAND TURKEY PORTUGAL BELGIUM IRELAND SERBIA TURKEY 0.18 NEW ZEAL SCOTLAND NEW ZEAL HONGKON RUSSIA AUSTRIA TURKEY ARA BEMIR TURKEY INDONESIA PERU PERU UNITED ST MALAYSIA MONTENE CHILE SERBIA SLOVAKIA CROATIA PORTUGAL INDONESIA MEXICO HUNGARY CANADA CHILE GREECE CANADA COSTA RIC LATVIA SCOTLAND URUGUAY GERMANY TAIPEI CHILE SCOTLAND SERBIA UNITED ST ROMANIA PORTUGAL FRANCE INDONESIA PERU SHANGHA LATVIA LITHUANIA RUSSIA LIECHTENS CROATIA COLOMBIA SINGAPOR CHILE SHANGHA COLOMBIA CANADA TURKEY AUSTRALIA SHANGHA ICELAND CANADA NORWAY TURKEY ARA BEMIR CANADA LIECHTENS BELGIUM DENMARK URUGUAY SCOTLAND ICELAND UNITED ST URUGUAY CANADA THAILAND PORTUGAL UNITED ST 0.28 URUGUAY CANADA CHILE VIETNAM LATVIA LATVIA UNITED STAT JORDAN ICELAND CZECH THAILAND SCOTLAND BULGARIA NETHERLA LITHUANIA BRAZIL CHILE PORTUGAL MALAYSIA CANADA URUGUAY MONTENE RUSSIA NEW ZEAL SERBIA ISRAEL SHANGHAI NEW ZEALAN IRELAND LATVIA IRELAND POLAND PORTUGAL GREECE TUNISIA SWEDEN SCOTLAND SHANGHA DENMARK SERBIA SWEDEN PORTUGAL UNITED KIN SWEDEN TURKEY PORTUGAL ARGENTIN KAZAKHSTA AUSTRALIA MONTENE TUNISIA AUSTRALIA MEXICO AUSTRALIA ISRAEL LUXEMBOU ALBANIA LITHUANIA LUXEMBOU MONTENE CANADA INDONESIA SCOTLAND LATVIA VIETNAM AUSTRALIA DENMARK GREECE AUSTRALIA ARGENTINA DENMARK ESTONIA INDONESIA SWITZERLA URUGUAY ARGENTIN AUSTRALIA FRANCE SWEDEN AUSTRALIA AUSTRIA LATVIA ESTONIA UNITED KIN SCOTLAND BULGARIA TUNISIA NORWAY SINGAPOR NEW ZEALAN SWITZERLAN UNITED KIN SCOTLAND THAILAND COSTA RICA NORWAY ICELAND MEXICO IRELAND LIECHTENS ARGENTIN LIECHTENS MACAO-CH RUSSIA LATVIA AUSTRALIA UNITED ST ISRAEL URUGUAY TUNISIA SLOVENIA FINLAND PERU IRELAND LATVIA PORTUGAL SWEDEN MACAO-CH AUSTRIA AUSTRALIA NORWAY AUSTRIA SPAIN LITHUANIA 0.15 SLOVENIA ICELAND RUSSIA FINLAND SLOVAKIA MONTENE MALAYSIA RUSSIA MONTENEGR 0.05 SWEDEN JAPAN TUNISIA ROMANIA ARGENTIN UNITED KIN ROMANIA ESTONIA ROMANIA PORTUGAL BULGARIA BRAZIL SHANGHA ROMANIA ARGENTIN TURKEY ITALY LATVIA LIECHTENS HONGKONG ICELAND BELGIUM SWITZERLA LITHUANIA MONTENEG IRELAND SWEDEN FINLAND NORWAY BULGARIA UNITED ST BELGIUM IRELAND LUXEMBOU CHILE DENMARK NEW ZEAL LATVIA ISRAEL LITHUANIA GREECE CANADA NETHERLA THAILAND SWITZERLA TAIPEI UNITED ST 0.12 PORTUGAL ESTONIA TURKEY IRELAND UNITED KIN IRELAND TUNISIA ICELAND ISRAEL NEW ZEAL DENMARK MEXICO LITHUANIA COSTA RIC IRELAND LATVIA SLOVENIA BELGIUM GREECE ARA BEMIR LATVIA NEW ZEAL NEW ZEAL ARA BEMIR SHANGHA FRANCE ROMANIA HONGKON AUSTRIA ARGENTIN SWEDEN TAIPEI SWEDEN LIECHTENS UNITED KIN CROATIA URUGUAY UNITED ST BRAZIL BRAZIL GERMANY LIECHTENS LITHUANIA ITALY JORDAN URUGUAY JAPAN URUGUAY SPAIN DENMARK AUSTRALIA SERBIA ROMANIA NORWAY MACAO-CH URUGUAY ALBANIA ESTONIA THAILAND PORTUGAL POLAND CANADA TURKEY LIECHTENS URUGUAY UNITED KIN PORTUGAL INDONESIA MACAO-CH SCOTLAND LIECHTENS NEW ZEAL QATAR IRELAND CROATIA CANADA SLOVENIA SINGAPORE THAILAND CANADA AUSTRIA MONTENE LUXEMBOU FINLAND AUSTRALIA LITHUANIA ITALY RUSSIA CROATIA LUXEMBOU CROATIA SPAIN HONGKON ISRAEL MONTENE LIECHTENS SWITZERLA CZECH SPAIN ISRAEL SCOTLAND MACAO-CH SPAIN ICELAND GREECE UNITED ST BELGIUM BULGARIA MONTENE FINLAND NORWAY UNITED KIN URUGUAY ITALY SWITZERLA LITHUANIA AUSTRIA SINGAPOR CROATIA MACAO-CHI MALAYSIA BULGARIA URUGUAY GERMANY SPAIN SHANGHA SCOTLAND TURKEY LATVIA SWITZERLA ALBANIA ITALY SLOVENIA SINGAPOR GREECE LIECHTENS LATVIA ARGENTIN NEW ZEAL AUSTRALIA MONTENEGR ITALY FRANCE CZECH MALAYSIA COLOMBIA ESTONIA TAIPEI SHANGHA CHILE ARGENTIN BELGIUM ARGENTIN AUSTRALIA SWITZERLA MONTENE TAIPEI HONGKON SLOVENIA SLOVAKIA SCOTLAND SERBIA ROMANIA NEW ZEAL SERBIA DENMARK PERU BULGARIA CANADA IRELAND RUSSIA HONGKON BRAZIL LITHUANIA ARGENTIN IRELAND MACAO-CH IRELAND ISRAEL VIETNAM AUSTRALIA HUNGARY ARA BEMIRA ARA BEMIRA GREECE CHILE SERBIA SLOVENIA HUNGARY IRELAND SLOVENIA DENMARK FINLAND GREECE GERMANY ROMANIA SCOTLAND SERBIA HONGKON SPAIN LITHUANIA FINLAND SWITZERLA VIETNAM SLOVENIA TUNISIA SLOVENIA FINLAND ITALY SPAIN MONTENE BRAZIL BELGIUM SPAIN LITHUANIA VIETNAM ITALY CROATIA IRELAND VIETNAM MACAO-CH HUNGARY TAIPEI SWEDEN BULGARIA SINGAPORE MALAYSIA AUSTRALIA TURKEY DENMARK SWITZERLA SWITZERLA THAILAND ROMANIA LATVIA CZECH HONGKON POLAND UNITED KIN ESTONIA TUNISIA NORWAY COSTA RIC CZECH MACAO-CH TURKEY GERMANY SINGAPOR LUXEMBOURG SLOVENIA TURKEY SERBIA ESTONIA BULGARIA QATAR VIETNAM SCOTLAND SWEDEN HUNGARY FINLAND CROATIA HUNGARY SWITZERLAND GREECE NETHERLANDS GERMANY QATAR LUXEMBOURG JAPAN HONGKONG ESTONIA NORWAY RUSSIA FRANCE LATVIA NEW ZEALAND GREECE GERMANY SWITZERLAND NEW ZEALAND LATVIA SPAIN MACAO-CHINA LIECHTENSTEIN POLAND GERMANY FRANCE ITALY BULGARIA BRAZIL QATAR FRANCE CROATIA GERMANY ITALY SPAIN GREECE GERMANY POLAND SERBIA GERMANY ITALY FINLAND FINLAND SERBIA SWITZERLAND ITALY SLOVAKIA FRANCE DENMARK FINLAND GERMANY HUNGARY SLOVAKIA POLAND LATVIA POLAND MEXICO FRANCE CANADA AUSTRALIA FINLAND AUSTRALIA DENMARK LUXEMBOURG JAPAN DENMARK POLAND FINLAND INDONESIA BELGIUM AUSTRALIA CROATIA AUSTRIA SERBIA FRANCE GERMANY PORTUGAL NETHERLANDS LUXEMBOURG NEW ZEALAND SLOVENIA AUSTRIA MEXICO SWEDEN ESTONIA TUNISIA NETHERLANDS UNITED KINGDOM ICELAND KOREA JAPAN RUSSIA HUNGARY LUXEMBOURG URUGUAY CZECH FINLAND ARGENTINA SERBIA ESTONIA TUNISIA GREECE DENMARK NETHERLANDS NORWAY SPAIN ESTONIA ESTONIA JAPAN TAIPEI TAIPEI SWEDEN TAIPEI ITALY ITALY AUSTRALIA ITALY NORWAY MONTENEGRO PERU KOREA ITALY FRANCE LUXEMBOURG ARGENTINA SWITZERLAND AUSTRIA GREECE IRELAND SCOTLAND NEW ZEALAND PERU GREECE ARGENTINA SLOVAKIA CROATIA NETHERLANDS SLOVENIA POLAND NORWAY THAILAND SLOVENIA IRELAND HUNGARY TAIPEI CZECH HUNGARY KOREA BELGIUM GERMANY SPAIN IRELAND COLOMBIA SERBIA ITALY SPAIN COSTA RICA BULGARIA HONGKONG CZECH HUNGARY INDONESIA ITALY SPAIN RUSSIA CZECH LITHUANIA SLOVENIA BELGIUM SLOVAKIA TAIPEI ESTONIA AUSTRIA CHILE SHANGHAI LATVIA TAIPEI TAIPEI MALAYSIA CROATIA LUXEMBOURG CHILE MALAYSIA AUSTRIA SLOVENIA NEW ZEALAND KOREA FRANCE JAPAN NORWAY HUNGARY URUGUAY GREECE NETHERLANDS JAPAN CZECH QATAR NEW ZEALAND ICELAND CHILE MONTENEGRO SPAIN CZECH FINLAND TAIPEI RUSSIA NETHERLANDS GREECE NEW ZEALAND SLOVAKIA GREECE ITALY SPAIN KOREA BELGIUM URUGUAY TAIPEI VIETNAM NORWAY BELGIUM SWEDEN MACAO-CHINA SERBIA SERBIA URUGUAY BELGIUM MACAO-CHINA ESTONIA LIECHTENSTEIN MALAYSIA POLAND CROATIA FRANCE ICELAND SHANGHAI LUXEMBOURG ROMANIA SLOVAKIA ROMANIA DENMARK SLOVAKIA NEW ZEALAND LUXEMBOURG HUNGARY HONGKONG VIETNAM POLAND SLOVAKIA POLAND ICELAND HONGKONG FRANCE NETHERLANDS POLAND FRANCE SLOVENIA ESTONIA MALAYSIA NORWAY SLOVAKIA FRANCE NORWAY GREECE TAIPEI SLOVAKIA SLOVAKIA MONTENEGRO SERBIA TAIPEI JAPAN NETHERLANDS BELGIUM CZECH JAPAN KOREA BELGIUM LUXEMBOURG ITALY COSTA RICA KOREA SHANGHAI SWEDEN HONGKONG JORDAN SLOVAKIA KOREA FRANCE TUNISIA HUNGARY KOREA CROATIA SWITZERLAND THAILAND BELGIUM FRANCE CZECH FINLAND AUSTRIA SPAIN MEXICO JORDAN KOREA POLAND SWEDEN FRANCE BELGIUM JAPAN MACAO-CHINA COSTA RICA MACAO-CHINA BULGARIA TAIPEI SWEDEN SLOVAKIA LIECHTENSTEIN ITALY GERMANY LUXEMBOURG BELGIUM NORWAY THAILAND NETHERLANDS POLAND FINLAND NETHERLANDS FINLAND NETHERLANDS SLOVENIA VIETNAM ARGENTINA NETHERLANDS HUNGARY LATVIA BELGIUM TAIPEI KOREA SLOVAKIA CROATIA HONGKONG HONGKONG POLAND ARGENTINA QATAR ESTONIA ARGENTINA MACAO-CHINA QATAR KOREA BELGIUM CROATIA KOREA SLOVAKIA HONGKONG FINLAND NORWAY MACAO-CHINA LUXEMBOURG CZECH SLOVENIA CZECH CROATIA DENMARK JORDAN HONGKONG CZECH GREECE VIETNAM BRAZIL AUSTRIA CROATIA KOREA JAPAN CROATIA ROMANIA CZECH FRANCE KOREA GERMANY SLOVENIA LUXEMBOURG MACAO-CHINA FRANCE JAPAN BRAZIL POLAND HONGKONG KOREA JAPAN TUNISIA JAPAN AUSTRIA TAIPEI COLOMBIA CZECH SERBIA SLOVAKIA SLOVAKIA VIETNAM NETHERLANDS GERMANY AUSTRIA JAPAN HUNGARY KOREA TUNISIA MACAO-CHINA MACAO-CHINA NETHERLANDS KOREA ARGENTINA ROMANIA NETHERLANDS JAPAN BRAZIL JAPAN MONTENEGRO KOREA JAPAN JAPAN AUSTRIA POLAND NETHERLANDS KOREA IRELAND HONGKONG

46 Number of Times Appeared in the Bottom 15 Country List in 21 Non-cognitive Variables (WORST) KOREA 20 JAPAN 15 SLOVAKIA 15 TAIPEI 13 BELGIUM 12 CZECH 12 NETHERLANDS 12 HONGKONG 11

47 Number of Times Appeared in the Bottom 15 Country List in 21 Non-cognitive Variables ARAB EMIRATES 0 CANADA 0 PORTUGAL 0 TURKEY 0 UNITED STATES 0 (BEST) AUSTRALIA 1 ALBANIA 1 BULGARIA 1 INDONESIA 1 ISRAEL 1 KAZAKHSTAN 1 MEXICO 1 SCOTLAND 1 SINGAPORE 1 UNITED KINGDOM 1

48 PISA 2012: Countries Rankings on Non-cognitive Outcomes Country ANXMAT _mean Country ATSCHL_ mean Country BELONG_ mean Country CLSMAN_ mean Country COGACT_ mean Country DISCLIMA _mean Country INSTMOT _mean Country INTMAT_ mean Country SCMAT_ mean Country MATHEFF _mean Country SUBNOR M_mean Country MATINTFC _mean MATWKE Country TH_mean Country PERSEV_ mean Country OPENPS_ mean Country MTSUP_ mean Country STUDREL _mean Country TEACHSUP_ mean Country TCHBEHFA _mean Country TCHBEHSO _mean Country TCHBEHTD _mean NETHERLAN ALBANIA LIECHTENS KAZAKHST JORDAN KAZAKHSTA PERU ALBANIA ARA BEMIR SHANGHA ALBANIA SWEDEN JORDAN KAZAKHST MONTENE ALBANIA KAZAKHST ALBANIA KAZAKHST QATAR ALBANIA SWEDEN KAZAKHSTAN AUSTRIA ALBANIA BULGARIA JAPAN MALAYSIA MALAYSIA 0.91 JORDAN LIECHTENS MALAYSIA DENMARK ARA BEMIR ALBANIA JORDAN KAZAKHST ALBANIA THAILAND BULGARIA JORDAN KAZAKHST ICELAND COSTA RICA ISRAEL COSTA RIC QATAR SHANGHAI ALBANIA KAZAKHST KAZAKHST SINGAPOR KAZAKHST VIETNAM COSTA RIC BULGARIA ALBANIA JORDAN COSTA RIC MEXICO ALBANIA ALBANIA RUSSIA DENMARK LITHUANIA COSTA RIC LITHUANIA ARA BEMIR ALBANIA MEXICO SINGAPOR QATAR GERMANY JORDAN NORWAY QATAR RUSSIA KAZAKHST RUSSIA SHANGHA KAZAKHST QATAR KAZAKHST JORDAN LIECHTENSTE COLOMBIA KAZAKHST RUSSIA UNITED ST VIETNAM JORDAN INDONESIA ISRAEL PORTUGAL INDONESIA PERU 0.25 INDONESIA COSTA RIC SERBIA SHANGHA MEXICO COSTA RIC JORDAN ARA BEMIR THAILAND GERMANY MEXICO ALBANIA JORDAN PERU RUSSIA KAZAKHST JORDAN UNITED ST SLOVENIA ARA BEMIR ISRAEL ALBANIA TURKEY QATAR SINGAPOR COLOMBIA COLOMBIA ARA BEMIR THAILAND ARA BEMIR FINLAND CHILE SWITZERLA LIECHTENS PORTUGAL HONGKONG BRAZIL THAILAND ICELAND HONGKON THAILAND LIECHTENS TUNISIA VIETNAM BULGARIA INDONESIA LITHUANIA ICELAND THAILAND COLOMBIA SHANGHA AUSTRIA INDONESIA ICELAND MEXICO KAZAKHST LITHUANIA SINGAPOR PERU CANADA HUNGARY SINGAPOR JORDAN KAZAKHST COLOMBIA POLAND ARA BEMIR INDONESIA BRAZIL RUSSIA INDONESIA BULGARIA SWITZERLAN AUSTRIA SPAIN COLOMBIA CANADA ISRAEL TUNISIA ARA BEMIR DENMARK UNITED ST PERU TURKEY SINGAPOR ARA BEMIR ARA BEMIR TUNISIA PERU JORDAN MALAYSIA BULGARIA QATAR ESTONIA SPAIN GERMANY SINGAPOR ALBANIA LIECHTENST THAILAND VIETNAM INDONESIA SWITZERLA QATAR ICELAND MALAYSIA UNITED ST ISRAEL MALAYSIA ROMANIA PORTUGAL COLOMBIA MALAYSIA COLOMBIA UNITED STAT BRAZIL COLOMBIA ROMANIA ARGENTIN AUSTRIA INDONESIA MEXICO ALBANIA TAIPEI TUNISIA MALAYSIA ISRAEL ISRAEL TUNISIA LITHUANIA SINGAPOR PERU PERU PERU INDONESIA LUXEMBOUR PERU LUXEMBOU SHANGHA UNITED KIN SINGAPORE VIETNAM QATAR SINGAPOR CZECH ISRAEL KAZAKHST SHANGHA PERU GREECE THAILAND ARA BEMIR CHILE ROMANIA TUNISIA TURKEY HUNGARY PORTUGAL CHILE ARA BEMIR 0.22 SINGAPOR KOREA COLOMBIA TUNISIA PERU LUXEMBOU MEXICO QATAR MEXICO MONTENE CHILE TURKEY JORDAN ARA BEMIR INDONESIA RUSSIA UNITED ST UNITED KING SCOTLAND URUGUAY VIETNAM COLOMBIA ESTONIA ARA BEMIR COLOMBIA COLOMBIA MACAO-CH ICELAND COSTA RIC COLOMBIA PORTUGAL ROMANIA COSTA RIC PORTUGAL INDONESIA PORTUGAL MEXICO BRAZIL LITHUANIA RUSSIA LITHUANIA MONTENE ISRAEL UNITED KIN UNITED KIN ROMANIA SWEDEN POLAND NEW ZEAL SWITZERLA UNITED ST MEXICO UNITED ST VIETNAM CANADA SINGAPOR UNITED ST BRAZIL CHILE CZECH TUNISIA TURKEY UNITED ST SCOTLAND INDONESIA ICELAND BRAZIL UNITED KIN ISRAEL CANADA THAILAND BRAZIL JORDAN COSTA RIC BRAZIL THAILAND SHANGHA BRAZIL SWEDEN MEXICO ISRAEL VIETNAM CROATIA ISRAEL CHILE IRELAND CHILE TURKEY SWITZERLA SPAIN DENMARK COLOMBIA ICELAND ESTONIA TURKEY COLOMBIA BRAZIL QATAR SINGAPOR ROMANIA PERU POLAND LIECHTENSTE HUNGARY BULGARIA ROMANIA UNITED STA QATAR SHANGHA COSTA RIC KAZAKHST UNITED KIN NETHERLA PERU SINGAPOR BRAZIL BULGARIA MALAYSIA ARGENTIN UNITED KIN COSTA RIC VIETNAM CANADA UNITED KING MEXICO MALAYSIA URUGUAY CZECH COSTA RIC DENMARK GERMANY SCOTLAND CHILE ESTONIA UNITED KIN CHILE HUNGARY MEXICO BULGARIA URUGUAY CANADA ARGENTIN MONTENE SCOTLAND HUNGARY NORWAY UNITED KIN RUSSIA MACAO-CH ISRAEL COSTA RIC MALAYSIA SLOVAKIA AUSTRALIA MEXICO THAILAND QATAR NORWAY QATAR ICELAND UNITED KIN SCOTLAND TURKEY PORTUGAL BELGIUM IRELAND SERBIA TURKEY 0.18 NEW ZEAL SCOTLAND NEW ZEAL HONGKON RUSSIA AUSTRIA TURKEY ARA BEMIR TURKEY INDONESIA PERU PERU UNITED ST MALAYSIA MONTENE CHILE SERBIA SLOVAKIA CROATIA PORTUGAL INDONESIA MEXICO HUNGARY CANADA CHILE GREECE CANADA COSTA RIC LATVIA SCOTLAND URUGUAY GERMANY TAIPEI CHILE SCOTLAND SERBIA UNITED ST ROMANIA PORTUGAL FRANCE INDONESIA PERU SHANGHA LATVIA LITHUANIA RUSSIA LIECHTENS CROATIA COLOMBIA SINGAPOR CHILE SHANGHA COLOMBIA CANADA TURKEY AUSTRALIA SHANGHA ICELAND CANADA NORWAY TURKEY ARA BEMIR CANADA LIECHTENS BELGIUM DENMARK URUGUAY SCOTLAND ICELAND UNITED ST URUGUAY CANADA THAILAND PORTUGAL UNITED ST 0.28 URUGUAY CANADA CHILE VIETNAM LATVIA LATVIA UNITED STAT JORDAN ICELAND CZECH THAILAND SCOTLAND BULGARIA NETHERLA LITHUANIA BRAZIL CHILE PORTUGAL MALAYSIA CANADA URUGUAY MONTENE RUSSIA NEW ZEAL SERBIA ISRAEL SHANGHAI NEW ZEALAN IRELAND LATVIA IRELAND POLAND PORTUGAL GREECE TUNISIA SWEDEN SCOTLAND SHANGHA DENMARK SERBIA SWEDEN PORTUGAL UNITED KIN SWEDEN TURKEY PORTUGAL ARGENTIN KAZAKHSTA AUSTRALIA MONTENE TUNISIA AUSTRALIA MEXICO AUSTRALIA ISRAEL LUXEMBOU ALBANIA LITHUANIA LUXEMBOU MONTENE CANADA INDONESIA SCOTLAND LATVIA VIETNAM AUSTRALIA DENMARK GREECE AUSTRALIA ARGENTINA DENMARK ESTONIA INDONESIA SWITZERLA URUGUAY ARGENTIN AUSTRALIA FRANCE SWEDEN AUSTRALIA AUSTRIA LATVIA ESTONIA UNITED KIN SCOTLAND BULGARIA TUNISIA NORWAY SINGAPOR NEW ZEALAN SWITZERLAN UNITED KIN SCOTLAND THAILAND COSTA RICA NORWAY ICELAND MEXICO IRELAND LIECHTENS ARGENTIN LIECHTENS MACAO-CH RUSSIA LATVIA AUSTRALIA UNITED ST ISRAEL URUGUAY TUNISIA SLOVENIA FINLAND PERU IRELAND LATVIA PORTUGAL SWEDEN MACAO-CH AUSTRIA AUSTRALIA NORWAY AUSTRIA SPAIN LITHUANIA 0.15 SLOVENIA ICELAND RUSSIA FINLAND SLOVAKIA MONTENE MALAYSIA RUSSIA MONTENEGR 0.05 SWEDEN JAPAN TUNISIA ROMANIA ARGENTIN UNITED KIN ROMANIA ESTONIA ROMANIA PORTUGAL BULGARIA BRAZIL SHANGHA ROMANIA ARGENTIN TURKEY ITALY LATVIA LIECHTENS HONGKONG ICELAND BELGIUM SWITZERLA LITHUANIA MONTENEG IRELAND SWEDEN FINLAND NORWAY BULGARIA UNITED ST BELGIUM IRELAND LUXEMBOU CHILE DENMARK NEW ZEAL LATVIA ISRAEL LITHUANIA GREECE CANADA NETHERLA THAILAND SWITZERLA TAIPEI UNITED ST 0.12 PORTUGAL ESTONIA TURKEY IRELAND UNITED KIN IRELAND TUNISIA ICELAND ISRAEL NEW ZEAL DENMARK MEXICO LITHUANIA COSTA RIC IRELAND LATVIA SLOVENIA BELGIUM GREECE ARA BEMIR LATVIA NEW ZEAL NEW ZEAL ARA BEMIR SHANGHA FRANCE ROMANIA HONGKON AUSTRIA ARGENTIN SWEDEN TAIPEI SWEDEN LIECHTENS UNITED KIN CROATIA URUGUAY UNITED ST BRAZIL BRAZIL GERMANY LIECHTENS LITHUANIA ITALY JORDAN URUGUAY JAPAN URUGUAY SPAIN DENMARK AUSTRALIA SERBIA ROMANIA NORWAY MACAO-CH URUGUAY ALBANIA ESTONIA THAILAND PORTUGAL POLAND CANADA TURKEY LIECHTENS URUGUAY UNITED KIN PORTUGAL INDONESIA MACAO-CH SCOTLAND LIECHTENS NEW ZEAL QATAR IRELAND CROATIA CANADA SLOVENIA SINGAPORE THAILAND CANADA AUSTRIA MONTENE LUXEMBOU FINLAND AUSTRALIA LITHUANIA ITALY RUSSIA CROATIA LUXEMBOU CROATIA SPAIN HONGKON ISRAEL MONTENE LIECHTENS SWITZERLA CZECH SPAIN ISRAEL SCOTLAND MACAO-CH SPAIN ICELAND GREECE UNITED ST BELGIUM BULGARIA MONTENE FINLAND NORWAY UNITED KIN URUGUAY ITALY SWITZERLA LITHUANIA AUSTRIA SINGAPOR CROATIA MACAO-CHI MALAYSIA BULGARIA URUGUAY GERMANY SPAIN SHANGHA SCOTLAND TURKEY LATVIA SWITZERLA ALBANIA ITALY SLOVENIA SINGAPOR GREECE LIECHTENS LATVIA ARGENTIN NEW ZEAL AUSTRALIA MONTENEGR ITALY FRANCE CZECH MALAYSIA COLOMBIA ESTONIA TAIPEI SHANGHA CHILE ARGENTIN BELGIUM ARGENTIN AUSTRALIA SWITZERLA MONTENE TAIPEI HONGKON SLOVENIA SLOVAKIA SCOTLAND SERBIA ROMANIA NEW ZEAL SERBIA DENMARK PERU BULGARIA CANADA IRELAND RUSSIA HONGKON BRAZIL LITHUANIA ARGENTIN IRELAND MACAO-CH IRELAND ISRAEL VIETNAM AUSTRALIA HUNGARY ARA BEMIRA ARA BEMIRA GREECE CHILE SERBIA SLOVENIA HUNGARY IRELAND SLOVENIA DENMARK FINLAND GREECE GERMANY ROMANIA SCOTLAND SERBIA HONGKON SPAIN LITHUANIA FINLAND SWITZERLA VIETNAM SLOVENIA TUNISIA SLOVENIA FINLAND ITALY SPAIN MONTENE BRAZIL BELGIUM SPAIN LITHUANIA VIETNAM ITALY CROATIA IRELAND VIETNAM MACAO-CH HUNGARY TAIPEI SWEDEN BULGARIA SINGAPORE MALAYSIA AUSTRALIA TURKEY DENMARK SWITZERLA SWITZERLA THAILAND ROMANIA LATVIA CZECH HONGKON POLAND UNITED KIN ESTONIA TUNISIA NORWAY COSTA RIC CZECH MACAO-CH TURKEY GERMANY SINGAPOR LUXEMBOU SLOVENIA TURKEY SERBIA ESTONIA BULGARIA QATAR VIETNAM SCOTLAND SWEDEN HUNGARY FINLAND CROATIA HUNGARY SWITZERLA GREECE NETHERLA GERMANY QATAR LUXEMBOUR JAPAN HONGKON ESTONIA NORWAY RUSSIA FRANCE LATVIA NEW ZEAL GREECE GERMANY SWITZERLA NEW ZEAL LATVIA SPAIN MACAO-CH LIECHTENSTEIN POLAND GERMANY FRANCE ITALY BULGARIA BRAZIL QATAR FRANCE CROATIA GERMANY ITALY SPAIN GREECE GERMANY POLAND SERBIA GERMANY ITALY FINLAND FINLAND SERBIA SWITZERLAND ITALY SLOVAKIA FRANCE DENMARK FINLAND GERMANY HUNGARY SLOVAKIA POLAND LATVIA POLAND MEXICO FRANCE CANADA AUSTRALIA FINLAND AUSTRALIA DENMARK LUXEMBOURG JAPAN DENMARK POLAND FINLAND INDONESIA BELGIUM AUSTRALIA CROATIA AUSTRIA SERBIA FRANCE GERMANY PORTUGAL NETHERLANDS LUXEMBOURG NEW ZEALAND SLOVENIA AUSTRIA MEXICO SWEDEN ESTONIA TUNISIA NETHERLANDS UNITED KINGDOM ICELAND KOREA JAPAN RUSSIA HUNGARY LUXEMBOURG URUGUAY CZECH FINLAND ARGENTINA SERBIA ESTONIA TUNISIA GREECE DENMARK NETHERLANDS NORWAY SPAIN ESTONIA ESTONIA JAPAN TAIPEI TAIPEI SWEDEN TAIPEI ITALY ITALY AUSTRALIA ITALY NORWAY MONTENEGRO PERU KOREA ITALY FRANCE LUXEMBOURG ARGENTINA SWITZERLAND AUSTRIA GREECE IRELAND SCOTLAND NEW ZEALAND PERU GREECE ARGENTINA SLOVAKIA CROATIA NETHERLANDS SLOVENIA POLAND NORWAY THAILAND SLOVENIA IRELAND HUNGARY TAIPEI CZECH HUNGARY KOREA BELGIUM GERMANY SPAIN IRELAND COLOMBIA SERBIA ITALY SPAIN COSTA RICA BULGARIA HONGKONG CZECH HUNGARY INDONESIA ITALY SPAIN RUSSIA CZECH LITHUANIA SLOVENIA BELGIUM SLOVAKIA TAIPEI ESTONIA AUSTRIA CHILE SHANGHAI LATVIA TAIPEI TAIPEI MALAYSIA CROATIA LUXEMBOURG CHILE MALAYSIA AUSTRIA SLOVENIA NEW ZEALAND KOREA FRANCE JAPAN NORWAY HUNGARY URUGUAY GREECE NETHERLANDS JAPAN CZECH QATAR NEW ZEALAND ICELAND CHILE MONTENEGRO SPAIN CZECH FINLAND TAIPEI RUSSIA NETHERLANDS GREECE NEW ZEALAND SLOVAKIA GREECE ITALY SPAIN KOREA BELGIUM URUGUAY TAIPEI VIETNAM NORWAY BELGIUM SWEDEN MACAO-CHINA SERBIA SERBIA URUGUAY BELGIUM MACAO-CHINA ESTONIA LIECHTENSTEIN MALAYSIA POLAND CROATIA FRANCE ICELAND SHANGHAI LUXEMBOURG ROMANIA SLOVAKIA ROMANIA DENMARK SLOVAKIA NEW ZEALAND LUXEMBOURG HUNGARY HONGKONG VIETNAM POLAND SLOVAKIA POLAND ICELAND HONGKONG FRANCE NETHERLANDS POLAND FRANCE SLOVENIA ESTONIA MALAYSIA NORWAY SLOVAKIA FRANCE NORWAY GREECE TAIPEI SLOVAKIA SLOVAKIA MONTENEGRO SERBIA TAIPEI JAPAN NETHERLANDS BELGIUM CZECH JAPAN KOREA BELGIUM LUXEMBOURG ITALY COSTA RICA KOREA SHANGHAI SWEDEN HONGKONG JORDAN SLOVAKIA KOREA FRANCE TUNISIA HUNGARY KOREA CROATIA SWITZERLAND THAILAND BELGIUM FRANCE CZECH FINLAND AUSTRIA SPAIN MEXICO JORDAN KOREA POLAND SWEDEN FRANCE BELGIUM JAPAN MACAO-CHINA COSTA RICA MACAO-CHINA BULGARIA TAIPEI SWEDEN SLOVAKIA LIECHTENSTEIN ITALY GERMANY LUXEMBOURG BELGIUM NORWAY THAILAND NETHERLANDS POLAND FINLAND NETHERLANDS FINLAND NETHERLANDS SLOVENIA VIETNAM ARGENTINA NETHERLANDS HUNGARY LATVIA BELGIUM TAIPEI KOREA SLOVAKIA CROATIA HONGKONG HONGKONG POLAND ARGENTINA QATAR ESTONIA ARGENTINA MACAO-CHINA QATAR KOREA BELGIUM CROATIA KOREA SLOVAKIA HONGKONG FINLAND NORWAY MACAO-CHINA LUXEMBOURG CZECH SLOVENIA CZECH CROATIA DENMARK JORDAN HONGKONG CZECH GREECE VIETNAM BRAZIL AUSTRIA CROATIA KOREA JAPAN CROATIA ROMANIA CZECH FRANCE KOREA GERMANY SLOVENIA LUXEMBOURG MACAO-CHINA FRANCE JAPAN BRAZIL POLAND HONGKONG KOREA JAPAN TUNISIA JAPAN AUSTRIA TAIPEI COLOMBIA CZECH SERBIA SLOVAKIA SLOVAKIA VIETNAM NETHERLANDS GERMANY AUSTRIA JAPAN HUNGARY KOREA TUNISIA MACAO-CHINA MACAO-CHINA NETHERLANDS KOREA ARGENTINA ROMANIA NETHERLANDS JAPAN BRAZIL JAPAN MONTENEGRO KOREA JAPAN JAPAN AUSTRIA POLAND NETHERLANDS KOREA IRELAND HONGKONG

49 Non-cognitive Variables PISA Anxiety about Mathematics 2. Attitude towards School 3. Sense of Belonging at School 4. Instrumental Motivation 5. Interest in Mathematics 6. Self-concept in Math 7. Self-efficacy in Math 8. Subjective Norm 9. Mathematics Intentions 10. Math Work Ethics 11. Perseverance 12. Openness for Problem Solving 13. Cognitive Activation in Mathematics 14. Classroom Management 15. Mathematics Teacher's Support 16. Disciplinary Climate 17. Student-teacher relationship 18. Teacher support 19. Formative Assessment 20. Student Orientation 21. Teacher-directed Instruction

50 Non-cognitive Variables PISA Anxiety about Mathematics 2. Attitude towards School 3. Sense of Belonging at School 4. Instrumental Motivation 5. Interest in Mathematics 6. Self-concept in Math 7. Self-efficacy in Math 8. Subjective Norm 9. Mathematics Intentions 10. Math Work Ethics 11. Perseverance 12. Openness for Problem Solving 13. Cognitive Activation in Mathematics 14. Classroom Management 15. Mathematics Teacher's Support 16. Disciplinary Climate 17. Student-teacher relationship 18. Teacher support 19. Formative Assessment 20. Student Orientation 21. Teacher-directed Instruction

51 Classroom Management & Disciplinary Climate Mathematics Teacher's Classroom Management Thinking about the mathematics teacher who taught your last mathematics class: to what extent do you agree with the following statements? My teacher gets students to listen to him or her (+); My teacher keeps the class orderly (+); My teacher starts lessons on time (+); The teacher has to wait a long time for students to <quiet down> (-) Disciplinary climate in math lessons Students don t listen to what the teacher says (-); There is noise and disorder (-); The teacher has to wait a long time for students to quieten down (-); Students cannot work well (-); Students don t start working for a long time after the lesson begins (-). 2012, 2009, 2003

52 Sense of Belonging to School Sense of Belonging to School I feel like an outsider (or left out of things) at school (-); I make friends easily at school (+); I feel like I belong at school (+); I feel awkward or out of place in my school (-); Other students seem to like me (+); I feel lonely at school (-); I feel happy at school (+); Things are ideal in my school (+); I am satisfied with my school (+). 2012, 2003

53 Work Ethic & Openness for Problem Solving Mathematics Work Ethic Openness for Problem Solving I finish my homework in time for mathematics class (+); I work hard on my mathematics homework (+); I am prepared for my mathematics exams (+); I study hard for mathematics quizzes (+); I keep studying until I understand mathematics material (+); I pay attention in mathematics class (+); I listen in mathematics class (+); I avoid distractions when I am studying mathematics (+); I keep my mathematics work well organized (+). I can handle a lot of information (+); I am quick to understand things (+); I seek explanations of things (+); I can easily link facts together (+); I like to solve complex problems (+)

54 Part 2. Any substantial breakthrough in the assessment of non-cognitive constructs for the past 30 years?

55 Any substantial breakthrough? Recent research effort on self-report questionnaire, adjustment of rating-scale responses Issue of measurement unit in assessing noncognitive constructs An IRT application in assessing confidence

56 Strategies Tried in PISA 2012 Field Trial Based on 63 countries (N = 200 ~ 1000 per country) 1. Anchoring Vignettes (King at et al., 2004; Mottus et al., 2012) 2. Forced-choice (Berkshire & Highland, 1953; Landy, F. J., & Farr, 1980; Bartram, 2013) 3. Signal-detection correction (cf. Green & Swets, 1966; Swets, 2014) A list of mathematics concepts such as logarithm, rational number, divisor, mixed in with several foils (e.g., subjunctive scaling) 4. Behavioural items (me ) 5. Situational-judgment tests (e.g., Lievens & Sackett, 2006)

57 Levels of Measurement: Self-concept, Self-efficacy, Confidence Item Task Domain Confidence Self-efficacy Self-concept

58 Typical Measurement of Self-Concept 1. I am just not good at mathematics 2. I get good <grades> in mathematics 3. I learn mathematics quickly 4. I have always believed that mathematics is one of my best subjects 5. In my mathematics class, I understand even the most difficult work

59 Typical Measurement of Self-Efficacy 1. Using a <train timetable> to work out how long it would take to get from one place to another 2. Calculating how much cheaper a TV would be after a 30% discount 3. Calculating how many square metres of tiles you need to cover a floor 4. Understanding graphs presented in newspapers; 5. Solving an equation like 3x+5=17 6. Finding the actual distance between two places on a map with a 1:10_000 scale 7. Solving an equation like 2(x+3)=(x+3)(x-3) 8. Calculating the petrol consumption rate of a car

60 Typical Measurement of Confidence

61 Blue line: Confidence Red line: Accuracy Medium-level Difficulty Item Overconfidence for participants of low ability; Good calibration; Slight under-confidence for participants of high ability.

62 Blue line: Confidence Red line: Accuracy Difficult Item People are overconfident across all ability levels.

63 Blue line: Confidence Red line: Accuracy Easy Item Most people are under-confident when judging their performance on an easy item. Stankov, L., Lee, J., & Paek, I. (2009). Realism of confidence judgment. European Journal of Psychological Assessment, 25,

64 So, has there been substantial breakthrough in the assessment of the noncognitive domain for the past 30 years?

65 Conclusion: Three Take-home Messages

66 Three Take-home Messages 1. What to measure? A group of Self-beliefs: Confidence (accurate selfassessment), Self-efficacy, Anxiety 2.How to measure? Self-report, close to the tasks, better at the item-level 3.How much overlap is there between cognitive and non-cognitive domains? Cognitive Noncognitive

67 Lee (2009) Lee, J. (2009). Universals and specifics of math self-concept, math self-efficacy, and math anxiety across 41 PISA 2003 participating countries. Learning and Individual Differences, 19, Lee & Shute (2010) Lee, J., & Shute, V. (2010). Personal and social-contextual factors in K-12 academic performance: An integrative perspective on student learning. Educational Psychologist, 45, Stankov, Lee, Luo, & Hogan (2012). Stankov, L., Lee, J., Luo, W., & Hogan, D. (2012). Confidence: A better predictor of academic achievement than self-efficacy, self-concept and anxiety? Learning and Individual Differences, 22, Lee & Stankov (2013) Lee, J. & Stankov, L. (2013). A higher-order factorial structure among fifteen major educational-psychological constructs. Learning and Individual Differences, 26, Lee (2014) Lee, J. (2014). Universal factors of student achievement in high-performing Eastern and Western countries. Journal of Educational Psychology, 106, Stankov & Lee (2014) Stankov, L., & Lee, J. (2014). Quest for the best noncognitive predictor of academic achievement. Educational Psychology: An international journal of experimental educational psychology, 34, 1-8. Lee & Paek (2014) Lee, J., & Paek, I. (2014). In search of the optimal number of response categories using IRT graded response modeling. Journal of Psychoeducational Assessment, 32,

68 Thank you!