Measurement of Non-cognitive Constructs in School Contexts: Implications from International and Large-scale Testing Programs

Measurement of Non-cognitive Constructs in School Contexts: Implications from International and Large-scale Testing Programs Jihyun Lee (Ph.D.) School of Education, University of New South Wales (UNSW) Sydney, Australia jihyun.lee@unsw.edu.au Presented at the 2015 Assessment in Schools Conference UNSW Global - EAA, 2nd Floor, Training Room 3 October 12, 2015

Outline of My Talk 1. Part 1. Research on what to measure in non-cognitive domain (80%) My own and others research Large-scale, international assessments 2. Part 2. Measurement/assessment of noncognitive constructs (20%) Any substantial breakthrough? 3. Conclusion. Three take-home messages

WOW! THIS IS THE BEST CLASS THIS WEEK! KIDS ARE ENJOYING SO MUCH!

Part 1. Non-cognitive What skills or dispositions should we measure (know about)?

Non-cognitive Skills? Defining non-cognitive any skills, behaviors, habits, tendencies, traits, and disposition that are typically not measured by <cognitive, achievement> tests. In Education: motivational, attitudinal, emotional, interpersonal characteristics and personality (Farkas, 2003, Annu. Rev. Sociology, Vol. 29, p.542)

Challenges in Research in Non-cognitive Skills Some argue that non-cognitive traits are hardly non-cognitive. Non-cognitive qualities cannot be developed without proper cognition. Cognition cannot be triggered without non-cognitive qualities. Cognitive Noncognitive Cognitive Noncognitive Construct /Domain Independence Boundary of non-cognitive domains (Farkas, 2003, Annu. Rev. Sociology, Vol. 29, p.542)

Challenges in Research in Non-cognitive Skills Flexibility Critical thinking Creativity Openmindedness Selfmanagement Motivation Problem-solving Collaboration Metacognition

National Council on Measurement in Education (2012) Based on the commonality across different frameworks Constructs with relatively mature research base: C3M2 Critical Thinking Creativity Collaboration Metacognition Motivation

Lee & Shute (2010) adaptability, attitude, affect, anxiety, belief, basic skills, classroom behaviors, control strategies, cooperation, curiosity, discipline, effort, elaboration strategies, emotion, engagement, extracurricular activity, habit, help-seeking, homework, independence, interest, leadership, Any particular learning strategies, learning style, liking, memorization, metacognition, motivation, non-cognitive, note-taking, organizational skills, parenting, parental involvement, non-cognitive peer, peer-teaching, perseverance, principal leadership, resilience, school climate, school leadership, school organization, sense of belonging, self-confidence, self-concept, selfcontrol, self-direction, self-discipline, self-efficacy, self-regulation, qualities of a learner? social competence, social context, social development, social relation, study habits, student-teacher relationship, teacher efficacy, teacher support, teacher relation, teamwork, test anxiety, test-taking strategies, thinking skills, time management, time spent on tasks, and value Lee, J., & Shute, V. (2010). Personal and social-contextual factors in K-12 academic performance: An integrative perspective on student learning. Educational Psychologist, 45, 185-202.

Lee & Shute (2010) Student-Personal Factor Social-Contextual Factor Student Engagement Learning Strategies School Climate Social-Family Influences Behavior Cognition Academic emphasis Parents Cognition Metacognition Teacher Peers Affect Behavior Leadership

Lee & Shute (2010) Student-Personal Factor Student Engagement Learning Strategies Behavior Cognition Affect Cognition Metacognition Behavior Attendance Intrinsic motivation Interest, curiosity Summarize Monitor Selfmanagement Punctuality Effort Sense of belonging Understand Regulate Strategies (e.g., notetaking) Participation Selfbeliefs Emotion Elaborate Evaluate Help-seeking

Lee & Shute (2010) Social-Contextual Factor School Climate Social-Family Influences Academic Teacher Leadership Parental Involvement Peer Influences Value in Academic Work Collective efficacy Collegiality Attitude Support Expectation Teacher empowerment High morale Behaviour Norms Positive response Sense of affiliation Goalsetting Style Attitude & Behavior

Hattie s (2009) Visible Learning Effect sizes of Non-cognitive Variables

Hattie (2009) Engagement (d =.48) Self-concept (d =.43) Student-level Non-cognitive Variables Anxiety (d =.40) Attitude to mathematics /science (d =.36) Personality (d =.19)

Hattie (2009) Teacher clarity (d =.75) Teacher-student relationships (d =.72) Not labeling students (d =.61) Teacher-level Non-cognitive Variables Teacher expectation (d =.43)

Classroom behavior (d =.80) Hattie (2009) Peer influences (d =.53) Classroom cohesion (d =.53) Classroom management (d =.52) School-level Non-cognitive Variables Implementation of social-skills Decreasing disruptive behavior (d =.34) (d =.40)

The Programme for International Student Assessment (PISA) conducted by Organization for Economic Co-operation and Development (OECD) What to measure? How to measure?

Overview of PISA PISA 2003 41 countries PISA 2000 43 countries PISA 2006 57 countries PISA 2009 65 countries Out of 65, 32 OECD countries. Plus 10 additional countries 2010. Further 10 countries (e.g., Costa Rica, India, Georgia, Malaysia) carried out the same assessment in 2010. PISA 2012 65 countries Out of 65, 33 OECD countries.

PISA A 30-minute selfcompleted school questionnaire administered to school principals Two hours of direct students' skill assessment through reading, mathematics and science tests. A 30-minute selfcompleted contextual questionnaire administered to students to collect background information to help understand the factors contributing to student achievement. A 10-minute section on Information Technology administered to students.

Overview of PISA 2000: Reading 2003: Mathematics 2006: Science 2009: Reading 2012: Mathematics 2015: Science Surveys of 15-year-olds in the industrialised countries. Assessing to what extent students near the end of compulsory education have acquired some of the knowledge and skills essential for full participation in society. Every 3 year cycle, each year has a focal subject area: Reading, Mathematics, or Science Every 9 years, repeat of the focal subject Each cycle, three domains are all measured.

Selection of the PISA variables OECD Development Team: Research Experts Education Psychology Sociology Economics Measurement Statistics National representatives Educational Psychology Cross-cultural Psychology Organizational Psychology Social Psychology

Lee & Stankov (2013) PISA 2003 Noncognitive Variables: 1. Mathematics self-efficacy 2. Mathematics anxiety 3. Mathematics self-concept 4. Interest in mathematics 5. Instrumental motivation in mathematics 6. Control strategies 7. Elaboration strategies 8. Memorization strategies 9. Competitive learning 10. Co-operative learning 11. Attitudes towards school 12. Student-teacher relations at school 13. Sense of belonging to school 14. Teacher support in math lessons 15. Disciplinary climate in math class Self-beliefs Motivation Learning Strategies Attitudes toward School Lee, J., & Stankov, L. (2013). A higher-order factorial structure among fifteen major educationalpsychological constructs. Learning and Individual Differences, 26, 119-130.

Lee & Stankov (2013)

Lee (2014) PISA 2009 Noncognitive Variables: 1. Enjoyment of Reading 2. Attitude towards school 3. Teacher student Relations 4. Diversity Reading 5. Online Reading 6. Metacognition: Summarising 7. Metacognition: Understanding 8. Control 9. Memorisation 10. Elaboration Attitudinal Variables Reading Habits Reading Strategies Learning Strategies Lee, J. (2014). Universal factors of student achievement in high-performing Eastern and Western countries. Journal of Educational Psychology, 106, 364-374.

Lee (2014): Predictability of Non-cognitive Skills/Dispositions based on PISA 2009 (r =.30 to.40) Enjoyment Summarizing Understanding Personal enjoyment Text-related cognitive, metacognitive strategies (r =.20) Control Strategies Online Reading Modes of reading General learning strategies Attitudes towards School Student-teacher relationship Diversity of Reading Memorization (r = nearly zero) Elaboration Strategies General attitudes towards school and teachers

Stankov & Lee (2014): Beyond Educational Psychology Constructs Educational Psychology: Motivation, Learning strategies, Self-concept, Self-efficacy, Anxiety Clinical Psychology: Depression, Wellbeing, Toughness, Modesty Social Psychology: Extraversion, Neuroticism, Agreeableness, Conscientiousness, Openness to experience Cognitive Psychology: Self-assessment of intelligence, Confidence

Stankov & Lee (2014) Educational Psychology Cognitive Psychology r = greater than.45 Clinical/ Educational Psychology r = nearly zero Social Psychology r =.20 r = up to.35 Openness to Experience Domain-specific selfconcept r = up to.45 Self-efficacy Anxiety Self-assessment Confidence Depression Wellbeing Agreeableness Conscientiousness Self-assessment of intelligence Toughness Modesty Motivation Learning strategies Domain Task Item Extraversion Neuroticism Well-being, Personality Self-beliefs

Lee (in preparation): Trends in International Mathematics and Science Study (TIMSS) TIMSS data: Pan-cultural Correlations TIMSS Math Test Scores TIMSS Science Test Scores 2011 2007 2003 2011 2007 2003 HOME POSSESSION 0.462 0.443 0.459 0.451 0.406 0.456 AMOUNT OF BOOKS IN YOUR HOME 0.383 0.346 0.372 0.388 0.343 0.379 SES-type PARENTS EDUCATION LEVEL 0.377 0.353 0.354 0.365 0.326 0.354 variables MOTHER S EDUCATION LEVEL 0.374 0.317 0.358 0.361 0.294 0.355 FATHER S EDUCATION LEVEL 0.346 0.283 0.332 0.335 0.267 0.333 STUDENT SELF-EXPECTATION OF EDUCATION 0.270 0.227 0.299 0.281 0.240 0.303 LEVEL Self-beliefs CONFIDENCE WITH MATH 0.253 0.217 0.208 0.191 0.175 0.138 School safety FEELING SAFE IN SCHOOL 0.207 0.177 0.260 0.213 0.181 0.258

Trends in International Mathematics and Science Study (TIMSS) TIMSS 2003 (N = 237,833) TIMSS 2007 (N = 246,112) TIMSS 2011 (N = 287,395) Conducted by the International Association for the Evaluation of Educational Achievement s (IEA)

Trends in International Mathematics and Science Study (TIMSS) The TIMSS variables were analyzed in this study are: home possession; amount of books at home; parents highest education level; mother s highest education level; father s highest education level; student s self-expectation of educational level; confidence in mathematics; positive affect to mathematics; valuing mathematics; student engagement in mathematics lessons; weekly time spent on mathematics homework; time spent on extra lessons on mathematics; confidence in science; positive affect to science; valuing science; student engagement in science lessons; weekly time spent on science homework; time spent on extra lessons on science; feeling safe in school; attitude towards school; parental involvement; self-activity and homework of mathematics; mathematics-specific learning strategies; traditional learning strategies of mathematics; self-activity and homework of science; traditional learning strategies of science; science-specific learning strategies.

Non-cognitive Variables PISA 2012 1. ANXMAT: Anxiety about Mathematics 2. ATSCHL: Attitude towards School 3. BELONG: Sense of Belonging at School 4. INSTMOT: Instrumental Motivation 5. INTMAT: Interest in Mathematics 6. SCMAT: Self-concept in Math 7. MATHEFF: Self-efficacy in Math 8. SUBNORM: Subjective Norm 9. MATINTFC: Mathematics Intentions 10. MATWKETH: Math Work Ethics 11. PERSEV: Perseverance 12. OPENPS: Openness for Problem Solving All together, 21 non-cognitive variables

Non-cognitive Variables PISA 2012 13. COGACT: Cognitive Activation in Mathematics 14. CLSMAN: Teacher's Mathematics Classroom Management 15. MTSUP: Mathematics Teacher's Support 16. DISCLIMA: Disciplinary Climate 17. STUDREL: Student-teacher relationship 18. TEACHSUP: Teacher support 19. TCHBEHFA: Formative Assessment 20. TCHBEHSO: Student Orientation 21. TCHBEHTD: Teacher-directed Instruction All together, 21 non-cognitive variables

Lee (in preparation): PISA 2012 Predictability of Math Achievement from PISA 2012 Mathematics Self-Efficacy (r =.46) Index of economic, social and cultural status (r =.40) Math Anxiety (r =.37)

Lee (in preparation): PISA 2012 Predictability of Math Achievement from PISA 2012 Mathematics Self-Concept (r =.26) Wealth (r =.26) Disciplinary Climate (r =.21)

Lee (in preparation): PISA 2012 OPENPS Openness for Problem Solving 0.17 PERSEV Perseverance 0.11 MATINTFC Mathematics Intentions 0.08 ATSCHL Attitude towards School: Learning Outcomes 0.05 CLSMAN Mathematics Teacher's Classroom Management 0.05 BELONG Sense of Belonging to School 0.04 INSTMOT Instrumental Motivation for Mathematics 0.03 MATWKETH Mathematics Work Ethic 0.02 INTMAT Mathematics Interest 0.01 COGACT Cognitive Activation in Mathematics Lessons -0.02 STUDREL Teacher Student Relations -0.05 MTSUP Mathematics Teacher's Support -0.06 TEACHSUP Teacher Support -0.07 SUBNORM Subjective Norms in Mathematics -0.12 TCHBEHTD Teacher Behaviour: Teacher-directed Instruction -0.12 TCHBEHFA Teacher Behaviour: Formative Assessment -0.17 TCHBEHSO Teacher Behaviour: Student Orientation -0.33

PISA 2012: Countries Rankings on Non-cognitive Outcomes Country ANXMAT _mean Country ATSCHL_ mean Country BELONG_ mean Country CLSMAN_ mean Country COGACT_ mean Country DISCLIMA _mean Country INSTMOT _mean Country INTMAT_ mean Country SCMAT_ mean Country MATHEFF _mean Country SUBNOR M_mean Country MATINTFC _mean MATWKE Country TH_mean Country PERSEV_ mean Country OPENPS_ mean Country MTSUP_ mean Country STUDREL _mean Country TEACHSUP_ mean Country TCHBEHFA _mean Country TCHBEHSO _mean Country TCHBEHTD _mean NETHERLAN -0.3984 ALBANIA 0.5274 LIECHTENS 0.5803 KAZAKHST 0.6537 JORDAN 0.6461 KAZAKHSTA 0.7137 PERU 0.5549 ALBANIA 0.9413 ARA BEMIR 0.4173 SHANGHA 0.9365 ALBANIA 1.0654 SWEDEN 0.3529 JORDAN 0.8114 KAZAKHST 0.7771 MONTENE 0.619 ALBANIA 0.8333 KAZAKHST 0.7419 ALBANIA 0.618 KAZAKHST 0.769 QATAR 1.0816 ALBANIA 1.0167 SWEDEN -0.3457 KAZAKHSTAN 0.471 AUSTRIA 0.5606 ALBANIA 0.5788 BULGARIA 0.5297 JAPAN 0.6775 MALAYSIA 0.5391 MALAYSIA 0.91 JORDAN 0.4006 LIECHTENS 0.507 MALAYSIA 1.0485 DENMARK 0.3436 ARA BEMIR 0.6747 ALBANIA 0.6403 JORDAN 0.6057 KAZAKHST 0.6896 ALBANIA 0.7211 THAILAND 0.6042 BULGARIA 0.737 JORDAN 0.9914 KAZAKHST 0.9196 ICELAND -0.3318 COSTA RICA 0.4273 ISRAEL 0.4182 COSTA RIC 0.4927 QATAR 0.4985 SHANGHAI 0.5779 ALBANIA 0.5363 KAZAKHST 0.8795 KAZAKHST 0.3918 SINGAPOR 0.4373 KAZAKHST 0.9701 VIETNAM 0.326 COSTA RIC 0.6576 BULGARIA 0.5849 ALBANIA 0.5065 JORDAN 0.6117 COSTA RIC 0.4639 MEXICO 0.5581 ALBANIA 0.6971 ALBANIA 0.9596 RUSSIA 0.7615 DENMARK -0.322 LITHUANIA 0.4189 COSTA RIC 0.3883 LITHUANIA 0.4705 ARA BEMIR 0.4573 ALBANIA 0.3867 MEXICO 0.4875 SINGAPOR 0.832 QATAR 0.3607 GERMANY 0.3397 JORDAN 0.9151 NORWAY 0.2698 QATAR 0.5826 RUSSIA 0.4979 KAZAKHST 0.4643 RUSSIA 0.5107 SHANGHA 0.4617 KAZAKHST 0.5272 QATAR 0.6721 KAZAKHST 0.9082 JORDAN 0.6641 LIECHTENSTE -0.2994 COLOMBIA 0.3954 KAZAKHST 0.3838 RUSSIA 0.4006 UNITED ST 0.4307 VIETNAM 0.3728 JORDAN 0.4377 INDONESIA 0.8186 ISRAEL 0.338 PORTUGAL 0.2487 INDONESIA 0.8517 PERU 0.25 INDONESIA 0.5681 COSTA RIC 0.4811 SERBIA 0.4603 SHANGHA 0.4993 MEXICO 0.4517 COSTA RIC 0.5129 JORDAN 0.6454 ARA BEMIR 0.893 THAILAND 0.5556 GERMANY -0.2822 MEXICO 0.3611 ALBANIA 0.373 JORDAN 0.3074 PERU 0.3908 RUSSIA 0.3167 KAZAKHST 0.4068 JORDAN 0.7924 UNITED ST 0.3213 SLOVENIA 0.2458 ARA BEMIR 0.8356 ISRAEL 0.2412 ALBANIA 0.5664 TURKEY 0.4527 QATAR 0.3785 SINGAPOR 0.4913 COLOMBIA 0.4411 COLOMBIA 0.4939 ARA BEMIR 0.5994 THAILAND 0.8798 ARA BEMIR 0.5514 FINLAND -0.2821 CHILE 0.3052 SWITZERLA 0.3649 LIECHTENS 0.2926 PORTUGAL 0.3739 HONGKONG 0.2833 BRAZIL 0.4038 THAILAND 0.7623 ICELAND 0.2418 HONGKON 0.2101 THAILAND 0.8317 LIECHTENS 0.2312 TUNISIA 0.4041 VIETNAM 0.4495 BULGARIA 0.3775 INDONESIA 0.488 LITHUANIA 0.4399 ICELAND 0.4925 THAILAND 0.5274 COLOMBIA 0.7285 SHANGHA 0.5457 AUSTRIA -0.249 INDONESIA 0.3033 ICELAND 0.352 MEXICO 0.2813 KAZAKHST 0.3521 LITHUANIA 0.2773 SINGAPOR 0.4021 PERU 0.7423 CANADA 0.2297 HUNGARY 0.1926 SINGAPOR 0.7978 JORDAN 0.2141 KAZAKHST 0.384 COLOMBIA 0.418 POLAND 0.3705 ARA BEMIR 0.4828 INDONESIA 0.4254 BRAZIL 0.4666 RUSSIA 0.5237 INDONESIA 0.7175 BULGARIA 0.5438 SWITZERLAN -0.2351 AUSTRIA 0.285 SPAIN 0.3417 COLOMBIA 0.2676 CANADA 0.3439 ISRAEL 0.2732 TUNISIA 0.4009 ARA BEMIR 0.7147 DENMARK 0.2295 UNITED ST 0.1919 PERU 0.7752 TURKEY 0.2113 SINGAPOR 0.3431 ARA BEMIR 0.3979 ARA BEMIR 0.364 TUNISIA 0.4746 PERU 0.3793 JORDAN 0.4585 MALAYSIA 0.4574 BULGARIA 0.671 QATAR 0.4832 ESTONIA -0.1655 SPAIN 0.2293 GERMANY 0.277 SINGAPOR 0.2356 ALBANIA 0.334 LIECHTENST 0.2558 THAILAND 0.389 VIETNAM 0.692 INDONESIA 0.2158 SWITZERLA 0.1883 QATAR 0.7076 ICELAND 0.1846 MALAYSIA 0.3424 UNITED ST 0.3772 ISRAEL 0.3416 MALAYSIA 0.4478 ROMANIA 0.3672 PORTUGAL 0.437 COLOMBIA 0.4416 MALAYSIA 0.6203 COLOMBIA 0.3937 UNITED STAT -0.1336 BRAZIL 0.229 COLOMBIA 0.223 ROMANIA 0.2334 ARGENTIN 0.3062 AUSTRIA 0.2252 INDONESIA 0.3709 MEXICO 0.6503 ALBANIA 0.2086 TAIPEI 0.1751 TUNISIA 0.5554 MALAYSIA 0.1768 ISRAEL 0.3295 ISRAEL 0.3658 TUNISIA 0.2522 LITHUANIA 0.4299 SINGAPOR 0.3566 PERU 0.3936 PERU 0.3554 PERU 0.6003 INDONESIA 0.3926 LUXEMBOUR -0.1059 PERU 0.2192 LUXEMBOU 0.2076 SHANGHA 0.2255 UNITED KIN 0.2969 SINGAPORE 0.2028 VIETNAM 0.3689 QATAR 0.6053 SINGAPOR 0.2077 CZECH 0.1461 ISRAEL 0.4669 KAZAKHST 0.168 SHANGHA 0.3248 PERU 0.355 GREECE 0.2411 THAILAND 0.4028 ARA BEMIR 0.3555 CHILE 0.3925 ROMANIA 0.3499 TUNISIA 0.5938 TURKEY 0.3828 HUNGARY -0.0799 PORTUGAL 0.2178 CHILE 0.2011 ARA BEMIR 0.22 SINGAPOR 0.2859 KOREA 0.1889 COLOMBIA 0.3551 TUNISIA 0.5961 PERU 0.1594 LUXEMBOU 0.1396 MEXICO 0.4238 QATAR 0.1442 MEXICO 0.3234 MONTENE 0.3462 CHILE 0.2244 TURKEY 0.3734 JORDAN 0.3549 ARA BEMIR 0.3843 INDONESIA 0.3232 RUSSIA 0.5446 UNITED ST 0.3353 UNITED KING -0.0781 SCOTLAND 0.2034 URUGUAY 0.162 VIETNAM 0.217 COLOMBIA 0.2797 ESTONIA 0.1860 ARA BEMIR 0.3529 COLOMBIA 0.5096 COLOMBIA 0.158 MACAO-CH 0.1374 ICELAND 0.4038 COSTA RIC 0.1407 COLOMBIA 0.322 PORTUGAL 0.3416 ROMANIA 0.2213 COSTA RIC 0.3721 PORTUGAL 0.3235 INDONESIA 0.3761 PORTUGAL 0.3094 MEXICO 0.5168 BRAZIL 0.329 LITHUANIA -0.0753 RUSSIA 0.1841 LITHUANIA 0.1501 MONTENE 0.2113 ISRAEL 0.2695 UNITED KIN 0.1449 UNITED KIN 0.3492 ROMANIA 0.4972 SWEDEN 0.1343 POLAND 0.1287 NEW ZEAL 0.3805 SWITZERLA 0.1393 UNITED ST 0.3149 MEXICO 0.3318 UNITED ST 0.2192 VIETNAM 0.3654 CANADA 0.2969 SINGAPOR 0.3593 UNITED ST 0.3079 BRAZIL 0.4496 CHILE 0.3242 CZECH -0.0725 TUNISIA 0.1715 TURKEY 0.1275 UNITED ST 0.2112 SCOTLAND 0.2681 INDONESIA 0.1282 ICELAND 0.3294 BRAZIL 0.4878 UNITED KIN 0.1288 ISRAEL 0.1284 CANADA 0.371 THAILAND 0.1221 BRAZIL 0.3073 JORDAN 0.3287 COSTA RIC 0.2107 BRAZIL 0.3648 THAILAND 0.2913 SHANGHA 0.3559 BRAZIL 0.2956 SWEDEN 0.4433 MEXICO 0.314 ISRAEL -0.0597 VIETNAM 0.1434 CROATIA 0.1255 ISRAEL 0.2049 CHILE 0.2403 IRELAND 0.1205 CHILE 0.3206 TURKEY 0.4368 SWITZERLA 0.115 SPAIN 0.1277 DENMARK 0.3482 COLOMBIA 0.0984 ICELAND 0.2952 ESTONIA 0.3091 TURKEY 0.2092 COLOMBIA 0.3567 BRAZIL 0.2778 QATAR 0.3449 SINGAPOR 0.2937 ROMANIA 0.4081 PERU 0.304 POLAND -0.037 LIECHTENSTE 0.1302 HUNGARY 0.1216 BULGARIA 0.2005 ROMANIA 0.2304 UNITED STA 0.1168 QATAR 0.2898 SHANGHA 0.4367 COSTA RIC 0.1141 KAZAKHST 0.1268 UNITED KIN 0.3197 NETHERLA 0.0947 PERU 0.2691 SINGAPOR 0.291 BRAZIL 0.2052 BULGARIA 0.3534 MALAYSIA 0.2297 ARGENTIN 0.3289 UNITED KIN 0.2778 COSTA RIC 0.3733 VIETNAM 0.297 CANADA -0.0173 UNITED KING 0.1282 MEXICO 0.1084 MALAYSIA 0.1897 URUGUAY 0.2257 CZECH 0.1111 COSTA RIC 0.2895 DENMARK 0.386 GERMANY 0.1094 SCOTLAND 0.1157 CHILE 0.3094 ESTONIA 0.0901 UNITED KIN 0.2676 CHILE 0.2874 HUNGARY 0.1948 MEXICO 0.339 BULGARIA 0.2255 URUGUAY 0.3247 CANADA 0.2735 ARGENTIN 0.3731 MONTENE 0.2917 SCOTLAND -0.0151 HUNGARY 0.1215 NORWAY 0.0825 UNITED KIN 0.1843 RUSSIA 0.2039 MACAO-CH 0.1009 ISRAEL 0.2877 COSTA RIC 0.3143 MALAYSIA 0.1064 SLOVAKIA 0.1002 AUSTRALIA 0.2865 MEXICO 0.0822 THAILAND 0.2605 QATAR 0.2664 NORWAY 0.1831 QATAR 0.325 ICELAND 0.2089 UNITED KIN 0.3246 SCOTLAND 0.269 TURKEY 0.3179 PORTUGAL 0.2702 BELGIUM 0.0061 IRELAND 0.1139 SERBIA 0.0666 TURKEY 0.18 NEW ZEAL 0.2033 SCOTLAND 0.0952 NEW ZEAL 0.2746 HONGKON 0.2925 RUSSIA 0.1029 AUSTRIA 0.0835 TURKEY 0.2699 ARA BEMIR 0.0735 TURKEY 0.2572 INDONESIA 0.2654 PERU 0.171 PERU 0.302 UNITED ST 0.1977 MALAYSIA 0.3155 MONTENE 0.2194 CHILE 0.3106 SERBIA 0.2676 SLOVAKIA 0.0105 CROATIA 0.0902 PORTUGAL 0.0105 INDONESIA 0.1787 MEXICO 0.203 HUNGARY 0.0772 CANADA 0.2739 CHILE 0.2871 GREECE 0.098 CANADA 0.0812 COSTA RIC 0.2545 LATVIA 0.0655 SCOTLAND 0.1803 URUGUAY 0.2593 GERMANY 0.1686 TAIPEI 0.3011 CHILE 0.1961 SCOTLAND 0.3007 SERBIA 0.2172 UNITED ST 0.3096 ROMANIA 0.2456 PORTUGAL 0.0171 FRANCE 0.0855 INDONESIA 0.0104 PERU 0.1783 SHANGHA 0.1675 LATVIA 0.071 LITHUANIA 0.2675 RUSSIA 0.2836 LIECHTENS 0.0881 CROATIA 0.072 COLOMBIA 0.2399 SINGAPOR 0.0599 CHILE 0.1793 SHANGHA 0.2529 COLOMBIA 0.1632 CANADA 0.2829 TURKEY 0.1941 AUSTRALIA 0.2932 SHANGHA 0.2093 ICELAND 0.3017 CANADA 0.2402 NORWAY 0.0209 TURKEY 0.0853 ARA BEMIR -0.0029 CANADA 0.1707 LIECHTENS 0.1637 BELGIUM 0.0695 DENMARK 0.2638 URUGUAY 0.2669 SCOTLAND 0.0712 ICELAND 0.0542 UNITED ST 0.2399 URUGUAY 0.0474 CANADA 0.1698 THAILAND 0.2461 PORTUGAL 0.151 UNITED ST 0.28 URUGUAY 0.1824 CANADA 0.2708 CHILE 0.2072 VIETNAM 0.2981 LATVIA 0.2393 LATVIA 0.0278 UNITED STAT 0.083 JORDAN -0.0074 ICELAND 0.1678 CZECH 0.1334 THAILAND 0.0693 SCOTLAND 0.261 BULGARIA 0.2062 NETHERLA 0.0655 LITHUANIA 0.0462 BRAZIL 0.2299 CHILE 0.0463 PORTUGAL 0.1437 MALAYSIA 0.2208 CANADA 0.1326 URUGUAY 0.2683 MONTENE 0.1705 RUSSIA 0.2557 NEW ZEAL 0.1929 SERBIA 0.288 ISRAEL 0.2371 SHANGHAI 0.0339 NEW ZEALAN 0.0744 IRELAND -0.0231 LATVIA 0.166 IRELAND 0.1316 POLAND 0.0659 PORTUGAL 0.2475 GREECE 0.206 TUNISIA 0.0628 SWEDEN 0.0336 SCOTLAND 0.2151 SHANGHA 0.0321 DENMARK 0.1415 SERBIA 0.1974 SWEDEN 0.1203 PORTUGAL 0.2665 UNITED KIN 0.1563 SWEDEN 0.2287 TURKEY 0.1728 PORTUGAL 0.2731 ARGENTIN 0.2326 KAZAKHSTA 0.0412 AUSTRALIA 0.059 MONTENE -0.0256 TUNISIA 0.1641 AUSTRALIA 0.1299 MEXICO 0.0639 AUSTRALIA 0.2312 ISRAEL 0.1686 LUXEMBOU 0.055 ALBANIA 0.0162 LITHUANIA 0.1708 LUXEMBOU 0.0199 MONTENE 0.1327 CANADA 0.1956 INDONESIA 0.0814 SCOTLAND 0.2641 LATVIA 0.1403 VIETNAM 0.2268 AUSTRALIA 0.1726 DENMARK 0.2391 GREECE 0.2086 AUSTRALIA 0.0432 ARGENTINA 0.0568 DENMARK -0.0266 ESTONIA 0.152 INDONESIA 0.1266 SWITZERLA 0.0501 URUGUAY 0.1968 ARGENTIN 0.1613 AUSTRALIA 0.0504 FRANCE 0.004 SWEDEN 0.1681 AUSTRALIA 0.0135 AUSTRIA 0.1167 LATVIA 0.194 ESTONIA 0.0704 UNITED KIN 0.2622 SCOTLAND 0.1345 BULGARIA 0.2147 TUNISIA 0.161 NORWAY 0.2371 SINGAPOR 0.2026 NEW ZEALAN 0.0947 SWITZERLAN 0.0568 UNITED KIN -0.0286 SCOTLAND 0.1392 THAILAND 0.1245 COSTA RICA 0.0356 NORWAY 0.1924 ICELAND 0.153 MEXICO 0.0438 IRELAND 0.001 LIECHTENS 0.1458 ARGENTIN 0.0039 LIECHTENS 0.1036 MACAO-CH 0.1521 RUSSIA 0.0691 LATVIA 0.2605 AUSTRALIA 0.1329 UNITED ST 0.2056 ISRAEL 0.1566 URUGUAY 0.2316 TUNISIA 0.1985 SLOVENIA 0.1005 FINLAND 0.0548 PERU -0.0365 IRELAND 0.1376 LATVIA 0.0936 PORTUGAL 0.0264 SWEDEN 0.1827 MACAO-CH 0.1472 AUSTRIA 0.0416 AUSTRALIA -0.0103 NORWAY 0.1439 AUSTRIA 0.0024 SPAIN 0.1027 LITHUANIA 0.15 SLOVENIA 0.0686 ICELAND 0.244 RUSSIA 0.1228 FINLAND 0.185 SLOVAKIA 0.1341 MONTENE 0.2165 MALAYSIA 0.1937 RUSSIA 0.1051 MONTENEGR 0.05 SWEDEN -0.0384 JAPAN 0.1306 TUNISIA 0.0874 ROMANIA 0.0083 ARGENTIN 0.1356 UNITED KIN 0.1437 ROMANIA 0.0284 ESTONIA -0.0129 ROMANIA 0.1401 PORTUGAL -0.0019 BULGARIA 0.1026 BRAZIL 0.1477 SHANGHA 0.0675 ROMANIA 0.2438 ARGENTIN 0.118 TURKEY 0.1812 ITALY 0.1261 LATVIA 0.2067 LIECHTENS 0.1715 HONGKONG 0.1162 ICELAND 0.0442 BELGIUM -0.0393 SWITZERLA 0.1223 LITHUANIA 0.0865 MONTENEG 0.0011 IRELAND 0.1217 SWEDEN 0.1283 FINLAND 0.0278 NORWAY -0.0139 BULGARIA 0.1344 UNITED ST -0.0056 BELGIUM 0.0877 IRELAND 0.1323 LUXEMBOU 0.062 CHILE 0.2312 DENMARK 0.1172 NEW ZEAL 0.1619 LATVIA 0.0949 ISRAEL 0.2038 LITHUANIA 0.1544 GREECE 0.1243 CANADA 0.0292 NETHERLA -0.0423 THAILAND 0.0721 SWITZERLA 0.0822 TAIPEI -0.0018 UNITED ST 0.12 PORTUGAL 0.1054 ESTONIA 0.0219 TURKEY -0.0144 IRELAND 0.123 UNITED KIN -0.0078 IRELAND 0.0708 TUNISIA 0.1296 ICELAND 0.0595 ISRAEL 0.2144 NEW ZEAL 0.1059 DENMARK 0.1404 MEXICO 0.0785 LITHUANIA 0.187 COSTA RIC 0.1399 IRELAND 0.1266 LATVIA 0.023 SLOVENIA -0.0473 BELGIUM 0.0611 GREECE 0.0809 ARA BEMIR -0.005 LATVIA 0.1161 NEW ZEAL 0.1018 NEW ZEAL 0.0104 ARA BEMIR -0.0171 SHANGHA 0.1164 FRANCE -0.0114 ROMANIA 0.0665 HONGKON 0.119 AUSTRIA 0.0555 ARGENTIN 0.2028 SWEDEN 0.0841 TAIPEI 0.122 SWEDEN 0.0694 LIECHTENS 0.128 UNITED KIN 0.1181 CROATIA 0.1376 URUGUAY 0.0211 UNITED ST -0.0682 BRAZIL 0.0461 BRAZIL 0.0665 GERMANY -0.0223 LIECHTENS 0.0748 LITHUANIA 0.0958 ITALY 0.0078 JORDAN -0.0449 URUGUAY 0.1145 JAPAN -0.0139 URUGUAY 0.065 SPAIN 0.1055 DENMARK 0.054 AUSTRALIA 0.155 SERBIA 0.0835 ROMANIA 0.1163 NORWAY 0.0633 MACAO-CH 0.1204 URUGUAY 0.0989 ALBANIA 0.1418 ESTONIA 0.0181 THAILAND -0.0797 PORTUGAL 0.0291 POLAND 0.0636 CANADA -0.0229 TURKEY 0.0469 LIECHTENS 0.0957 URUGUAY -0.0151 UNITED KIN -0.0536 PORTUGAL 0.1135 INDONESIA -0.0207 MACAO-CH 0.0602 SCOTLAND 0.1051 LIECHTENS 0.0504 NEW ZEAL 0.1261 QATAR 0.0825 IRELAND 0.0727 CROATIA 0.0596 CANADA 0.0976 SLOVENIA 0.0982 SINGAPORE 0.1781 THAILAND 0.0093 CANADA -0.0832 AUSTRIA 0.0207 MONTENE 0.0472 LUXEMBOU -0.0232 FINLAND 0.0407 AUSTRALIA 0.0955 LITHUANIA -0.0218 ITALY -0.0731 RUSSIA 0.0378 CROATIA -0.0223 LUXEMBOU 0.059 CROATIA 0.0902 SPAIN 0.0444 HONGKON 0.1214 ISRAEL 0.0676 MONTENE 0.0626 LIECHTENS 0.059 SWITZERLA 0.0955 CZECH 0.097 SPAIN 0.179 ISRAEL 0.0066 SCOTLAND -0.0938 MACAO-CH 0.0013 SPAIN 0.0324 ICELAND -0.0255 GREECE 0.0222 UNITED ST 0.0795 BELGIUM -0.0344 BULGARIA -0.0898 MONTENE 0.0289 FINLAND -0.025 NORWAY 0.0556 UNITED KIN 0.0874 URUGUAY 0.0327 ITALY 0.1051 SWITZERLA 0.0532 LITHUANIA 0.0591 AUSTRIA 0.0583 SINGAPOR 0.0912 CROATIA 0.0534 MACAO-CHI 0.1884 MALAYSIA -0.0044 BULGARIA -0.094 URUGUAY -0.0027 GERMANY 0.0144 SPAIN -0.0267 SHANGHA 0.0218 SCOTLAND 0.0722 TURKEY -0.0458 LATVIA -0.0998 SWITZERLA -0.0023 ALBANIA -0.0416 ITALY 0.0413 SLOVENIA 0.0683 SINGAPOR -0.0011 GREECE 0.1022 LIECHTENS 0.0485 LATVIA 0.0355 ARGENTIN 0.0458 NEW ZEAL 0.0565 AUSTRALIA 0.0241 MONTENEGR 0.1921 ITALY -0.0071 FRANCE -0.1027 CZECH -0.0097 MALAYSIA -0.0027 COLOMBIA -0.0286 ESTONIA 0.0151 TAIPEI 0.0636 SHANGHA -0.0497 CHILE -0.0999 ARGENTIN -0.0104 BELGIUM -0.0483 ARGENTIN 0.0223 AUSTRALIA 0.0659 SWITZERLA -0.0167 MONTENE 0.0975 TAIPEI 0.0382 HONGKON 0.0346 SLOVENIA 0.0378 SLOVAKIA 0.0331 SCOTLAND 0.0208 SERBIA 0.1924 ROMANIA -0.0119 NEW ZEAL -0.143 SERBIA -0.0106 DENMARK -0.0146 PERU -0.0293 BULGARIA -0.0353 CANADA 0.059 IRELAND -0.0542 RUSSIA -0.1015 HONGKON -0.0203 BRAZIL -0.0706 LITHUANIA 0.0157 ARGENTIN 0.0573 IRELAND -0.0232 MACAO-CH 0.0888 IRELAND 0.0336 ISRAEL 0.0279 VIETNAM 0.007 AUSTRALIA 0.029 HUNGARY -0.0013 ARA BEMIRA 0.1943 ARA BEMIRA -0.016 GREECE -0.1502 CHILE -0.012 SERBIA -0.0166 SLOVENIA -0.0321 HUNGARY -0.044 IRELAND 0.0405 SLOVENIA -0.06 DENMARK -0.1075 FINLAND -0.0225 GREECE -0.0806 GERMANY -0.0006 ROMANIA 0.0391 SCOTLAND -0.0242 SERBIA 0.079 HONGKON 0.0269 SPAIN 0.0029 LITHUANIA 0.0066 FINLAND 0.0195 SWITZERLA -0.0066 VIETNAM 0.2146 SLOVENIA -0.0341 TUNISIA -0.1532 SLOVENIA -0.0123 FINLAND -0.0289 ITALY -0.0356 SPAIN -0.0444 MONTENE 0.0193 BRAZIL -0.0612 BELGIUM -0.1186 SPAIN -0.0586 LITHUANIA -0.0838 VIETNAM -0.002 ITALY 0.0386 CROATIA -0.0371 IRELAND 0.066 VIETNAM 0.0181 MACAO-CH -0.0268 HUNGARY -0.0008 TAIPEI 0.0054 SWEDEN -0.0394 BULGARIA 0.2381 SINGAPORE -0.0529 MALAYSIA -0.1559 AUSTRALIA -0.0158 TURKEY -0.0462 DENMARK -0.0455 SWITZERLA -0.0531 SWITZERLA 0.0106 THAILAND -0.0657 ROMANIA -0.1315 LATVIA -0.0913 CZECH -0.0862 HONGKON -0.0119 POLAND 0.0322 UNITED KIN -0.0451 ESTONIA 0.0514 TUNISIA -0.0187 NORWAY -0.047 COSTA RIC -0.0393 CZECH -0.024 MACAO-CH -0.047 TURKEY 0.2656 GERMANY -0.0587 SINGAPOR -0.1591 LUXEMBOU -0.027 SLOVENIA -0.0567 TURKEY -0.0776 SERBIA -0.0867 ESTONIA -0.0026 BULGARIA -0.0695 QATAR -0.1463 VIETNAM -0.0913 SCOTLAND -0.0918 SWEDEN -0.0122 HUNGARY -0.0019 FINLAND -0.0653 CROATIA -0.0186 HUNGARY -0.0188 SWITZERLA -0.0518 GREECE -0.0468 NETHERLA -0.0545 GERMANY -0.049 QATAR 0.2667 LUXEMBOUR -0.0712 JAPAN -0.1622 HONGKON -0.0331 ESTONIA -0.0605 NORWAY -0.0816 RUSSIA -0.0917 FRANCE -0.0137 LATVIA -0.072 NEW ZEAL -0.1574 GREECE -0.0934 GERMANY -0.0931 SWITZERLA -0.0235 NEW ZEAL -0.0077 LATVIA -0.0772 SPAIN -0.0231 MACAO-CH -0.0454 LIECHTENS -0.0645 POLAND -0.0521 GERMANY -0.0582 FRANCE -0.0531 ITALY 0.2729 BULGARIA -0.0953 BRAZIL -0.166 QATAR -0.0357 FRANCE -0.07 CROATIA -0.1264 GERMANY -0.1339 ITALY -0.0273 SPAIN -0.0734 GREECE -0.1581 GERMANY -0.1049 POLAND -0.0931 SERBIA -0.0498 GERMANY -0.0088 ITALY -0.0913 FINLAND -0.0235 FINLAND -0.0474 SERBIA -0.0665 SWITZERLA -0.0542 ITALY -0.064 SLOVAKIA -0.0587 FRANCE 0.2802 DENMARK -0.1001 FINLAND -0.1708 GERMANY -0.0396 HUNGARY -0.0759 SLOVAKIA -0.1276 POLAND -0.144 LATVIA -0.0492 POLAND -0.0748 MEXICO -0.1789 FRANCE -0.1141 CANADA -0.1135 AUSTRALIA -0.055 FINLAND -0.0126 AUSTRALIA -0.0933 DENMARK -0.0399 LUXEMBOU -0.0533 JAPAN -0.0829 DENMARK -0.0642 POLAND -0.0966 FINLAND -0.0608 INDONESIA 0.2807 BELGIUM -0.1106 AUSTRALIA -0.1817 CROATIA -0.0527 AUSTRIA -0.0852 SERBIA -0.1589 FRANCE -0.1588 GERMANY -0.113 PORTUGAL -0.0845 NETHERLANDS -0.1883 LUXEMBOURG-0.1547 NEW ZEALAND-0.1172 SLOVENIA -0.0568 AUSTRIA -0.0259 MEXICO -0.096 SWEDEN -0.0408 ESTONIA -0.0919 TUNISIA -0.0946 NETHERLANDS-0.0694 UNITED KINGDOM -0.0986 ICELAND -0.0658 KOREA 0.3065 JAPAN -0.1334 RUSSIA -0.1856 HUNGARY -0.0533 LUXEMBOURG-0.0961 URUGUAY -0.1605 CZECH -0.191 FINLAND -0.1291 ARGENTINA -0.0924 SERBIA -0.2152 ESTONIA -0.1842 TUNISIA -0.129 GREECE -0.061 DENMARK -0.0345 NETHERLANDS -0.0984 NORWAY -0.0996 SPAIN -0.098 ESTONIA -0.1254 ESTONIA -0.0825 JAPAN -0.1288 TAIPEI -0.068 TAIPEI 0.3185 SWEDEN -0.1355 TAIPEI -0.1966 ITALY -0.0593 ITALY -0.1105 AUSTRALIA -0.1724 ITALY -0.2002 NORWAY -0.1482 MONTENEGRO-0.0997 PERU -0.2158 KOREA -0.2058 ITALY -0.1321 FRANCE -0.0840 LUXEMBOURG-0.0621 ARGENTINA -0.1233 SWITZERLAND-0.1019 AUSTRIA -0.1151 GREECE -0.1689 IRELAND -0.0831 SCOTLAND -0.1367 NEW ZEALAND-0.0737 PERU 0.3348 GREECE -0.1734 ARGENTINA -0.1988 SLOVAKIA -0.0867 CROATIA -0.1491 NETHERLANDS-0.1782 SLOVENIA -0.2092 POLAND -0.1609 NORWAY -0.1009 THAILAND -0.2359 SLOVENIA -0.2110 IRELAND -0.1357 HUNGARY -0.0951 TAIPEI -0.0783 CZECH -0.1318 HUNGARY -0.1106 KOREA -0.1158 BELGIUM -0.1784 GERMANY -0.0922 SPAIN -0.1473 IRELAND -0.0907 COLOMBIA 0.3500 SERBIA -0.2197 ITALY -0.2015 SPAIN -0.0911 COSTA RICA -0.1662 BULGARIA -0.19 HONGKONG -0.2296 CZECH -0.1616 HUNGARY -0.1053 INDONESIA -0.2439 ITALY -0.2307 SPAIN -0.1393 RUSSIA -0.1105 CZECH -0.0846 LITHUANIA -0.1497 SLOVENIA -0.1142 BELGIUM -0.1278 SLOVAKIA -0.1828 TAIPEI -0.0985 ESTONIA -0.1505 AUSTRIA -0.0962 CHILE 0.3569 SHANGHAI -0.2267 LATVIA -0.2138 TAIPEI -0.104 TAIPEI -0.1712 MALAYSIA -0.1928 CROATIA -0.2534 LUXEMBOURG-0.162 CHILE -0.1101 MALAYSIA -0.2446 AUSTRIA -0.2428 SLOVENIA -0.1433 NEW ZEALAND-0.1162 KOREA -0.0895 FRANCE -0.1862 JAPAN -0.1224 NORWAY -0.1393 HUNGARY -0.1917 URUGUAY -0.0989 GREECE -0.1682 NETHERLANDS-0.1153 JAPAN 0.3612 CZECH -0.2319 QATAR -0.2184 NEW ZEALAND-0.1059 ICELAND -0.1719 CHILE -0.1937 MONTENEGRO-0.2623 SPAIN -0.1635 CZECH -0.1180 FINLAND -0.2489 TAIPEI -0.2513 RUSSIA -0.1434 NETHERLANDS -0.1279 GREECE -0.0952 NEW ZEALAND-0.194 SLOVAKIA -0.1238 GREECE -0.1508 ITALY -0.2236 SPAIN -0.1043 KOREA -0.1697 BELGIUM -0.1154 URUGUAY 0.3626 TAIPEI -0.2499 VIETNAM -0.2558 NORWAY -0.1079 BELGIUM -0.1951 SWEDEN -0.1951 MACAO-CHINA -0.2633 SERBIA -0.1684 SERBIA -0.1553 URUGUAY -0.2676 BELGIUM -0.2742 MACAO-CHINA-0.1733 ESTONIA -0.1486 LIECHTENSTEIN -0.0963 MALAYSIA -0.2014 POLAND -0.1545 CROATIA -0.1526 FRANCE -0.2402 ICELAND -0.1075 SHANGHAI -0.1961 LUXEMBOURG-0.1238 ROMANIA 0.3928 SLOVAKIA -0.2621 ROMANIA -0.2827 DENMARK -0.1189 SLOVAKIA -0.1954 NEW ZEALAND-0.2423 LUXEMBOURG-0.2753 HUNGARY -0.1817 HONGKONG -0.1620 VIETNAM -0.268 POLAND -0.2948 SLOVAKIA -0.1834 POLAND -0.1597 ICELAND -0.1054 HONGKONG -0.2485 FRANCE -0.1605 NETHERLANDS -0.1551 POLAND -0.2489 FRANCE -0.1115 SLOVENIA -0.2208 ESTONIA -0.1686 MALAYSIA 0.4263 NORWAY -0.2689 SLOVAKIA -0.3099 FRANCE -0.1255 NORWAY -0.212 GREECE -0.2564 TAIPEI -0.3393 SLOVAKIA -0.1951 SLOVAKIA -0.1681 MONTENEGRO-0.2725 SERBIA -0.3187 TAIPEI -0.1855 JAPAN -0.1773 NETHERLANDS -0.1374 BELGIUM -0.2736 CZECH -0.1900 JAPAN -0.1723 KOREA -0.2490 BELGIUM -0.1190 LUXEMBOURG-0.2450 ITALY -0.1777 COSTA RICA 0.4292 KOREA -0.2879 SHANGHAI -0.3132 SWEDEN -0.1404 HONGKONG -0.2135 JORDAN -0.2642 SLOVAKIA -0.3501 KOREA -0.2013 FRANCE -0.1699 TUNISIA -0.3138 HUNGARY -0.3202 KOREA -0.2080 CROATIA -0.1793 SWITZERLAND-0.1769 THAILAND -0.2784 BELGIUM -0.2145 FRANCE -0.1802 CZECH -0.2693 FINLAND -0.1488 AUSTRIA -0.2571 SPAIN -0.1882 MEXICO 0.4386 JORDAN -0.3242 KOREA -0.3209 POLAND -0.1596 SWEDEN -0.2143 FRANCE -0.2841 BELGIUM -0.3683 JAPAN -0.2367 MACAO-CHINA -0.1878 COSTA RICA -0.3315 MACAO-CHINA -0.3358 BULGARIA -0.2297 TAIPEI -0.2039 SWEDEN -0.2501 SLOVAKIA -0.3162 LIECHTENSTEIN -0.2448 ITALY -0.1930 GERMANY -0.3216 LUXEMBOURG-0.1537 BELGIUM -0.2718 NORWAY -0.2118 THAILAND 0.5019 NETHERLANDS -0.3705 POLAND -0.3232 FINLAND -0.1911 NETHERLANDS -0.2209 FINLAND -0.3044 NETHERLANDS -0.3712 SLOVENIA -0.238 VIETNAM -0.1919 ARGENTINA -0.3352 NETHERLANDS -0.3538 HUNGARY -0.3098 LATVIA -0.2188 BELGIUM -0.3143 TAIPEI -0.3334 KOREA -0.2525 SLOVAKIA -0.1939 CROATIA -0.3618 HONGKONG -0.1755 HONGKONG -0.3549 POLAND -0.2133 ARGENTINA 0.5168 QATAR -0.3918 ESTONIA -0.3267 ARGENTINA -0.1912 MACAO-CHINA-0.228 QATAR -0.3222 KOREA -0.3894 BELGIUM -0.2564 CROATIA -0.2570 KOREA -0.3571 SLOVAKIA -0.3798 HONGKONG -0.3161 FINLAND -0.2890 NORWAY -0.3454 MACAO-CHINA -0.3377 LUXEMBOURG-0.3645 CZECH -0.2071 SLOVENIA -0.3676 CZECH -0.1917 CROATIA -0.3775 DENMARK -0.2319 JORDAN 0.5206 HONGKONG -0.4169 CZECH -0.3597 GREECE -0.1927 VIETNAM -0.3261 BRAZIL -0.3322 AUSTRIA -0.3988 CROATIA -0.2635 KOREA -0.3835 JAPAN -0.4156 CROATIA -0.4541 ROMANIA -0.3371 CZECH -0.3841 FRANCE -0.4506 KOREA -0.3708 GERMANY -0.4057 SLOVENIA -0.2171 LUXEMBOURG -0.3889 MACAO-CHINA-0.3782 FRANCE -0.4167 JAPAN -0.2590 BRAZIL 0.5282 POLAND -0.4189 HONGKONG -0.3963 KOREA -0.2600 JAPAN -0.5042 TUNISIA -0.4225 JAPAN -0.4972 AUSTRIA -0.3243 TAIPEI -0.4571 COLOMBIA -0.4460 CZECH -0.5253 SERBIA -0.4239 SLOVAKIA -0.4276 SLOVAKIA -0.4891 VIETNAM -0.6131 NETHERLANDS -0.4181 GERMANY -0.2196 AUSTRIA -0.4349 JAPAN -0.6310 HUNGARY -0.4565 KOREA -0.3141 TUNISIA 0.6452 MACAO-CHINA -0.4339 MACAO-CHINA -0.4958 NETHERLANDS -0.3006 KOREA -0.7320 ARGENTINA -0.4921 ROMANIA -0.5637 NETHERLANDS -0.3402 JAPAN -0.5251 BRAZIL -0.4499 JAPAN -0.5848 MONTENEGRO-0.4495 KOREA -0.5443 JAPAN -0.5961 JAPAN -0.7306 AUSTRIA -0.482 POLAND -0.4152 NETHERLANDS -0.4456 KOREA -0.7692 IRELAND -0.5682 HONGKONG -0.3228

PISA 2012: Countries Rankings on Non-cognitive Outcomes Country ANXMAT _mean Country ATSCHL_ mean Country BELONG_ mean Country CLSMAN_ mean Country COGACT_ mean Country DISCLIMA _mean Country INSTMOT _mean Country INTMAT_ mean Country SCMAT_ mean Country MATHEFF _mean Country SUBNOR M_mean Country MATINTFC _mean MATWKE Country TH_mean Country PERSEV_ mean Country OPENPS_ mean Country MTSUP_ mean Country STUDREL _mean Country TEACHSUP_ mean Country TCHBEHFA _mean Country TCHBEHSO _mean Country TCHBEHTD _mean NETHERLAN -0.3984 ALBANIA 0.5274 LIECHTENS 0.5803 KAZAKHST 0.6537 JORDAN 0.6461 KAZAKHSTA 0.7137 PERU 0.5549 ALBANIA 0.9413 ARA BEMIR 0.4173 SHANGHA 0.9365 ALBANIA 1.0654 SWEDEN 0.3529 JORDAN 0.8114 KAZAKHST 0.7771 MONTENE 0.619 ALBANIA 0.8333 KAZAKHST 0.7419 ALBANIA 0.618 KAZAKHST 0.769 QATAR 1.0816 ALBANIA 1.0167 SWEDEN -0.3457 KAZAKHSTAN 0.471 AUSTRIA 0.5606 ALBANIA 0.5788 BULGARIA 0.5297 JAPAN 0.6775 MALAYSIA 0.5391 MALAYSIA 0.91 JORDAN 0.4006 LIECHTENS 0.507 MALAYSIA 1.0485 DENMARK 0.3436 ARA BEMIR 0.6747 ALBANIA 0.6403 JORDAN 0.6057 KAZAKHST 0.6896 ALBANIA 0.7211 THAILAND 0.6042 BULGARIA 0.737 JORDAN 0.9914 KAZAKHST 0.9196 ICELAND -0.3318 COSTA RICA 0.4273 ISRAEL 0.4182 COSTA RIC 0.4927 QATAR 0.4985 SHANGHAI 0.5779 ALBANIA 0.5363 KAZAKHST 0.8795 KAZAKHST 0.3918 SINGAPOR 0.4373 KAZAKHST 0.9701 VIETNAM 0.326 COSTA RIC 0.6576 BULGARIA 0.5849 ALBANIA 0.5065 JORDAN 0.6117 COSTA RIC 0.4639 MEXICO 0.5581 ALBANIA 0.6971 ALBANIA 0.9596 RUSSIA 0.7615 DENMARK -0.322 LITHUANIA 0.4189 COSTA RIC 0.3883 LITHUANIA 0.4705 ARA BEMIR 0.4573 ALBANIA 0.3867 MEXICO 0.4875 SINGAPOR 0.832 QATAR 0.3607 GERMANY 0.3397 JORDAN 0.9151 NORWAY 0.2698 QATAR 0.5826 RUSSIA 0.4979 KAZAKHST 0.4643 RUSSIA 0.5107 SHANGHA 0.4617 KAZAKHST 0.5272 QATAR 0.6721 KAZAKHST 0.9082 JORDAN 0.6641 LIECHTENSTE -0.2994 COLOMBIA 0.3954 KAZAKHST 0.3838 RUSSIA 0.4006 UNITED ST 0.4307 VIETNAM 0.3728 JORDAN 0.4377 INDONESIA 0.8186 ISRAEL 0.338 PORTUGAL 0.2487 INDONESIA 0.8517 PERU 0.25 INDONESIA 0.5681 COSTA RIC 0.4811 SERBIA 0.4603 SHANGHA 0.4993 MEXICO 0.4517 COSTA RIC 0.5129 JORDAN 0.6454 ARA BEMIR 0.893 THAILAND 0.5556 GERMANY -0.2822 MEXICO 0.3611 ALBANIA 0.373 JORDAN 0.3074 PERU 0.3908 RUSSIA 0.3167 KAZAKHST 0.4068 JORDAN 0.7924 UNITED ST 0.3213 SLOVENIA 0.2458 ARA BEMIR 0.8356 ISRAEL 0.2412 ALBANIA 0.5664 TURKEY 0.4527 QATAR 0.3785 SINGAPOR 0.4913 COLOMBIA 0.4411 COLOMBIA 0.4939 ARA BEMIR 0.5994 THAILAND 0.8798 ARA BEMIR 0.5514 FINLAND -0.2821 CHILE 0.3052 SWITZERLA 0.3649 LIECHTENS 0.2926 PORTUGAL 0.3739 HONGKONG 0.2833 BRAZIL 0.4038 THAILAND 0.7623 ICELAND 0.2418 HONGKON 0.2101 THAILAND 0.8317 LIECHTENS 0.2312 TUNISIA 0.4041 VIETNAM 0.4495 BULGARIA 0.3775 INDONESIA 0.488 LITHUANIA 0.4399 ICELAND 0.4925 THAILAND 0.5274 COLOMBIA 0.7285 SHANGHA 0.5457 AUSTRIA -0.249 INDONESIA 0.3033 ICELAND 0.352 MEXICO 0.2813 KAZAKHST 0.3521 LITHUANIA 0.2773 SINGAPOR 0.4021 PERU 0.7423 CANADA 0.2297 HUNGARY 0.1926 SINGAPOR 0.7978 JORDAN 0.2141 KAZAKHST 0.384 COLOMBIA 0.418 POLAND 0.3705 ARA BEMIR 0.4828 INDONESIA 0.4254 BRAZIL 0.4666 RUSSIA 0.5237 INDONESIA 0.7175 BULGARIA 0.5438 SWITZERLAN -0.2351 AUSTRIA 0.285 SPAIN 0.3417 COLOMBIA 0.2676 CANADA 0.3439 ISRAEL 0.2732 TUNISIA 0.4009 ARA BEMIR 0.7147 DENMARK 0.2295 UNITED ST 0.1919 PERU 0.7752 TURKEY 0.2113 SINGAPOR 0.3431 ARA BEMIR 0.3979 ARA BEMIR 0.364 TUNISIA 0.4746 PERU 0.3793 JORDAN 0.4585 MALAYSIA 0.4574 BULGARIA 0.671 QATAR 0.4832 ESTONIA -0.1655 SPAIN 0.2293 GERMANY 0.277 SINGAPOR 0.2356 ALBANIA 0.334 LIECHTENST 0.2558 THAILAND 0.389 VIETNAM 0.692 INDONESIA 0.2158 SWITZERLA 0.1883 QATAR 0.7076 ICELAND 0.1846 MALAYSIA 0.3424 UNITED ST 0.3772 ISRAEL 0.3416 MALAYSIA 0.4478 ROMANIA 0.3672 PORTUGAL 0.437 COLOMBIA 0.4416 MALAYSIA 0.6203 COLOMBIA 0.3937 UNITED STAT -0.1336 BRAZIL 0.229 COLOMBIA 0.223 ROMANIA 0.2334 ARGENTIN 0.3062 AUSTRIA 0.2252 INDONESIA 0.3709 MEXICO 0.6503 ALBANIA 0.2086 TAIPEI 0.1751 TUNISIA 0.5554 MALAYSIA 0.1768 ISRAEL 0.3295 ISRAEL 0.3658 TUNISIA 0.2522 LITHUANIA 0.4299 SINGAPOR 0.3566 PERU 0.3936 PERU 0.3554 PERU 0.6003 INDONESIA 0.3926 LUXEMBOUR -0.1059 PERU 0.2192 LUXEMBOU 0.2076 SHANGHA 0.2255 UNITED KIN 0.2969 SINGAPORE 0.2028 VIETNAM 0.3689 QATAR 0.6053 SINGAPOR 0.2077 CZECH 0.1461 ISRAEL 0.4669 KAZAKHST 0.168 SHANGHA 0.3248 PERU 0.355 GREECE 0.2411 THAILAND 0.4028 ARA BEMIR 0.3555 CHILE 0.3925 ROMANIA 0.3499 TUNISIA 0.5938 TURKEY 0.3828 HUNGARY -0.0799 PORTUGAL 0.2178 CHILE 0.2011 ARA BEMIR 0.22 SINGAPOR 0.2859 KOREA 0.1889 COLOMBIA 0.3551 TUNISIA 0.5961 PERU 0.1594 LUXEMBOU 0.1396 MEXICO 0.4238 QATAR 0.1442 MEXICO 0.3234 MONTENE 0.3462 CHILE 0.2244 TURKEY 0.3734 JORDAN 0.3549 ARA BEMIR 0.3843 INDONESIA 0.3232 RUSSIA 0.5446 UNITED ST 0.3353 UNITED KING -0.0781 SCOTLAND 0.2034 URUGUAY 0.162 VIETNAM 0.217 COLOMBIA 0.2797 ESTONIA 0.1860 ARA BEMIR 0.3529 COLOMBIA 0.5096 COLOMBIA 0.158 MACAO-CH 0.1374 ICELAND 0.4038 COSTA RIC 0.1407 COLOMBIA 0.322 PORTUGAL 0.3416 ROMANIA 0.2213 COSTA RIC 0.3721 PORTUGAL 0.3235 INDONESIA 0.3761 PORTUGAL 0.3094 MEXICO 0.5168 BRAZIL 0.329 LITHUANIA -0.0753 RUSSIA 0.1841 LITHUANIA 0.1501 MONTENE 0.2113 ISRAEL 0.2695 UNITED KIN 0.1449 UNITED KIN 0.3492 ROMANIA 0.4972 SWEDEN 0.1343 POLAND 0.1287 NEW ZEAL 0.3805 SWITZERLA 0.1393 UNITED ST 0.3149 MEXICO 0.3318 UNITED ST 0.2192 VIETNAM 0.3654 CANADA 0.2969 SINGAPOR 0.3593 UNITED ST 0.3079 BRAZIL 0.4496 CHILE 0.3242 CZECH -0.0725 TUNISIA 0.1715 TURKEY 0.1275 UNITED ST 0.2112 SCOTLAND 0.2681 INDONESIA 0.1282 ICELAND 0.3294 BRAZIL 0.4878 UNITED KIN 0.1288 ISRAEL 0.1284 CANADA 0.371 THAILAND 0.1221 BRAZIL 0.3073 JORDAN 0.3287 COSTA RIC 0.2107 BRAZIL 0.3648 THAILAND 0.2913 SHANGHA 0.3559 BRAZIL 0.2956 SWEDEN 0.4433 MEXICO 0.314 ISRAEL -0.0597 VIETNAM 0.1434 CROATIA 0.1255 ISRAEL 0.2049 CHILE 0.2403 IRELAND 0.1205 CHILE 0.3206 TURKEY 0.4368 SWITZERLA 0.115 SPAIN 0.1277 DENMARK 0.3482 COLOMBIA 0.0984 ICELAND 0.2952 ESTONIA 0.3091 TURKEY 0.2092 COLOMBIA 0.3567 BRAZIL 0.2778 QATAR 0.3449 SINGAPOR 0.2937 ROMANIA 0.4081 PERU 0.304 POLAND -0.037 LIECHTENSTE 0.1302 HUNGARY 0.1216 BULGARIA 0.2005 ROMANIA 0.2304 UNITED STA 0.1168 QATAR 0.2898 SHANGHA 0.4367 COSTA RIC 0.1141 KAZAKHST 0.1268 UNITED KIN 0.3197 NETHERLA 0.0947 PERU 0.2691 SINGAPOR 0.291 BRAZIL 0.2052 BULGARIA 0.3534 MALAYSIA 0.2297 ARGENTIN 0.3289 UNITED KIN 0.2778 COSTA RIC 0.3733 VIETNAM 0.297 CANADA -0.0173 UNITED KING 0.1282 MEXICO 0.1084 MALAYSIA 0.1897 URUGUAY 0.2257 CZECH 0.1111 COSTA RIC 0.2895 DENMARK 0.386 GERMANY 0.1094 SCOTLAND 0.1157 CHILE 0.3094 ESTONIA 0.0901 UNITED KIN 0.2676 CHILE 0.2874 HUNGARY 0.1948 MEXICO 0.339 BULGARIA 0.2255 URUGUAY 0.3247 CANADA 0.2735 ARGENTIN 0.3731 MONTENE 0.2917 SCOTLAND -0.0151 HUNGARY 0.1215 NORWAY 0.0825 UNITED KIN 0.1843 RUSSIA 0.2039 MACAO-CH 0.1009 ISRAEL 0.2877 COSTA RIC 0.3143 MALAYSIA 0.1064 SLOVAKIA 0.1002 AUSTRALIA 0.2865 MEXICO 0.0822 THAILAND 0.2605 QATAR 0.2664 NORWAY 0.1831 QATAR 0.325 ICELAND 0.2089 UNITED KIN 0.3246 SCOTLAND 0.269 TURKEY 0.3179 PORTUGAL 0.2702 BELGIUM 0.0061 IRELAND 0.1139 SERBIA 0.0666 TURKEY 0.18 NEW ZEAL 0.2033 SCOTLAND 0.0952 NEW ZEAL 0.2746 HONGKON 0.2925 RUSSIA 0.1029 AUSTRIA 0.0835 TURKEY 0.2699 ARA BEMIR 0.0735 TURKEY 0.2572 INDONESIA 0.2654 PERU 0.171 PERU 0.302 UNITED ST 0.1977 MALAYSIA 0.3155 MONTENE 0.2194 CHILE 0.3106 SERBIA 0.2676 SLOVAKIA 0.0105 CROATIA 0.0902 PORTUGAL 0.0105 INDONESIA 0.1787 MEXICO 0.203 HUNGARY 0.0772 CANADA 0.2739 CHILE 0.2871 GREECE 0.098 CANADA 0.0812 COSTA RIC 0.2545 LATVIA 0.0655 SCOTLAND 0.1803 URUGUAY 0.2593 GERMANY 0.1686 TAIPEI 0.3011 CHILE 0.1961 SCOTLAND 0.3007 SERBIA 0.2172 UNITED ST 0.3096 ROMANIA 0.2456 PORTUGAL 0.0171 FRANCE 0.0855 INDONESIA 0.0104 PERU 0.1783 SHANGHA 0.1675 LATVIA 0.071 LITHUANIA 0.2675 RUSSIA 0.2836 LIECHTENS 0.0881 CROATIA 0.072 COLOMBIA 0.2399 SINGAPOR 0.0599 CHILE 0.1793 SHANGHA 0.2529 COLOMBIA 0.1632 CANADA 0.2829 TURKEY 0.1941 AUSTRALIA 0.2932 SHANGHA 0.2093 ICELAND 0.3017 CANADA 0.2402 NORWAY 0.0209 TURKEY 0.0853 ARA BEMIR -0.0029 CANADA 0.1707 LIECHTENS 0.1637 BELGIUM 0.0695 DENMARK 0.2638 URUGUAY 0.2669 SCOTLAND 0.0712 ICELAND 0.0542 UNITED ST 0.2399 URUGUAY 0.0474 CANADA 0.1698 THAILAND 0.2461 PORTUGAL 0.151 UNITED ST 0.28 URUGUAY 0.1824 CANADA 0.2708 CHILE 0.2072 VIETNAM 0.2981 LATVIA 0.2393 LATVIA 0.0278 UNITED STAT 0.083 JORDAN -0.0074 ICELAND 0.1678 CZECH 0.1334 THAILAND 0.0693 SCOTLAND 0.261 BULGARIA 0.2062 NETHERLA 0.0655 LITHUANIA 0.0462 BRAZIL 0.2299 CHILE 0.0463 PORTUGAL 0.1437 MALAYSIA 0.2208 CANADA 0.1326 URUGUAY 0.2683 MONTENE 0.1705 RUSSIA 0.2557 NEW ZEAL 0.1929 SERBIA 0.288 ISRAEL 0.2371 SHANGHAI 0.0339 NEW ZEALAN 0.0744 IRELAND -0.0231 LATVIA 0.166 IRELAND 0.1316 POLAND 0.0659 PORTUGAL 0.2475 GREECE 0.206 TUNISIA 0.0628 SWEDEN 0.0336 SCOTLAND 0.2151 SHANGHA 0.0321 DENMARK 0.1415 SERBIA 0.1974 SWEDEN 0.1203 PORTUGAL 0.2665 UNITED KIN 0.1563 SWEDEN 0.2287 TURKEY 0.1728 PORTUGAL 0.2731 ARGENTIN 0.2326 KAZAKHSTA 0.0412 AUSTRALIA 0.059 MONTENE -0.0256 TUNISIA 0.1641 AUSTRALIA 0.1299 MEXICO 0.0639 AUSTRALIA 0.2312 ISRAEL 0.1686 LUXEMBOU 0.055 ALBANIA 0.0162 LITHUANIA 0.1708 LUXEMBOU 0.0199 MONTENE 0.1327 CANADA 0.1956 INDONESIA 0.0814 SCOTLAND 0.2641 LATVIA 0.1403 VIETNAM 0.2268 AUSTRALIA 0.1726 DENMARK 0.2391 GREECE 0.2086 AUSTRALIA 0.0432 ARGENTINA 0.0568 DENMARK -0.0266 ESTONIA 0.152 INDONESIA 0.1266 SWITZERLA 0.0501 URUGUAY 0.1968 ARGENTIN 0.1613 AUSTRALIA 0.0504 FRANCE 0.004 SWEDEN 0.1681 AUSTRALIA 0.0135 AUSTRIA 0.1167 LATVIA 0.194 ESTONIA 0.0704 UNITED KIN 0.2622 SCOTLAND 0.1345 BULGARIA 0.2147 TUNISIA 0.161 NORWAY 0.2371 SINGAPOR 0.2026 NEW ZEALAN 0.0947 SWITZERLAN 0.0568 UNITED KIN -0.0286 SCOTLAND 0.1392 THAILAND 0.1245 COSTA RICA 0.0356 NORWAY 0.1924 ICELAND 0.153 MEXICO 0.0438 IRELAND 0.001 LIECHTENS 0.1458 ARGENTIN 0.0039 LIECHTENS 0.1036 MACAO-CH 0.1521 RUSSIA 0.0691 LATVIA 0.2605 AUSTRALIA 0.1329 UNITED ST 0.2056 ISRAEL 0.1566 URUGUAY 0.2316 TUNISIA 0.1985 SLOVENIA 0.1005 FINLAND 0.0548 PERU -0.0365 IRELAND 0.1376 LATVIA 0.0936 PORTUGAL 0.0264 SWEDEN 0.1827 MACAO-CH 0.1472 AUSTRIA 0.0416 AUSTRALIA -0.0103 NORWAY 0.1439 AUSTRIA 0.0024 SPAIN 0.1027 LITHUANIA 0.15 SLOVENIA 0.0686 ICELAND 0.244 RUSSIA 0.1228 FINLAND 0.185 SLOVAKIA 0.1341 MONTENE 0.2165 MALAYSIA 0.1937 RUSSIA 0.1051 MONTENEGR 0.05 SWEDEN -0.0384 JAPAN 0.1306 TUNISIA 0.0874 ROMANIA 0.0083 ARGENTIN 0.1356 UNITED KIN 0.1437 ROMANIA 0.0284 ESTONIA -0.0129 ROMANIA 0.1401 PORTUGAL -0.0019 BULGARIA 0.1026 BRAZIL 0.1477 SHANGHA 0.0675 ROMANIA 0.2438 ARGENTIN 0.118 TURKEY 0.1812 ITALY 0.1261 LATVIA 0.2067 LIECHTENS 0.1715 HONGKONG 0.1162 ICELAND 0.0442 BELGIUM -0.0393 SWITZERLA 0.1223 LITHUANIA 0.0865 MONTENEG 0.0011 IRELAND 0.1217 SWEDEN 0.1283 FINLAND 0.0278 NORWAY -0.0139 BULGARIA 0.1344 UNITED ST -0.0056 BELGIUM 0.0877 IRELAND 0.1323 LUXEMBOU 0.062 CHILE 0.2312 DENMARK 0.1172 NEW ZEAL 0.1619 LATVIA 0.0949 ISRAEL 0.2038 LITHUANIA 0.1544 GREECE 0.1243 CANADA 0.0292 NETHERLA -0.0423 THAILAND 0.0721 SWITZERLA 0.0822 TAIPEI -0.0018 UNITED ST 0.12 PORTUGAL 0.1054 ESTONIA 0.0219 TURKEY -0.0144 IRELAND 0.123 UNITED KIN -0.0078 IRELAND 0.0708 TUNISIA 0.1296 ICELAND 0.0595 ISRAEL 0.2144 NEW ZEAL 0.1059 DENMARK 0.1404 MEXICO 0.0785 LITHUANIA 0.187 COSTA RIC 0.1399 IRELAND 0.1266 LATVIA 0.023 SLOVENIA -0.0473 BELGIUM 0.0611 GREECE 0.0809 ARA BEMIR -0.005 LATVIA 0.1161 NEW ZEAL 0.1018 NEW ZEAL 0.0104 ARA BEMIR -0.0171 SHANGHA 0.1164 FRANCE -0.0114 ROMANIA 0.0665 HONGKON 0.119 AUSTRIA 0.0555 ARGENTIN 0.2028 SWEDEN 0.0841 TAIPEI 0.122 SWEDEN 0.0694 LIECHTENS 0.128 UNITED KIN 0.1181 CROATIA 0.1376 URUGUAY 0.0211 UNITED ST -0.0682 BRAZIL 0.0461 BRAZIL 0.0665 GERMANY -0.0223 LIECHTENS 0.0748 LITHUANIA 0.0958 ITALY 0.0078 JORDAN -0.0449 URUGUAY 0.1145 JAPAN -0.0139 URUGUAY 0.065 SPAIN 0.1055 DENMARK 0.054 AUSTRALIA 0.155 SERBIA 0.0835 ROMANIA 0.1163 NORWAY 0.0633 MACAO-CH 0.1204 URUGUAY 0.0989 ALBANIA 0.1418 ESTONIA 0.0181 THAILAND -0.0797 PORTUGAL 0.0291 POLAND 0.0636 CANADA -0.0229 TURKEY 0.0469 LIECHTENS 0.0957 URUGUAY -0.0151 UNITED KIN -0.0536 PORTUGAL 0.1135 INDONESIA -0.0207 MACAO-CH 0.0602 SCOTLAND 0.1051 LIECHTENS 0.0504 NEW ZEAL 0.1261 QATAR 0.0825 IRELAND 0.0727 CROATIA 0.0596 CANADA 0.0976 SLOVENIA 0.0982 SINGAPORE 0.1781 THAILAND 0.0093 CANADA -0.0832 AUSTRIA 0.0207 MONTENE 0.0472 LUXEMBOU -0.0232 FINLAND 0.0407 AUSTRALIA 0.0955 LITHUANIA -0.0218 ITALY -0.0731 RUSSIA 0.0378 CROATIA -0.0223 LUXEMBOU 0.059 CROATIA 0.0902 SPAIN 0.0444 HONGKON 0.1214 ISRAEL 0.0676 MONTENE 0.0626 LIECHTENS 0.059 SWITZERLA 0.0955 CZECH 0.097 SPAIN 0.179 ISRAEL 0.0066 SCOTLAND -0.0938 MACAO-CH 0.0013 SPAIN 0.0324 ICELAND -0.0255 GREECE 0.0222 UNITED ST 0.0795 BELGIUM -0.0344 BULGARIA -0.0898 MONTENE 0.0289 FINLAND -0.025 NORWAY 0.0556 UNITED KIN 0.0874 URUGUAY 0.0327 ITALY 0.1051 SWITZERLA 0.0532 LITHUANIA 0.0591 AUSTRIA 0.0583 SINGAPOR 0.0912 CROATIA 0.0534 MACAO-CHI 0.1884 MALAYSIA -0.0044 BULGARIA -0.094 URUGUAY -0.0027 GERMANY 0.0144 SPAIN -0.0267 SHANGHA 0.0218 SCOTLAND 0.0722 TURKEY -0.0458 LATVIA -0.0998 SWITZERLA -0.0023 ALBANIA -0.0416 ITALY 0.0413 SLOVENIA 0.0683 SINGAPOR -0.0011 GREECE 0.1022 LIECHTENS 0.0485 LATVIA 0.0355 ARGENTIN 0.0458 NEW ZEAL 0.0565 AUSTRALIA 0.0241 MONTENEGR 0.1921 ITALY -0.0071 FRANCE -0.1027 CZECH -0.0097 MALAYSIA -0.0027 COLOMBIA -0.0286 ESTONIA 0.0151 TAIPEI 0.0636 SHANGHA -0.0497 CHILE -0.0999 ARGENTIN -0.0104 BELGIUM -0.0483 ARGENTIN 0.0223 AUSTRALIA 0.0659 SWITZERLA -0.0167 MONTENE 0.0975 TAIPEI 0.0382 HONGKON 0.0346 SLOVENIA 0.0378 SLOVAKIA 0.0331 SCOTLAND 0.0208 SERBIA 0.1924 ROMANIA -0.0119 NEW ZEAL -0.143 SERBIA -0.0106 DENMARK -0.0146 PERU -0.0293 BULGARIA -0.0353 CANADA 0.059 IRELAND -0.0542 RUSSIA -0.1015 HONGKON -0.0203 BRAZIL -0.0706 LITHUANIA 0.0157 ARGENTIN 0.0573 IRELAND -0.0232 MACAO-CH 0.0888 IRELAND 0.0336 ISRAEL 0.0279 VIETNAM 0.007 AUSTRALIA 0.029 HUNGARY -0.0013 ARA BEMIRA 0.1943 ARA BEMIRA -0.016 GREECE -0.1502 CHILE -0.012 SERBIA -0.0166 SLOVENIA -0.0321 HUNGARY -0.044 IRELAND 0.0405 SLOVENIA -0.06 DENMARK -0.1075 FINLAND -0.0225 GREECE -0.0806 GERMANY -0.0006 ROMANIA 0.0391 SCOTLAND -0.0242 SERBIA 0.079 HONGKON 0.0269 SPAIN 0.0029 LITHUANIA 0.0066 FINLAND 0.0195 SWITZERLA -0.0066 VIETNAM 0.2146 SLOVENIA -0.0341 TUNISIA -0.1532 SLOVENIA -0.0123 FINLAND -0.0289 ITALY -0.0356 SPAIN -0.0444 MONTENE 0.0193 BRAZIL -0.0612 BELGIUM -0.1186 SPAIN -0.0586 LITHUANIA -0.0838 VIETNAM -0.002 ITALY 0.0386 CROATIA -0.0371 IRELAND 0.066 VIETNAM 0.0181 MACAO-CH -0.0268 HUNGARY -0.0008 TAIPEI 0.0054 SWEDEN -0.0394 BULGARIA 0.2381 SINGAPORE -0.0529 MALAYSIA -0.1559 AUSTRALIA -0.0158 TURKEY -0.0462 DENMARK -0.0455 SWITZERLA -0.0531 SWITZERLA 0.0106 THAILAND -0.0657 ROMANIA -0.1315 LATVIA -0.0913 CZECH -0.0862 HONGKON -0.0119 POLAND 0.0322 UNITED KIN -0.0451 ESTONIA 0.0514 TUNISIA -0.0187 NORWAY -0.047 COSTA RIC -0.0393 CZECH -0.024 MACAO-CH -0.047 TURKEY 0.2656 GERMANY -0.0587 SINGAPOR -0.1591 LUXEMBOURG-0.027 SLOVENIA -0.0567 TURKEY -0.0776 SERBIA -0.0867 ESTONIA -0.0026 BULGARIA -0.0695 QATAR -0.1463 VIETNAM -0.0913 SCOTLAND -0.0918 SWEDEN -0.0122 HUNGARY -0.0019 FINLAND -0.0653 CROATIA -0.0186 HUNGARY -0.0188 SWITZERLAND -0.0518 GREECE -0.0468 NETHERLANDS-0.0545 GERMANY -0.049 QATAR 0.2667 LUXEMBOURG -0.0712 JAPAN -0.1622 HONGKONG -0.0331 ESTONIA -0.0605 NORWAY -0.0816 RUSSIA -0.0917 FRANCE -0.0137 LATVIA -0.072 NEW ZEALAND-0.1574 GREECE -0.0934 GERMANY -0.0931 SWITZERLAND-0.0235 NEW ZEALAND-0.0077 LATVIA -0.0772 SPAIN -0.0231 MACAO-CHINA -0.0454 LIECHTENSTEIN-0.0645 POLAND -0.0521 GERMANY -0.0582 FRANCE -0.0531 ITALY 0.2729 BULGARIA -0.0953 BRAZIL -0.166 QATAR -0.0357 FRANCE -0.07 CROATIA -0.1264 GERMANY -0.1339 ITALY -0.0273 SPAIN -0.0734 GREECE -0.1581 GERMANY -0.1049 POLAND -0.0931 SERBIA -0.0498 GERMANY -0.0088 ITALY -0.0913 FINLAND -0.0235 FINLAND -0.0474 SERBIA -0.0665 SWITZERLAND-0.0542 ITALY -0.064 SLOVAKIA -0.0587 FRANCE 0.2802 DENMARK -0.1001 FINLAND -0.1708 GERMANY -0.0396 HUNGARY -0.0759 SLOVAKIA -0.1276 POLAND -0.144 LATVIA -0.0492 POLAND -0.0748 MEXICO -0.1789 FRANCE -0.1141 CANADA -0.1135 AUSTRALIA -0.055 FINLAND -0.0126 AUSTRALIA -0.0933 DENMARK -0.0399 LUXEMBOURG-0.0533 JAPAN -0.0829 DENMARK -0.0642 POLAND -0.0966 FINLAND -0.0608 INDONESIA 0.2807 BELGIUM -0.1106 AUSTRALIA -0.1817 CROATIA -0.0527 AUSTRIA -0.0852 SERBIA -0.1589 FRANCE -0.1588 GERMANY -0.113 PORTUGAL -0.0845 NETHERLANDS -0.1883 LUXEMBOURG-0.1547 NEW ZEALAND-0.1172 SLOVENIA -0.0568 AUSTRIA -0.0259 MEXICO -0.096 SWEDEN -0.0408 ESTONIA -0.0919 TUNISIA -0.0946 NETHERLANDS-0.0694 UNITED KINGDOM -0.0986 ICELAND -0.0658 KOREA 0.3065 JAPAN -0.1334 RUSSIA -0.1856 HUNGARY -0.0533 LUXEMBOURG-0.0961 URUGUAY -0.1605 CZECH -0.191 FINLAND -0.1291 ARGENTINA -0.0924 SERBIA -0.2152 ESTONIA -0.1842 TUNISIA -0.129 GREECE -0.061 DENMARK -0.0345 NETHERLANDS -0.0984 NORWAY -0.0996 SPAIN -0.098 ESTONIA -0.1254 ESTONIA -0.0825 JAPAN -0.1288 TAIPEI -0.068 TAIPEI 0.3185 SWEDEN -0.1355 TAIPEI -0.1966 ITALY -0.0593 ITALY -0.1105 AUSTRALIA -0.1724 ITALY -0.2002 NORWAY -0.1482 MONTENEGRO-0.0997 PERU -0.2158 KOREA -0.2058 ITALY -0.1321 FRANCE -0.0840 LUXEMBOURG-0.0621 ARGENTINA -0.1233 SWITZERLAND-0.1019 AUSTRIA -0.1151 GREECE -0.1689 IRELAND -0.0831 SCOTLAND -0.1367 NEW ZEALAND-0.0737 PERU 0.3348 GREECE -0.1734 ARGENTINA -0.1988 SLOVAKIA -0.0867 CROATIA -0.1491 NETHERLANDS-0.1782 SLOVENIA -0.2092 POLAND -0.1609 NORWAY -0.1009 THAILAND -0.2359 SLOVENIA -0.2110 IRELAND -0.1357 HUNGARY -0.0951 TAIPEI -0.0783 CZECH -0.1318 HUNGARY -0.1106 KOREA -0.1158 BELGIUM -0.1784 GERMANY -0.0922 SPAIN -0.1473 IRELAND -0.0907 COLOMBIA 0.3500 SERBIA -0.2197 ITALY -0.2015 SPAIN -0.0911 COSTA RICA -0.1662 BULGARIA -0.19 HONGKONG -0.2296 CZECH -0.1616 HUNGARY -0.1053 INDONESIA -0.2439 ITALY -0.2307 SPAIN -0.1393 RUSSIA -0.1105 CZECH -0.0846 LITHUANIA -0.1497 SLOVENIA -0.1142 BELGIUM -0.1278 SLOVAKIA -0.1828 TAIPEI -0.0985 ESTONIA -0.1505 AUSTRIA -0.0962 CHILE 0.3569 SHANGHAI -0.2267 LATVIA -0.2138 TAIPEI -0.104 TAIPEI -0.1712 MALAYSIA -0.1928 CROATIA -0.2534 LUXEMBOURG-0.162 CHILE -0.1101 MALAYSIA -0.2446 AUSTRIA -0.2428 SLOVENIA -0.1433 NEW ZEALAND-0.1162 KOREA -0.0895 FRANCE -0.1862 JAPAN -0.1224 NORWAY -0.1393 HUNGARY -0.1917 URUGUAY -0.0989 GREECE -0.1682 NETHERLANDS-0.1153 JAPAN 0.3612 CZECH -0.2319 QATAR -0.2184 NEW ZEALAND-0.1059 ICELAND -0.1719 CHILE -0.1937 MONTENEGRO-0.2623 SPAIN -0.1635 CZECH -0.1180 FINLAND -0.2489 TAIPEI -0.2513 RUSSIA -0.1434 NETHERLANDS -0.1279 GREECE -0.0952 NEW ZEALAND-0.194 SLOVAKIA -0.1238 GREECE -0.1508 ITALY -0.2236 SPAIN -0.1043 KOREA -0.1697 BELGIUM -0.1154 URUGUAY 0.3626 TAIPEI -0.2499 VIETNAM -0.2558 NORWAY -0.1079 BELGIUM -0.1951 SWEDEN -0.1951 MACAO-CHINA -0.2633 SERBIA -0.1684 SERBIA -0.1553 URUGUAY -0.2676 BELGIUM -0.2742 MACAO-CHINA-0.1733 ESTONIA -0.1486 LIECHTENSTEIN -0.0963 MALAYSIA -0.2014 POLAND -0.1545 CROATIA -0.1526 FRANCE -0.2402 ICELAND -0.1075 SHANGHAI -0.1961 LUXEMBOURG-0.1238 ROMANIA 0.3928 SLOVAKIA -0.2621 ROMANIA -0.2827 DENMARK -0.1189 SLOVAKIA -0.1954 NEW ZEALAND-0.2423 LUXEMBOURG-0.2753 HUNGARY -0.1817 HONGKONG -0.1620 VIETNAM -0.268 POLAND -0.2948 SLOVAKIA -0.1834 POLAND -0.1597 ICELAND -0.1054 HONGKONG -0.2485 FRANCE -0.1605 NETHERLANDS -0.1551 POLAND -0.2489 FRANCE -0.1115 SLOVENIA -0.2208 ESTONIA -0.1686 MALAYSIA 0.4263 NORWAY -0.2689 SLOVAKIA -0.3099 FRANCE -0.1255 NORWAY -0.212 GREECE -0.2564 TAIPEI -0.3393 SLOVAKIA -0.1951 SLOVAKIA -0.1681 MONTENEGRO-0.2725 SERBIA -0.3187 TAIPEI -0.1855 JAPAN -0.1773 NETHERLANDS -0.1374 BELGIUM -0.2736 CZECH -0.1900 JAPAN -0.1723 KOREA -0.2490 BELGIUM -0.1190 LUXEMBOURG-0.2450 ITALY -0.1777 COSTA RICA 0.4292 KOREA -0.2879 SHANGHAI -0.3132 SWEDEN -0.1404 HONGKONG -0.2135 JORDAN -0.2642 SLOVAKIA -0.3501 KOREA -0.2013 FRANCE -0.1699 TUNISIA -0.3138 HUNGARY -0.3202 KOREA -0.2080 CROATIA -0.1793 SWITZERLAND-0.1769 THAILAND -0.2784 BELGIUM -0.2145 FRANCE -0.1802 CZECH -0.2693 FINLAND -0.1488 AUSTRIA -0.2571 SPAIN -0.1882 MEXICO 0.4386 JORDAN -0.3242 KOREA -0.3209 POLAND -0.1596 SWEDEN -0.2143 FRANCE -0.2841 BELGIUM -0.3683 JAPAN -0.2367 MACAO-CHINA -0.1878 COSTA RICA -0.3315 MACAO-CHINA -0.3358 BULGARIA -0.2297 TAIPEI -0.2039 SWEDEN -0.2501 SLOVAKIA -0.3162 LIECHTENSTEIN -0.2448 ITALY -0.1930 GERMANY -0.3216 LUXEMBOURG-0.1537 BELGIUM -0.2718 NORWAY -0.2118 THAILAND 0.5019 NETHERLANDS -0.3705 POLAND -0.3232 FINLAND -0.1911 NETHERLANDS -0.2209 FINLAND -0.3044 NETHERLANDS -0.3712 SLOVENIA -0.238 VIETNAM -0.1919 ARGENTINA -0.3352 NETHERLANDS -0.3538 HUNGARY -0.3098 LATVIA -0.2188 BELGIUM -0.3143 TAIPEI -0.3334 KOREA -0.2525 SLOVAKIA -0.1939 CROATIA -0.3618 HONGKONG -0.1755 HONGKONG -0.3549 POLAND -0.2133 ARGENTINA 0.5168 QATAR -0.3918 ESTONIA -0.3267 ARGENTINA -0.1912 MACAO-CHINA-0.228 QATAR -0.3222 KOREA -0.3894 BELGIUM -0.2564 CROATIA -0.2570 KOREA -0.3571 SLOVAKIA -0.3798 HONGKONG -0.3161 FINLAND -0.2890 NORWAY -0.3454 MACAO-CHINA -0.3377 LUXEMBOURG-0.3645 CZECH -0.2071 SLOVENIA -0.3676 CZECH -0.1917 CROATIA -0.3775 DENMARK -0.2319 JORDAN 0.5206 HONGKONG -0.4169 CZECH -0.3597 GREECE -0.1927 VIETNAM -0.3261 BRAZIL -0.3322 AUSTRIA -0.3988 CROATIA -0.2635 KOREA -0.3835 JAPAN -0.4156 CROATIA -0.4541 ROMANIA -0.3371 CZECH -0.3841 FRANCE -0.4506 KOREA -0.3708 GERMANY -0.4057 SLOVENIA -0.2171 LUXEMBOURG -0.3889 MACAO-CHINA-0.3782 FRANCE -0.4167 JAPAN -0.2590 BRAZIL 0.5282 POLAND -0.4189 HONGKONG -0.3963 KOREA -0.2600 JAPAN -0.5042 TUNISIA -0.4225 JAPAN -0.4972 AUSTRIA -0.3243 TAIPEI -0.4571 COLOMBIA -0.4460 CZECH -0.5253 SERBIA -0.4239 SLOVAKIA -0.4276 SLOVAKIA -0.4891 VIETNAM -0.6131 NETHERLANDS -0.4181 GERMANY -0.2196 AUSTRIA -0.4349 JAPAN -0.6310 HUNGARY -0.4565 KOREA -0.3141 TUNISIA 0.6452 MACAO-CHINA -0.4339 MACAO-CHINA -0.4958 NETHERLANDS -0.3006 KOREA -0.7320 ARGENTINA -0.4921 ROMANIA -0.5637 NETHERLANDS -0.3402 JAPAN -0.5251 BRAZIL -0.4499 JAPAN -0.5848 MONTENEGRO-0.4495 KOREA -0.5443 JAPAN -0.5961 JAPAN -0.7306 AUSTRIA -0.482 POLAND -0.4152 NETHERLANDS -0.4456 KOREA -0.7692 IRELAND -0.5682 HONGKONG -0.3228

Number of Times Appeared in the Bottom 15 Country List in 21 Non-cognitive Variables (WORST) KOREA 20 JAPAN 15 SLOVAKIA 15 TAIPEI 13 BELGIUM 12 CZECH 12 NETHERLANDS 12 HONGKONG 11

Number of Times Appeared in the Bottom 15 Country List in 21 Non-cognitive Variables ARAB EMIRATES 0 CANADA 0 PORTUGAL 0 TURKEY 0 UNITED STATES 0 (BEST) AUSTRALIA 1 ALBANIA 1 BULGARIA 1 INDONESIA 1 ISRAEL 1 KAZAKHSTAN 1 MEXICO 1 SCOTLAND 1 SINGAPORE 1 UNITED KINGDOM 1

PISA 2012: Countries Rankings on Non-cognitive Outcomes Country ANXMAT _mean Country ATSCHL_ mean Country BELONG_ mean Country CLSMAN_ mean Country COGACT_ mean Country DISCLIMA _mean Country INSTMOT _mean Country INTMAT_ mean Country SCMAT_ mean Country MATHEFF _mean Country SUBNOR M_mean Country MATINTFC _mean MATWKE Country TH_mean Country PERSEV_ mean Country OPENPS_ mean Country MTSUP_ mean Country STUDREL _mean Country TEACHSUP_ mean Country TCHBEHFA _mean Country TCHBEHSO _mean Country TCHBEHTD _mean NETHERLAN -0.3984 ALBANIA 0.5274 LIECHTENS 0.5803 KAZAKHST 0.6537 JORDAN 0.6461 KAZAKHSTA 0.7137 PERU 0.5549 ALBANIA 0.9413 ARA BEMIR 0.4173 SHANGHA 0.9365 ALBANIA 1.0654 SWEDEN 0.3529 JORDAN 0.8114 KAZAKHST 0.7771 MONTENE 0.619 ALBANIA 0.8333 KAZAKHST 0.7419 ALBANIA 0.618 KAZAKHST 0.769 QATAR 1.0816 ALBANIA 1.0167 SWEDEN -0.3457 KAZAKHSTAN 0.471 AUSTRIA 0.5606 ALBANIA 0.5788 BULGARIA 0.5297 JAPAN 0.6775 MALAYSIA 0.5391 MALAYSIA 0.91 JORDAN 0.4006 LIECHTENS 0.507 MALAYSIA 1.0485 DENMARK 0.3436 ARA BEMIR 0.6747 ALBANIA 0.6403 JORDAN 0.6057 KAZAKHST 0.6896 ALBANIA 0.7211 THAILAND 0.6042 BULGARIA 0.737 JORDAN 0.9914 KAZAKHST 0.9196 ICELAND -0.3318 COSTA RICA 0.4273 ISRAEL 0.4182 COSTA RIC 0.4927 QATAR 0.4985 SHANGHAI 0.5779 ALBANIA 0.5363 KAZAKHST 0.8795 KAZAKHST 0.3918 SINGAPOR 0.4373 KAZAKHST 0.9701 VIETNAM 0.326 COSTA RIC 0.6576 BULGARIA 0.5849 ALBANIA 0.5065 JORDAN 0.6117 COSTA RIC 0.4639 MEXICO 0.5581 ALBANIA 0.6971 ALBANIA 0.9596 RUSSIA 0.7615 DENMARK -0.322 LITHUANIA 0.4189 COSTA RIC 0.3883 LITHUANIA 0.4705 ARA BEMIR 0.4573 ALBANIA 0.3867 MEXICO 0.4875 SINGAPOR 0.832 QATAR 0.3607 GERMANY 0.3397 JORDAN 0.9151 NORWAY 0.2698 QATAR 0.5826 RUSSIA 0.4979 KAZAKHST 0.4643 RUSSIA 0.5107 SHANGHA 0.4617 KAZAKHST 0.5272 QATAR 0.6721 KAZAKHST 0.9082 JORDAN 0.6641 LIECHTENSTE -0.2994 COLOMBIA 0.3954 KAZAKHST 0.3838 RUSSIA 0.4006 UNITED ST 0.4307 VIETNAM 0.3728 JORDAN 0.4377 INDONESIA 0.8186 ISRAEL 0.338 PORTUGAL 0.2487 INDONESIA 0.8517 PERU 0.25 INDONESIA 0.5681 COSTA RIC 0.4811 SERBIA 0.4603 SHANGHA 0.4993 MEXICO 0.4517 COSTA RIC 0.5129 JORDAN 0.6454 ARA BEMIR 0.893 THAILAND 0.5556 GERMANY -0.2822 MEXICO 0.3611 ALBANIA 0.373 JORDAN 0.3074 PERU 0.3908 RUSSIA 0.3167 KAZAKHST 0.4068 JORDAN 0.7924 UNITED ST 0.3213 SLOVENIA 0.2458 ARA BEMIR 0.8356 ISRAEL 0.2412 ALBANIA 0.5664 TURKEY 0.4527 QATAR 0.3785 SINGAPOR 0.4913 COLOMBIA 0.4411 COLOMBIA 0.4939 ARA BEMIR 0.5994 THAILAND 0.8798 ARA BEMIR 0.5514 FINLAND -0.2821 CHILE 0.3052 SWITZERLA 0.3649 LIECHTENS 0.2926 PORTUGAL 0.3739 HONGKONG 0.2833 BRAZIL 0.4038 THAILAND 0.7623 ICELAND 0.2418 HONGKON 0.2101 THAILAND 0.8317 LIECHTENS 0.2312 TUNISIA 0.4041 VIETNAM 0.4495 BULGARIA 0.3775 INDONESIA 0.488 LITHUANIA 0.4399 ICELAND 0.4925 THAILAND 0.5274 COLOMBIA 0.7285 SHANGHA 0.5457 AUSTRIA -0.249 INDONESIA 0.3033 ICELAND 0.352 MEXICO 0.2813 KAZAKHST 0.3521 LITHUANIA 0.2773 SINGAPOR 0.4021 PERU 0.7423 CANADA 0.2297 HUNGARY 0.1926 SINGAPOR 0.7978 JORDAN 0.2141 KAZAKHST 0.384 COLOMBIA 0.418 POLAND 0.3705 ARA BEMIR 0.4828 INDONESIA 0.4254 BRAZIL 0.4666 RUSSIA 0.5237 INDONESIA 0.7175 BULGARIA 0.5438 SWITZERLAN -0.2351 AUSTRIA 0.285 SPAIN 0.3417 COLOMBIA 0.2676 CANADA 0.3439 ISRAEL 0.2732 TUNISIA 0.4009 ARA BEMIR 0.7147 DENMARK 0.2295 UNITED ST 0.1919 PERU 0.7752 TURKEY 0.2113 SINGAPOR 0.3431 ARA BEMIR 0.3979 ARA BEMIR 0.364 TUNISIA 0.4746 PERU 0.3793 JORDAN 0.4585 MALAYSIA 0.4574 BULGARIA 0.671 QATAR 0.4832 ESTONIA -0.1655 SPAIN 0.2293 GERMANY 0.277 SINGAPOR 0.2356 ALBANIA 0.334 LIECHTENST 0.2558 THAILAND 0.389 VIETNAM 0.692 INDONESIA 0.2158 SWITZERLA 0.1883 QATAR 0.7076 ICELAND 0.1846 MALAYSIA 0.3424 UNITED ST 0.3772 ISRAEL 0.3416 MALAYSIA 0.4478 ROMANIA 0.3672 PORTUGAL 0.437 COLOMBIA 0.4416 MALAYSIA 0.6203 COLOMBIA 0.3937 UNITED STAT -0.1336 BRAZIL 0.229 COLOMBIA 0.223 ROMANIA 0.2334 ARGENTIN 0.3062 AUSTRIA 0.2252 INDONESIA 0.3709 MEXICO 0.6503 ALBANIA 0.2086 TAIPEI 0.1751 TUNISIA 0.5554 MALAYSIA 0.1768 ISRAEL 0.3295 ISRAEL 0.3658 TUNISIA 0.2522 LITHUANIA 0.4299 SINGAPOR 0.3566 PERU 0.3936 PERU 0.3554 PERU 0.6003 INDONESIA 0.3926 LUXEMBOUR -0.1059 PERU 0.2192 LUXEMBOU 0.2076 SHANGHA 0.2255 UNITED KIN 0.2969 SINGAPORE 0.2028 VIETNAM 0.3689 QATAR 0.6053 SINGAPOR 0.2077 CZECH 0.1461 ISRAEL 0.4669 KAZAKHST 0.168 SHANGHA 0.3248 PERU 0.355 GREECE 0.2411 THAILAND 0.4028 ARA BEMIR 0.3555 CHILE 0.3925 ROMANIA 0.3499 TUNISIA 0.5938 TURKEY 0.3828 HUNGARY -0.0799 PORTUGAL 0.2178 CHILE 0.2011 ARA BEMIR 0.22 SINGAPOR 0.2859 KOREA 0.1889 COLOMBIA 0.3551 TUNISIA 0.5961 PERU 0.1594 LUXEMBOU 0.1396 MEXICO 0.4238 QATAR 0.1442 MEXICO 0.3234 MONTENE 0.3462 CHILE 0.2244 TURKEY 0.3734 JORDAN 0.3549 ARA BEMIR 0.3843 INDONESIA 0.3232 RUSSIA 0.5446 UNITED ST 0.3353 UNITED KING -0.0781 SCOTLAND 0.2034 URUGUAY 0.162 VIETNAM 0.217 COLOMBIA 0.2797 ESTONIA 0.1860 ARA BEMIR 0.3529 COLOMBIA 0.5096 COLOMBIA 0.158 MACAO-CH 0.1374 ICELAND 0.4038 COSTA RIC 0.1407 COLOMBIA 0.322 PORTUGAL 0.3416 ROMANIA 0.2213 COSTA RIC 0.3721 PORTUGAL 0.3235 INDONESIA 0.3761 PORTUGAL 0.3094 MEXICO 0.5168 BRAZIL 0.329 LITHUANIA -0.0753 RUSSIA 0.1841 LITHUANIA 0.1501 MONTENE 0.2113 ISRAEL 0.2695 UNITED KIN 0.1449 UNITED KIN 0.3492 ROMANIA 0.4972 SWEDEN 0.1343 POLAND 0.1287 NEW ZEAL 0.3805 SWITZERLA 0.1393 UNITED ST 0.3149 MEXICO 0.3318 UNITED ST 0.2192 VIETNAM 0.3654 CANADA 0.2969 SINGAPOR 0.3593 UNITED ST 0.3079 BRAZIL 0.4496 CHILE 0.3242 CZECH -0.0725 TUNISIA 0.1715 TURKEY 0.1275 UNITED ST 0.2112 SCOTLAND 0.2681 INDONESIA 0.1282 ICELAND 0.3294 BRAZIL 0.4878 UNITED KIN 0.1288 ISRAEL 0.1284 CANADA 0.371 THAILAND 0.1221 BRAZIL 0.3073 JORDAN 0.3287 COSTA RIC 0.2107 BRAZIL 0.3648 THAILAND 0.2913 SHANGHA 0.3559 BRAZIL 0.2956 SWEDEN 0.4433 MEXICO 0.314 ISRAEL -0.0597 VIETNAM 0.1434 CROATIA 0.1255 ISRAEL 0.2049 CHILE 0.2403 IRELAND 0.1205 CHILE 0.3206 TURKEY 0.4368 SWITZERLA 0.115 SPAIN 0.1277 DENMARK 0.3482 COLOMBIA 0.0984 ICELAND 0.2952 ESTONIA 0.3091 TURKEY 0.2092 COLOMBIA 0.3567 BRAZIL 0.2778 QATAR 0.3449 SINGAPOR 0.2937 ROMANIA 0.4081 PERU 0.304 POLAND -0.037 LIECHTENSTE 0.1302 HUNGARY 0.1216 BULGARIA 0.2005 ROMANIA 0.2304 UNITED STA 0.1168 QATAR 0.2898 SHANGHA 0.4367 COSTA RIC 0.1141 KAZAKHST 0.1268 UNITED KIN 0.3197 NETHERLA 0.0947 PERU 0.2691 SINGAPOR 0.291 BRAZIL 0.2052 BULGARIA 0.3534 MALAYSIA 0.2297 ARGENTIN 0.3289 UNITED KIN 0.2778 COSTA RIC 0.3733 VIETNAM 0.297 CANADA -0.0173 UNITED KING 0.1282 MEXICO 0.1084 MALAYSIA 0.1897 URUGUAY 0.2257 CZECH 0.1111 COSTA RIC 0.2895 DENMARK 0.386 GERMANY 0.1094 SCOTLAND 0.1157 CHILE 0.3094 ESTONIA 0.0901 UNITED KIN 0.2676 CHILE 0.2874 HUNGARY 0.1948 MEXICO 0.339 BULGARIA 0.2255 URUGUAY 0.3247 CANADA 0.2735 ARGENTIN 0.3731 MONTENE 0.2917 SCOTLAND -0.0151 HUNGARY 0.1215 NORWAY 0.0825 UNITED KIN 0.1843 RUSSIA 0.2039 MACAO-CH 0.1009 ISRAEL 0.2877 COSTA RIC 0.3143 MALAYSIA 0.1064 SLOVAKIA 0.1002 AUSTRALIA 0.2865 MEXICO 0.0822 THAILAND 0.2605 QATAR 0.2664 NORWAY 0.1831 QATAR 0.325 ICELAND 0.2089 UNITED KIN 0.3246 SCOTLAND 0.269 TURKEY 0.3179 PORTUGAL 0.2702 BELGIUM 0.0061 IRELAND 0.1139 SERBIA 0.0666 TURKEY 0.18 NEW ZEAL 0.2033 SCOTLAND 0.0952 NEW ZEAL 0.2746 HONGKON 0.2925 RUSSIA 0.1029 AUSTRIA 0.0835 TURKEY 0.2699 ARA BEMIR 0.0735 TURKEY 0.2572 INDONESIA 0.2654 PERU 0.171 PERU 0.302 UNITED ST 0.1977 MALAYSIA 0.3155 MONTENE 0.2194 CHILE 0.3106 SERBIA 0.2676 SLOVAKIA 0.0105 CROATIA 0.0902 PORTUGAL 0.0105 INDONESIA 0.1787 MEXICO 0.203 HUNGARY 0.0772 CANADA 0.2739 CHILE 0.2871 GREECE 0.098 CANADA 0.0812 COSTA RIC 0.2545 LATVIA 0.0655 SCOTLAND 0.1803 URUGUAY 0.2593 GERMANY 0.1686 TAIPEI 0.3011 CHILE 0.1961 SCOTLAND 0.3007 SERBIA 0.2172 UNITED ST 0.3096 ROMANIA 0.2456 PORTUGAL 0.0171 FRANCE 0.0855 INDONESIA 0.0104 PERU 0.1783 SHANGHA 0.1675 LATVIA 0.071 LITHUANIA 0.2675 RUSSIA 0.2836 LIECHTENS 0.0881 CROATIA 0.072 COLOMBIA 0.2399 SINGAPOR 0.0599 CHILE 0.1793 SHANGHA 0.2529 COLOMBIA 0.1632 CANADA 0.2829 TURKEY 0.1941 AUSTRALIA 0.2932 SHANGHA 0.2093 ICELAND 0.3017 CANADA 0.2402 NORWAY 0.0209 TURKEY 0.0853 ARA BEMIR -0.0029 CANADA 0.1707 LIECHTENS 0.1637 BELGIUM 0.0695 DENMARK 0.2638 URUGUAY 0.2669 SCOTLAND 0.0712 ICELAND 0.0542 UNITED ST 0.2399 URUGUAY 0.0474 CANADA 0.1698 THAILAND 0.2461 PORTUGAL 0.151 UNITED ST 0.28 URUGUAY 0.1824 CANADA 0.2708 CHILE 0.2072 VIETNAM 0.2981 LATVIA 0.2393 LATVIA 0.0278 UNITED STAT 0.083 JORDAN -0.0074 ICELAND 0.1678 CZECH 0.1334 THAILAND 0.0693 SCOTLAND 0.261 BULGARIA 0.2062 NETHERLA 0.0655 LITHUANIA 0.0462 BRAZIL 0.2299 CHILE 0.0463 PORTUGAL 0.1437 MALAYSIA 0.2208 CANADA 0.1326 URUGUAY 0.2683 MONTENE 0.1705 RUSSIA 0.2557 NEW ZEAL 0.1929 SERBIA 0.288 ISRAEL 0.2371 SHANGHAI 0.0339 NEW ZEALAN 0.0744 IRELAND -0.0231 LATVIA 0.166 IRELAND 0.1316 POLAND 0.0659 PORTUGAL 0.2475 GREECE 0.206 TUNISIA 0.0628 SWEDEN 0.0336 SCOTLAND 0.2151 SHANGHA 0.0321 DENMARK 0.1415 SERBIA 0.1974 SWEDEN 0.1203 PORTUGAL 0.2665 UNITED KIN 0.1563 SWEDEN 0.2287 TURKEY 0.1728 PORTUGAL 0.2731 ARGENTIN 0.2326 KAZAKHSTA 0.0412 AUSTRALIA 0.059 MONTENE -0.0256 TUNISIA 0.1641 AUSTRALIA 0.1299 MEXICO 0.0639 AUSTRALIA 0.2312 ISRAEL 0.1686 LUXEMBOU 0.055 ALBANIA 0.0162 LITHUANIA 0.1708 LUXEMBOU 0.0199 MONTENE 0.1327 CANADA 0.1956 INDONESIA 0.0814 SCOTLAND 0.2641 LATVIA 0.1403 VIETNAM 0.2268 AUSTRALIA 0.1726 DENMARK 0.2391 GREECE 0.2086 AUSTRALIA 0.0432 ARGENTINA 0.0568 DENMARK -0.0266 ESTONIA 0.152 INDONESIA 0.1266 SWITZERLA 0.0501 URUGUAY 0.1968 ARGENTIN 0.1613 AUSTRALIA 0.0504 FRANCE 0.004 SWEDEN 0.1681 AUSTRALIA 0.0135 AUSTRIA 0.1167 LATVIA 0.194 ESTONIA 0.0704 UNITED KIN 0.2622 SCOTLAND 0.1345 BULGARIA 0.2147 TUNISIA 0.161 NORWAY 0.2371 SINGAPOR 0.2026 NEW ZEALAN 0.0947 SWITZERLAN 0.0568 UNITED KIN -0.0286 SCOTLAND 0.1392 THAILAND 0.1245 COSTA RICA 0.0356 NORWAY 0.1924 ICELAND 0.153 MEXICO 0.0438 IRELAND 0.001 LIECHTENS 0.1458 ARGENTIN 0.0039 LIECHTENS 0.1036 MACAO-CH 0.1521 RUSSIA 0.0691 LATVIA 0.2605 AUSTRALIA 0.1329 UNITED ST 0.2056 ISRAEL 0.1566 URUGUAY 0.2316 TUNISIA 0.1985 SLOVENIA 0.1005 FINLAND 0.0548 PERU -0.0365 IRELAND 0.1376 LATVIA 0.0936 PORTUGAL 0.0264 SWEDEN 0.1827 MACAO-CH 0.1472 AUSTRIA 0.0416 AUSTRALIA -0.0103 NORWAY 0.1439 AUSTRIA 0.0024 SPAIN 0.1027 LITHUANIA 0.15 SLOVENIA 0.0686 ICELAND 0.244 RUSSIA 0.1228 FINLAND 0.185 SLOVAKIA 0.1341 MONTENE 0.2165 MALAYSIA 0.1937 RUSSIA 0.1051 MONTENEGR 0.05 SWEDEN -0.0384 JAPAN 0.1306 TUNISIA 0.0874 ROMANIA 0.0083 ARGENTIN 0.1356 UNITED KIN 0.1437 ROMANIA 0.0284 ESTONIA -0.0129 ROMANIA 0.1401 PORTUGAL -0.0019 BULGARIA 0.1026 BRAZIL 0.1477 SHANGHA 0.0675 ROMANIA 0.2438 ARGENTIN 0.118 TURKEY 0.1812 ITALY 0.1261 LATVIA 0.2067 LIECHTENS 0.1715 HONGKONG 0.1162 ICELAND 0.0442 BELGIUM -0.0393 SWITZERLA 0.1223 LITHUANIA 0.0865 MONTENEG 0.0011 IRELAND 0.1217 SWEDEN 0.1283 FINLAND 0.0278 NORWAY -0.0139 BULGARIA 0.1344 UNITED ST -0.0056 BELGIUM 0.0877 IRELAND 0.1323 LUXEMBOU 0.062 CHILE 0.2312 DENMARK 0.1172 NEW ZEAL 0.1619 LATVIA 0.0949 ISRAEL 0.2038 LITHUANIA 0.1544 GREECE 0.1243 CANADA 0.0292 NETHERLA -0.0423 THAILAND 0.0721 SWITZERLA 0.0822 TAIPEI -0.0018 UNITED ST 0.12 PORTUGAL 0.1054 ESTONIA 0.0219 TURKEY -0.0144 IRELAND 0.123 UNITED KIN -0.0078 IRELAND 0.0708 TUNISIA 0.1296 ICELAND 0.0595 ISRAEL 0.2144 NEW ZEAL 0.1059 DENMARK 0.1404 MEXICO 0.0785 LITHUANIA 0.187 COSTA RIC 0.1399 IRELAND 0.1266 LATVIA 0.023 SLOVENIA -0.0473 BELGIUM 0.0611 GREECE 0.0809 ARA BEMIR -0.005 LATVIA 0.1161 NEW ZEAL 0.1018 NEW ZEAL 0.0104 ARA BEMIR -0.0171 SHANGHA 0.1164 FRANCE -0.0114 ROMANIA 0.0665 HONGKON 0.119 AUSTRIA 0.0555 ARGENTIN 0.2028 SWEDEN 0.0841 TAIPEI 0.122 SWEDEN 0.0694 LIECHTENS 0.128 UNITED KIN 0.1181 CROATIA 0.1376 URUGUAY 0.0211 UNITED ST -0.0682 BRAZIL 0.0461 BRAZIL 0.0665 GERMANY -0.0223 LIECHTENS 0.0748 LITHUANIA 0.0958 ITALY 0.0078 JORDAN -0.0449 URUGUAY 0.1145 JAPAN -0.0139 URUGUAY 0.065 SPAIN 0.1055 DENMARK 0.054 AUSTRALIA 0.155 SERBIA 0.0835 ROMANIA 0.1163 NORWAY 0.0633 MACAO-CH 0.1204 URUGUAY 0.0989 ALBANIA 0.1418 ESTONIA 0.0181 THAILAND -0.0797 PORTUGAL 0.0291 POLAND 0.0636 CANADA -0.0229 TURKEY 0.0469 LIECHTENS 0.0957 URUGUAY -0.0151 UNITED KIN -0.0536 PORTUGAL 0.1135 INDONESIA -0.0207 MACAO-CH 0.0602 SCOTLAND 0.1051 LIECHTENS 0.0504 NEW ZEAL 0.1261 QATAR 0.0825 IRELAND 0.0727 CROATIA 0.0596 CANADA 0.0976 SLOVENIA 0.0982 SINGAPORE 0.1781 THAILAND 0.0093 CANADA -0.0832 AUSTRIA 0.0207 MONTENE 0.0472 LUXEMBOU -0.0232 FINLAND 0.0407 AUSTRALIA 0.0955 LITHUANIA -0.0218 ITALY -0.0731 RUSSIA 0.0378 CROATIA -0.0223 LUXEMBOU 0.059 CROATIA 0.0902 SPAIN 0.0444 HONGKON 0.1214 ISRAEL 0.0676 MONTENE 0.0626 LIECHTENS 0.059 SWITZERLA 0.0955 CZECH 0.097 SPAIN 0.179 ISRAEL 0.0066 SCOTLAND -0.0938 MACAO-CH 0.0013 SPAIN 0.0324 ICELAND -0.0255 GREECE 0.0222 UNITED ST 0.0795 BELGIUM -0.0344 BULGARIA -0.0898 MONTENE 0.0289 FINLAND -0.025 NORWAY 0.0556 UNITED KIN 0.0874 URUGUAY 0.0327 ITALY 0.1051 SWITZERLA 0.0532 LITHUANIA 0.0591 AUSTRIA 0.0583 SINGAPOR 0.0912 CROATIA 0.0534 MACAO-CHI 0.1884 MALAYSIA -0.0044 BULGARIA -0.094 URUGUAY -0.0027 GERMANY 0.0144 SPAIN -0.0267 SHANGHA 0.0218 SCOTLAND 0.0722 TURKEY -0.0458 LATVIA -0.0998 SWITZERLA -0.0023 ALBANIA -0.0416 ITALY 0.0413 SLOVENIA 0.0683 SINGAPOR -0.0011 GREECE 0.1022 LIECHTENS 0.0485 LATVIA 0.0355 ARGENTIN 0.0458 NEW ZEAL 0.0565 AUSTRALIA 0.0241 MONTENEGR 0.1921 ITALY -0.0071 FRANCE -0.1027 CZECH -0.0097 MALAYSIA -0.0027 COLOMBIA -0.0286 ESTONIA 0.0151 TAIPEI 0.0636 SHANGHA -0.0497 CHILE -0.0999 ARGENTIN -0.0104 BELGIUM -0.0483 ARGENTIN 0.0223 AUSTRALIA 0.0659 SWITZERLA -0.0167 MONTENE 0.0975 TAIPEI 0.0382 HONGKON 0.0346 SLOVENIA 0.0378 SLOVAKIA 0.0331 SCOTLAND 0.0208 SERBIA 0.1924 ROMANIA -0.0119 NEW ZEAL -0.143 SERBIA -0.0106 DENMARK -0.0146 PERU -0.0293 BULGARIA -0.0353 CANADA 0.059 IRELAND -0.0542 RUSSIA -0.1015 HONGKON -0.0203 BRAZIL -0.0706 LITHUANIA 0.0157 ARGENTIN 0.0573 IRELAND -0.0232 MACAO-CH 0.0888 IRELAND 0.0336 ISRAEL 0.0279 VIETNAM 0.007 AUSTRALIA 0.029 HUNGARY -0.0013 ARA BEMIRA 0.1943 ARA BEMIRA -0.016 GREECE -0.1502 CHILE -0.012 SERBIA -0.0166 SLOVENIA -0.0321 HUNGARY -0.044 IRELAND 0.0405 SLOVENIA -0.06 DENMARK -0.1075 FINLAND -0.0225 GREECE -0.0806 GERMANY -0.0006 ROMANIA 0.0391 SCOTLAND -0.0242 SERBIA 0.079 HONGKON 0.0269 SPAIN 0.0029 LITHUANIA 0.0066 FINLAND 0.0195 SWITZERLA -0.0066 VIETNAM 0.2146 SLOVENIA -0.0341 TUNISIA -0.1532 SLOVENIA -0.0123 FINLAND -0.0289 ITALY -0.0356 SPAIN -0.0444 MONTENE 0.0193 BRAZIL -0.0612 BELGIUM -0.1186 SPAIN -0.0586 LITHUANIA -0.0838 VIETNAM -0.002 ITALY 0.0386 CROATIA -0.0371 IRELAND 0.066 VIETNAM 0.0181 MACAO-CH -0.0268 HUNGARY -0.0008 TAIPEI 0.0054 SWEDEN -0.0394 BULGARIA 0.2381 SINGAPORE -0.0529 MALAYSIA -0.1559 AUSTRALIA -0.0158 TURKEY -0.0462 DENMARK -0.0455 SWITZERLA -0.0531 SWITZERLA 0.0106 THAILAND -0.0657 ROMANIA -0.1315 LATVIA -0.0913 CZECH -0.0862 HONGKON -0.0119 POLAND 0.0322 UNITED KIN -0.0451 ESTONIA 0.0514 TUNISIA -0.0187 NORWAY -0.047 COSTA RIC -0.0393 CZECH -0.024 MACAO-CH -0.047 TURKEY 0.2656 GERMANY -0.0587 SINGAPOR -0.1591 LUXEMBOU -0.027 SLOVENIA -0.0567 TURKEY -0.0776 SERBIA -0.0867 ESTONIA -0.0026 BULGARIA -0.0695 QATAR -0.1463 VIETNAM -0.0913 SCOTLAND -0.0918 SWEDEN -0.0122 HUNGARY -0.0019 FINLAND -0.0653 CROATIA -0.0186 HUNGARY -0.0188 SWITZERLA -0.0518 GREECE -0.0468 NETHERLA -0.0545 GERMANY -0.049 QATAR 0.2667 LUXEMBOUR -0.0712 JAPAN -0.1622 HONGKON -0.0331 ESTONIA -0.0605 NORWAY -0.0816 RUSSIA -0.0917 FRANCE -0.0137 LATVIA -0.072 NEW ZEAL -0.1574 GREECE -0.0934 GERMANY -0.0931 SWITZERLA -0.0235 NEW ZEAL -0.0077 LATVIA -0.0772 SPAIN -0.0231 MACAO-CH -0.0454 LIECHTENSTEIN-0.0645 POLAND -0.0521 GERMANY -0.0582 FRANCE -0.0531 ITALY 0.2729 BULGARIA -0.0953 BRAZIL -0.166 QATAR -0.0357 FRANCE -0.07 CROATIA -0.1264 GERMANY -0.1339 ITALY -0.0273 SPAIN -0.0734 GREECE -0.1581 GERMANY -0.1049 POLAND -0.0931 SERBIA -0.0498 GERMANY -0.0088 ITALY -0.0913 FINLAND -0.0235 FINLAND -0.0474 SERBIA -0.0665 SWITZERLAND-0.0542 ITALY -0.064 SLOVAKIA -0.0587 FRANCE 0.2802 DENMARK -0.1001 FINLAND -0.1708 GERMANY -0.0396 HUNGARY -0.0759 SLOVAKIA -0.1276 POLAND -0.144 LATVIA -0.0492 POLAND -0.0748 MEXICO -0.1789 FRANCE -0.1141 CANADA -0.1135 AUSTRALIA -0.055 FINLAND -0.0126 AUSTRALIA -0.0933 DENMARK -0.0399 LUXEMBOURG-0.0533 JAPAN -0.0829 DENMARK -0.0642 POLAND -0.0966 FINLAND -0.0608 INDONESIA 0.2807 BELGIUM -0.1106 AUSTRALIA -0.1817 CROATIA -0.0527 AUSTRIA -0.0852 SERBIA -0.1589 FRANCE -0.1588 GERMANY -0.113 PORTUGAL -0.0845 NETHERLANDS -0.1883 LUXEMBOURG-0.1547 NEW ZEALAND-0.1172 SLOVENIA -0.0568 AUSTRIA -0.0259 MEXICO -0.096 SWEDEN -0.0408 ESTONIA -0.0919 TUNISIA -0.0946 NETHERLANDS-0.0694 UNITED KINGDOM -0.0986 ICELAND -0.0658 KOREA 0.3065 JAPAN -0.1334 RUSSIA -0.1856 HUNGARY -0.0533 LUXEMBOURG-0.0961 URUGUAY -0.1605 CZECH -0.191 FINLAND -0.1291 ARGENTINA -0.0924 SERBIA -0.2152 ESTONIA -0.1842 TUNISIA -0.129 GREECE -0.061 DENMARK -0.0345 NETHERLANDS -0.0984 NORWAY -0.0996 SPAIN -0.098 ESTONIA -0.1254 ESTONIA -0.0825 JAPAN -0.1288 TAIPEI -0.068 TAIPEI 0.3185 SWEDEN -0.1355 TAIPEI -0.1966 ITALY -0.0593 ITALY -0.1105 AUSTRALIA -0.1724 ITALY -0.2002 NORWAY -0.1482 MONTENEGRO-0.0997 PERU -0.2158 KOREA -0.2058 ITALY -0.1321 FRANCE -0.0840 LUXEMBOURG-0.0621 ARGENTINA -0.1233 SWITZERLAND-0.1019 AUSTRIA -0.1151 GREECE -0.1689 IRELAND -0.0831 SCOTLAND -0.1367 NEW ZEALAND-0.0737 PERU 0.3348 GREECE -0.1734 ARGENTINA -0.1988 SLOVAKIA -0.0867 CROATIA -0.1491 NETHERLANDS-0.1782 SLOVENIA -0.2092 POLAND -0.1609 NORWAY -0.1009 THAILAND -0.2359 SLOVENIA -0.2110 IRELAND -0.1357 HUNGARY -0.0951 TAIPEI -0.0783 CZECH -0.1318 HUNGARY -0.1106 KOREA -0.1158 BELGIUM -0.1784 GERMANY -0.0922 SPAIN -0.1473 IRELAND -0.0907 COLOMBIA 0.3500 SERBIA -0.2197 ITALY -0.2015 SPAIN -0.0911 COSTA RICA -0.1662 BULGARIA -0.19 HONGKONG -0.2296 CZECH -0.1616 HUNGARY -0.1053 INDONESIA -0.2439 ITALY -0.2307 SPAIN -0.1393 RUSSIA -0.1105 CZECH -0.0846 LITHUANIA -0.1497 SLOVENIA -0.1142 BELGIUM -0.1278 SLOVAKIA -0.1828 TAIPEI -0.0985 ESTONIA -0.1505 AUSTRIA -0.0962 CHILE 0.3569 SHANGHAI -0.2267 LATVIA -0.2138 TAIPEI -0.104 TAIPEI -0.1712 MALAYSIA -0.1928 CROATIA -0.2534 LUXEMBOURG-0.162 CHILE -0.1101 MALAYSIA -0.2446 AUSTRIA -0.2428 SLOVENIA -0.1433 NEW ZEALAND-0.1162 KOREA -0.0895 FRANCE -0.1862 JAPAN -0.1224 NORWAY -0.1393 HUNGARY -0.1917 URUGUAY -0.0989 GREECE -0.1682 NETHERLANDS-0.1153 JAPAN 0.3612 CZECH -0.2319 QATAR -0.2184 NEW ZEALAND-0.1059 ICELAND -0.1719 CHILE -0.1937 MONTENEGRO-0.2623 SPAIN -0.1635 CZECH -0.1180 FINLAND -0.2489 TAIPEI -0.2513 RUSSIA -0.1434 NETHERLANDS -0.1279 GREECE -0.0952 NEW ZEALAND-0.194 SLOVAKIA -0.1238 GREECE -0.1508 ITALY -0.2236 SPAIN -0.1043 KOREA -0.1697 BELGIUM -0.1154 URUGUAY 0.3626 TAIPEI -0.2499 VIETNAM -0.2558 NORWAY -0.1079 BELGIUM -0.1951 SWEDEN -0.1951 MACAO-CHINA -0.2633 SERBIA -0.1684 SERBIA -0.1553 URUGUAY -0.2676 BELGIUM -0.2742 MACAO-CHINA-0.1733 ESTONIA -0.1486 LIECHTENSTEIN -0.0963 MALAYSIA -0.2014 POLAND -0.1545 CROATIA -0.1526 FRANCE -0.2402 ICELAND -0.1075 SHANGHAI -0.1961 LUXEMBOURG-0.1238 ROMANIA 0.3928 SLOVAKIA -0.2621 ROMANIA -0.2827 DENMARK -0.1189 SLOVAKIA -0.1954 NEW ZEALAND-0.2423 LUXEMBOURG-0.2753 HUNGARY -0.1817 HONGKONG -0.1620 VIETNAM -0.268 POLAND -0.2948 SLOVAKIA -0.1834 POLAND -0.1597 ICELAND -0.1054 HONGKONG -0.2485 FRANCE -0.1605 NETHERLANDS -0.1551 POLAND -0.2489 FRANCE -0.1115 SLOVENIA -0.2208 ESTONIA -0.1686 MALAYSIA 0.4263 NORWAY -0.2689 SLOVAKIA -0.3099 FRANCE -0.1255 NORWAY -0.212 GREECE -0.2564 TAIPEI -0.3393 SLOVAKIA -0.1951 SLOVAKIA -0.1681 MONTENEGRO-0.2725 SERBIA -0.3187 TAIPEI -0.1855 JAPAN -0.1773 NETHERLANDS -0.1374 BELGIUM -0.2736 CZECH -0.1900 JAPAN -0.1723 KOREA -0.2490 BELGIUM -0.1190 LUXEMBOURG-0.2450 ITALY -0.1777 COSTA RICA 0.4292 KOREA -0.2879 SHANGHAI -0.3132 SWEDEN -0.1404 HONGKONG -0.2135 JORDAN -0.2642 SLOVAKIA -0.3501 KOREA -0.2013 FRANCE -0.1699 TUNISIA -0.3138 HUNGARY -0.3202 KOREA -0.2080 CROATIA -0.1793 SWITZERLAND-0.1769 THAILAND -0.2784 BELGIUM -0.2145 FRANCE -0.1802 CZECH -0.2693 FINLAND -0.1488 AUSTRIA -0.2571 SPAIN -0.1882 MEXICO 0.4386 JORDAN -0.3242 KOREA -0.3209 POLAND -0.1596 SWEDEN -0.2143 FRANCE -0.2841 BELGIUM -0.3683 JAPAN -0.2367 MACAO-CHINA -0.1878 COSTA RICA -0.3315 MACAO-CHINA -0.3358 BULGARIA -0.2297 TAIPEI -0.2039 SWEDEN -0.2501 SLOVAKIA -0.3162 LIECHTENSTEIN -0.2448 ITALY -0.1930 GERMANY -0.3216 LUXEMBOURG-0.1537 BELGIUM -0.2718 NORWAY -0.2118 THAILAND 0.5019 NETHERLANDS -0.3705 POLAND -0.3232 FINLAND -0.1911 NETHERLANDS -0.2209 FINLAND -0.3044 NETHERLANDS -0.3712 SLOVENIA -0.238 VIETNAM -0.1919 ARGENTINA -0.3352 NETHERLANDS -0.3538 HUNGARY -0.3098 LATVIA -0.2188 BELGIUM -0.3143 TAIPEI -0.3334 KOREA -0.2525 SLOVAKIA -0.1939 CROATIA -0.3618 HONGKONG -0.1755 HONGKONG -0.3549 POLAND -0.2133 ARGENTINA 0.5168 QATAR -0.3918 ESTONIA -0.3267 ARGENTINA -0.1912 MACAO-CHINA-0.228 QATAR -0.3222 KOREA -0.3894 BELGIUM -0.2564 CROATIA -0.2570 KOREA -0.3571 SLOVAKIA -0.3798 HONGKONG -0.3161 FINLAND -0.2890 NORWAY -0.3454 MACAO-CHINA -0.3377 LUXEMBOURG-0.3645 CZECH -0.2071 SLOVENIA -0.3676 CZECH -0.1917 CROATIA -0.3775 DENMARK -0.2319 JORDAN 0.5206 HONGKONG -0.4169 CZECH -0.3597 GREECE -0.1927 VIETNAM -0.3261 BRAZIL -0.3322 AUSTRIA -0.3988 CROATIA -0.2635 KOREA -0.3835 JAPAN -0.4156 CROATIA -0.4541 ROMANIA -0.3371 CZECH -0.3841 FRANCE -0.4506 KOREA -0.3708 GERMANY -0.4057 SLOVENIA -0.2171 LUXEMBOURG -0.3889 MACAO-CHINA-0.3782 FRANCE -0.4167 JAPAN -0.2590 BRAZIL 0.5282 POLAND -0.4189 HONGKONG -0.3963 KOREA -0.2600 JAPAN -0.5042 TUNISIA -0.4225 JAPAN -0.4972 AUSTRIA -0.3243 TAIPEI -0.4571 COLOMBIA -0.4460 CZECH -0.5253 SERBIA -0.4239 SLOVAKIA -0.4276 SLOVAKIA -0.4891 VIETNAM -0.6131 NETHERLANDS -0.4181 GERMANY -0.2196 AUSTRIA -0.4349 JAPAN -0.6310 HUNGARY -0.4565 KOREA -0.3141 TUNISIA 0.6452 MACAO-CHINA -0.4339 MACAO-CHINA -0.4958 NETHERLANDS -0.3006 KOREA -0.7320 ARGENTINA -0.4921 ROMANIA -0.5637 NETHERLANDS -0.3402 JAPAN -0.5251 BRAZIL -0.4499 JAPAN -0.5848 MONTENEGRO-0.4495 KOREA -0.5443 JAPAN -0.5961 JAPAN -0.7306 AUSTRIA -0.482 POLAND -0.4152 NETHERLANDS -0.4456 KOREA -0.7692 IRELAND -0.5682 HONGKONG -0.3228

Non-cognitive Variables PISA 2012 1. Anxiety about Mathematics 2. Attitude towards School 3. Sense of Belonging at School 4. Instrumental Motivation 5. Interest in Mathematics 6. Self-concept in Math 7. Self-efficacy in Math 8. Subjective Norm 9. Mathematics Intentions 10. Math Work Ethics 11. Perseverance 12. Openness for Problem Solving 13. Cognitive Activation in Mathematics 14. Classroom Management 15. Mathematics Teacher's Support 16. Disciplinary Climate 17. Student-teacher relationship 18. Teacher support 19. Formative Assessment 20. Student Orientation 21. Teacher-directed Instruction

Non-cognitive Variables PISA 2012 1. Anxiety about Mathematics 2. Attitude towards School 3. Sense of Belonging at School 4. Instrumental Motivation 5. Interest in Mathematics 6. Self-concept in Math 7. Self-efficacy in Math 8. Subjective Norm 9. Mathematics Intentions 10. Math Work Ethics 11. Perseverance 12. Openness for Problem Solving 13. Cognitive Activation in Mathematics 14. Classroom Management 15. Mathematics Teacher's Support 16. Disciplinary Climate 17. Student-teacher relationship 18. Teacher support 19. Formative Assessment 20. Student Orientation 21. Teacher-directed Instruction

Classroom Management & Disciplinary Climate Mathematics Teacher's Classroom Management Thinking about the mathematics teacher who taught your last mathematics class: to what extent do you agree with the following statements? My teacher gets students to listen to him or her (+); My teacher keeps the class orderly (+); My teacher starts lessons on time (+); The teacher has to wait a long time for students to <quiet down> (-). 2012 Disciplinary climate in math lessons Students don t listen to what the teacher says (-); There is noise and disorder (-); The teacher has to wait a long time for students to quieten down (-); Students cannot work well (-); Students don t start working for a long time after the lesson begins (-). 2012, 2009, 2003

Sense of Belonging to School Sense of Belonging to School I feel like an outsider (or left out of things) at school (-); I make friends easily at school (+); I feel like I belong at school (+); I feel awkward or out of place in my school (-); Other students seem to like me (+); I feel lonely at school (-); I feel happy at school (+); Things are ideal in my school (+); I am satisfied with my school (+). 2012, 2003

Work Ethic & Openness for Problem Solving Mathematics Work Ethic Openness for Problem Solving I finish my homework in time for mathematics class (+); I work hard on my mathematics homework (+); I am prepared for my mathematics exams (+); I study hard for mathematics quizzes (+); I keep studying until I understand mathematics material (+); I pay attention in mathematics class (+); I listen in mathematics class (+); I avoid distractions when I am studying mathematics (+); I keep my mathematics work well organized (+). I can handle a lot of information (+); I am quick to understand things (+); I seek explanations of things (+); I can easily link facts together (+); I like to solve complex problems (+). 2012 2012

Part 2. Any substantial breakthrough in the assessment of non-cognitive constructs for the past 30 years?

Any substantial breakthrough? Recent research effort on self-report questionnaire, adjustment of rating-scale responses Issue of measurement unit in assessing noncognitive constructs An IRT application in assessing confidence

Strategies Tried in PISA 2012 Field Trial Based on 63 countries (N = 200 ~ 1000 per country) 1. Anchoring Vignettes (King at et al., 2004; Mottus et al., 2012) 2. Forced-choice (Berkshire & Highland, 1953; Landy, F. J., & Farr, 1980; Bartram, 2013) 3. Signal-detection correction (cf. Green & Swets, 1966; Swets, 2014) A list of mathematics concepts such as logarithm, rational number, divisor, mixed in with several foils (e.g., subjunctive scaling) 4. Behavioural items (me ) 5. Situational-judgment tests (e.g., Lievens & Sackett, 2006)

Levels of Measurement: Self-concept, Self-efficacy, Confidence Item Task Domain Confidence Self-efficacy Self-concept

Typical Measurement of Self-Concept 1. I am just not good at mathematics 2. I get good <grades> in mathematics 3. I learn mathematics quickly 4. I have always believed that mathematics is one of my best subjects 5. In my mathematics class, I understand even the most difficult work

Typical Measurement of Self-Efficacy 1. Using a <train timetable> to work out how long it would take to get from one place to another 2. Calculating how much cheaper a TV would be after a 30% discount 3. Calculating how many square metres of tiles you need to cover a floor 4. Understanding graphs presented in newspapers; 5. Solving an equation like 3x+5=17 6. Finding the actual distance between two places on a map with a 1:10_000 scale 7. Solving an equation like 2(x+3)=(x+3)(x-3) 8. Calculating the petrol consumption rate of a car

Typical Measurement of Confidence

Blue line: Confidence Red line: Accuracy Medium-level Difficulty Item Overconfidence for participants of low ability; Good calibration; Slight under-confidence for participants of high ability.

Blue line: Confidence Red line: Accuracy Difficult Item People are overconfident across all ability levels.

Blue line: Confidence Red line: Accuracy Easy Item Most people are under-confident when judging their performance on an easy item. Stankov, L., Lee, J., & Paek, I. (2009). Realism of confidence judgment. European Journal of Psychological Assessment, 25, 123-130.

So, has there been substantial breakthrough in the assessment of the noncognitive domain for the past 30 years?

Conclusion: Three Take-home Messages

Three Take-home Messages 1. What to measure? A group of Self-beliefs: Confidence (accurate selfassessment), Self-efficacy, Anxiety 2.How to measure? Self-report, close to the tasks, better at the item-level 3.How much overlap is there between cognitive and non-cognitive domains? Cognitive Noncognitive

Lee (2009) Lee, J. (2009). Universals and specifics of math self-concept, math self-efficacy, and math anxiety across 41 PISA 2003 participating countries. Learning and Individual Differences, 19, 355-365. Lee & Shute (2010) Lee, J., & Shute, V. (2010). Personal and social-contextual factors in K-12 academic performance: An integrative perspective on student learning. Educational Psychologist, 45, 185-202. Stankov, Lee, Luo, & Hogan (2012). Stankov, L., Lee, J., Luo, W., & Hogan, D. (2012). Confidence: A better predictor of academic achievement than self-efficacy, self-concept and anxiety? Learning and Individual Differences, 22, 747-758. Lee & Stankov (2013) Lee, J. & Stankov, L. (2013). A higher-order factorial structure among fifteen major educational-psychological constructs. Learning and Individual Differences, 26, 119-130. Lee (2014) Lee, J. (2014). Universal factors of student achievement in high-performing Eastern and Western countries. Journal of Educational Psychology, 106, 364-374. Stankov & Lee (2014) Stankov, L., & Lee, J. (2014). Quest for the best noncognitive predictor of academic achievement. Educational Psychology: An international journal of experimental educational psychology, 34, 1-8. Lee & Paek (2014) Lee, J., & Paek, I. (2014). In search of the optimal number of response categories using IRT graded response modeling. Journal of Psychoeducational Assessment, 32, 663-673.

Thank you!