Ya-Hui Su, UC Berkeley

DIF: Past work and current studies

Poster Session (2:10 to 3:00)

Karen Draney, Jinnie Choi, & Yong-Sang Lee, University of California, Berkeley.

Yong-Sang Lee, Xiaoting Huang, University of California, Berkeley.

Xiaoting Huang, University of California, Berkeley.

Kavita Seeratan, Wenyan Zhou, Tzur Karelitz, Kristen Burmester, Min-Joung Kim, Mary Candace Full, Robert Schwartz, Mark Wilson, University of California, Berkeley.

Sun-Joo Cho, University of California, Berkeley; Allan S. Cohen and Jonathan Templin, University of Georgia.

Presentations (3:00 to 4:00)

Ya-Hui Su, University of California, Berkeley. Wen-Chung Wang, National Chung Cheng University, Taiwan.

Xiaohui Zheng, Sophia Rabe-Hesketh, University of California, Berkeley.

AERA/NCME Preview Day

Xiahui Sheng, UC Berkeley;

Claus Carstensen & Anne C. Frenzel, IPN, Kiel University, Germany;

Leah Walker, UC Berkeley

Latent Growth Modeling

Derek Briggs, University of Colorado at Boulder

Note change of day & time:

Thursday 3-5pm

Using Explanatory Item Response Models to Analyze Group Differences in Science Achievement

Cheryl Schwab, UC Berkeley

What Can We Learn from PISA? Investigating PISA’s Approach to Scientific Literacy

Frank Rijmen, ETS

Note this is an extra meeting—Frank will be here giving a series of workshops

Kathy Long, LHS; Cathleen Kennedy, Linda Woodward & Rachael Tan, BEAR Center, UC Berkeley

Recovering Science's Meaning for Life in Educational Research

William P. Fisher, Jr. , AVATAR INTERNATIONAL, INC.

The critique of science and modernity offered under the various headings of existentialism, phenomenology, hermeneutics, feminism, deconstruction, etc. has stalled in recent years, with no major new advances. Increasingly, entrenched camps each guard fixed opinions based in unexamined assumptions. Close reading of key texts, however, opens doors to the resolution of some fundamental dilemmas. For instance, what are the implications of Gadamer's characterization of method as the activity of the thing itself? How might educational research make use of Heidegger's description of the ontological method's three moments of phenomenological reduction, practical application, and deconstructive return to a new reduction? How might the mathematical metaphysics of all "Academic" thinking influence qualitative research as much as-or even more than-it does quantitative research? These questions are pursued with the goal of arriving at sound recommendations for enhancing the impact of educational research and practice.

top

DIF: Past work and current studies

Ya-Hui Su, UC Berkeley

In this presentation, an important issue of item response theory (IRT) that I have been working will be reviewed: differential item functioning (DIF). Here is a short review of my past work and current studies. It has been established that as long as a test has many DIF items, then any DIF detection methods will lose control over Type I error rate and power, even with scale purification procedures (Fidalgo, Mellenbergh, & Muñiz, 2000; Miller & Oshima, 1992). In a series of our papers (Su & Wang, 2005; Wang & Su, 2004a; Wang & Su, 2004b), I show that it is the average signed area (ASA) between the item characteristic curves of the reference and focal groups that determines the performances of DIF detection methods (such as the Mantel-Haenszel, the generalized Mantel-Haenszel, and the logistic regression methods), rather than the percentage of DIF items. For example, these methods perform fairly well even there are 50% DIF items, as long as the ASA approaches zero. I also show that DIF detection methods with scale purification procedures are helpful to reduce DIF contamination in the matching variable so as to yield a better control over Type I error rate and power than those methods without such procedures. In another study of DIF detection in 2PLM data (Su & Glas, 2006), I use true and estimated a-parameters to weight item scores and use the weighted total score as the matching variable. I found that this new method is superior to standard methods that use un-weighted total score as the matching variable, especially when tests contain many DIF items. Scale purification procedures cannot always eliminate DIF contamination completely so as to guarantee appropriate results, especially when there are many DIF items and all of them favor the same group. In another study (Su, Shih, & Wang, 2006), I show how to use DIF detection methods to locate a set of DIF-free items to serve as a pure anchor, such that DIF detection is based on a pure matching variable and also show that the results are appropriate.

Download References

top

AERA/NCME Preview Day

Poster Session (2:10 to 3:00)

Karen Draney, Jinnie Choi, & Yong-Sang Lee, University of California, Berkeley.

Empirical Validation of a Learning Progression (Using the BEAR Assessment System).

Yong-Sang Lee, Xiaoting Huang, University of California, Berkeley.

A Simulation Study of Collinearity and Centering Effects in HLM.

Xiaoting Huang, University of California, Berkeley

Language Skills and Components Measured in the Cambridge Young Learner's English Test.

Kavita Seeratan, Wenyan Zhou, Tzur Karelitz, Kristen Burmester, Min-Joung Kim, Mary Candace Full, Robert Schwartz, Mark Wilson, University of California, Berkeley.

Making Sense of Student Responses to Assessment Items Using Scoring Exemplars.

Sun-Joo Cho, University of California, Berkeley; Allan S. Cohen and Jonathan Templin, University of Georgia.

A Multidimensional Mixture IRT model for DIF Analysis

Presentations (3:00 to 4:00)

Ya-Hui Su, University of California, Berkeley. Wen-Chung Wang, National Chung Cheng University, Taiwan.

Collusion Effects on the Conditional Sympson and Hetter Online Procedure with Freeze Control.

Xiaohui Zheng, Sophia Rabe-Hesketh, University of California, Berkeley.

Constructing Between-School Differential Item Functioning.

Guemin Lee, Yonsi University; Do-Soon Park, Korea University; Myungho Nam and Myunghwa Kim Korea Institute of Curriculum & Evaluation; Min-Jeong Jeon, University of California, Berkeley; Keon-Seob Kim, Yonsei University. 

Cheating Proof Testing System (CPTS) and Its Validity.

top

Latent Growth Modeling

Presentation 1. Xiahui Zheng, UC Berkeley:
Latent Growth Models, Revisited

Presentation 2. Claus Carstensen & Anne C. Frenzel, IPN, Kiel University, Germany:
Modeling latent growth curves from item responses to analyse the development of mathematics enjoyment between grades 5 and 9

Presentation 3. Leah Walker, UC Berkeley:
Consequences of Longitudinal Measurement Invariance for the Latent Growth Item Response Model

Download Power Point presentation

top

Apr. 3

Change of day and time:

Thursday 3-5 pm

Using Explanatory Item Response Models to Analyze Group Differences in Science Achievement

Derek Briggs, University of Colorado, Boulder

This paper illustrates the use of an explanatory item response modeling (EIRM) approach in the context of measuring group differences in science achievement. The distinction between item response models and EIRMs, recently elaborated by De Boeck & Wilson (2004), is presented within the statistical framework of generalized linear mixed models. It is shown that the EIRM approach provides a powerful framework for both a psychometric and statistical analysis of group differences. This is contrasted with the more typical two-step approach, in which psychometric analysis (i.e., measurement) and statistical analysis (i.e., explanation) occur independently. The two approaches are each used to describe and explain racial/ethnic gaps on a standardized science test. It is shown that the EIRM approach results in estimated racial/ethnic achievement gaps that are larger than those found in the two-step approach. In addition, when science achievement is examined by subdomains, the magnitude of racial/ethnic gap estimates under the EIRM approach are more variable and sensitive to the inclusion of contextual variables. These differences stem from the fact that the EIRM approach allows for disattenuated estimates of group level parameters, while the two-step approach depends upon estimates of science achievement that are shrunken as a function of measurement error.

Download paper
Download Power Point presentation

top

What Can We Learn from PISA?
Investigating PISA’s Approach to Scientific Literacy

Cheryl Schwab, UC Berkeley

The Programme for International Student Assessment (PISA), developed under the auspices of the Organization for Economic Cooperation and Development (OECD), offers a unique opportunity to evaluate the assessment of scientific literacy.  In this presentation, I discuss the relationship of PISA’s complex definition of scientific literacy to other definitions of scientific literacy, and the results of investigating the complexity of student responses to the PISA 2003 science items. 
           
PISA’s definition of scientific literacy is composed of three competencies (i.e., process, content, and situation), each defined as a continuum of performance.  I argue that PISA’s definition of scientific literacy incorporates three themes drawn from history: (a) scientific way of thinking, (b) everyday relevance of science, and (c) scientific literacy for all students. 
          
Two studies were conducted using item response modeling to investigate the nature of student responses to the PISA 2003 science items.  The first study used multidimensional models of the internal structure of the PISA 2003 science items to reflect the complex character of PISA’s definition of scientific literacy.  The second study used different scoring approaches and item response models to illustrate the different ways to interpret student responses to the science assessment items.  These analyses highlighted the variation of students’ responses to the PISA science items and the use of the ordered partition model to accommodate different but equal item responses.  The results of the two studies are presented and used to discuss ways to improve the development and interpretation of PISA’s science items.

Download Power Point presentation

top

An Efficient EM Algorithm Using Local Computations:
Full Information Maximum Likelihood Estimation in Limited Time

Frank Rijmen, ETS

Latent variable models are being widely used in the behavioural and social sciences. For categorical data however, the use of models with a more complex (multidimensional) latent structure has been hampered
because maximum likelihood estimation involves an integration over the latent variables, and the number of computations involved in brute force integration is exponential in the dimensionality of the latent space. However, brute force integration can often be avoided by exploiting the conditional independence relations between the latent variables that are implied by the model. In this respect, more efficient maximum likelihood estimation algorithms have been described for the multilevel latent class model (Vermunt, 2003), the item bifactor model (Gibbons & Hedeker, 1992), and most notoriously, the hidden Markov model (Rabiner, 1989).

In this presentation, a general method is described for the construction of efficient estimation algorithms that exploit the conditional independence relations between the latent variables. Specifically, the statistical model is associated with a graph in which nodes correspond to random variables and the (absence of) edges represent conditional (in)dependence relations between the variables. The core of the method consists of applying transformations to this initial graph. The structure of the transformed graph provides a factorization of the joint probability function of the manifest and latent variables, which is the basis of a modified and more efficient E-step of the EM algorithm. For the models mentioned above, the algorithm is equivalent to the estimation algorithms proposed by the respective authors. The method is applicable to a large family of models, and proceeds in a fully algorithmic way. Hence, it overcomes the computational burden that has (too) long been clouded the sky for latent variable models for categorical data.

Download Power Point presentation

top

Developing Formative Assessments for Science Curriculum Using the BEAR Assessment System

Kathy Long, LHS; Cathleen Kennedy, Linda Woodward & Rachael Tan, BEAR Center, UC Berkeley

The Assessing Science Knowledge (ASK) project is a collaboration among researchers and curriculum developers at the Lawrence Hall of Science, the BEAR Center, and SRI International; teachers, district facilitators, Delta Education, the National Science Foundation, and others.  It was funded in 2003 with two major goals:

1. Develop classroom assessment strategies and tools that lead to better instructional practices and improved student achievement.
2. Develop assessments with the technical quality needed to provide accountability information to districts.

This seminar will discuss the approaches used to reach these goals, including what worked well and what did not, and where we are today.  One of the sixteen FOSS (Full Option Science System) hands-on modules studied, Magnetism and Electricity, will be used as an example to demonstrate the changes in the embedded materials and benchmark assessments, as well as how we analyzed the quality of the assessment system.  ClassMap, a program for use by classroom teachers, developed by the BEAR Center with input from FOSS, will also be introduced.

top