IRT Modeling Lab

Procedure for DIF Detection Using Lord's Chi-Square (2PL and 3PL Models)


As a prelude to conducting an IRT DIF analysis based on the 2PL and 3PL models, one must verify that the data are essentially unidimensional. Once the unidimensionality of each scale has been verified, a computer program such as BILOG (Mislevy & Bock, 1991) can be used to estimate the a, b, and c item parameters. Then, the item parameters of the focal and reference groups must be placed on a common metric, using an iterative linking procedure such as described by Stocking and Lord's (1983).

Stocking and Lord (1983) developed a method of "linking" the theta metrics for different subpopulations of examinees. The rationale for this procedure is that item parameters can be considered invariant only after the theta values are expressed on a common metric. However, a paradox exists in IRT DIF analysis; namely, one must use only unbiased items to link the theta metrics; yet, these unbiased items are precisely those that one must identify with a DIF analysis (Candell & Drasgow, 1988). To circumvent this situation, the linking of the metrics must be done iteratively. Linking constants are generated from initial estimates of theta and the scales are equated. Item parameters are then compared using a DIF index to determine which items display DIF. Items displaying DIF are removed and new linking constants are generated and the scales are reequated. DIF items are removed again and theta is again linked. This process continues until the same DIF items are identified on consecutive trials.

Lord's chi-square statistic is given by

formula


where vi is a vector of the differences in the estimated item parameters for the ith item between the focal and reference groups, and Sigmai is the variance-covariance matrix for the differences in item parameter estimates (Lord, 1980). Observed values can be compared to the critical value corresponding to p = .05 with 3 degrees of freedom under the null hypothesis of no DIF. When numerous comparisons are made, a Bonferroni correction may be applied to obtain a more conservative rejection criterion. It is customary to use the critical chi-square corresponding to p = .05 divided by the number of items in the scale.

Conducting Lord's Chi-Square DIF Analysis

The basic steps are as follows:

  1. Estimate item parameters and their covariances for focal and reference groups separately using BILOG.

  2. Obtain linking constants for putting the focal and reference parameters on a common metric (These are also used to transform the covariance matrices, as shown in Lord [1980]). Typically, we transform the focal group parameters to the reference group's metric, using a modified version of Frank Baker's equating computer program, called EQUATE99. This program is capable of linking metrics not only for the 2PL and 3PL models, but also for the SGR and BNM models.

  3. Compute Lord's chi-square to identify DIF items using the reference and transformed focal group parameters and their covariances.

  4. Once the DIF items have been identified, reequate the focal and reference group metrics using only the non-DIF items.

  5. Repeat steps 2 through 4 until the same items are identified on consecutive trials. Note, in some cases, the removal of one item may cause another item to exhibit DIF, resulting in a lack of convergence on items identified on consecutive trials. If this occurs, remove one or both items and obtain new linking constants.

Note: For the 2PL and 3PL models, the iterative linking procedure described above can be performed using Stark's ITERLINK computer program. ITERLINK identifies DIF items and puts the focal group parameters on the reference group metric, using Lord's chi-square statistics corrected for the number of items in a scale. To learn how to run the ITERLINK program, click here.

Running ITERLINK

  1. Create a folder for storing the following files required for Lord's chi-square DIF analysis. Put the ITERLINK.EXE and PARTO3PL programs into that folder, along with the BILOG output files containing item parameters and covariances for the reference and focal groups. The item parameter files must be named using the extension *.par; the covariance files should have the extension *.cov.

  2. Double click on the PARTO3PL executable file. It will find all the *.par files in that folder, delete the first four lines and subsequent alternating lines, and rename the files using the *.3PL extension. (In our example, focal and reference parameter files are named 3plfoc.3pl and 3plref.3pl, respectively. Similarly, the covariance files are 3plfoc.cov and 3plref.cov).

  3. Double click on the ITERLINK.EXE file to begin the equating process. You must answer the following questions:

    1. Type a TITLE for this analysis ->
      Title must be less than 80 characters.

    2. Enter the number of ITEM PARAMETERS: (2)2PL/2PN; (3)3PL/3PN ->
      Type 2 for 2PL or 3 for 3PL. (Ignore the 2PN and 3PN)

    3. Enter file name for "FROM" (FOCAL) METRIC ITEM PARAMETERS ->
      Enter the name of the item parameter file for the focal group (e.g., 3PLFOC.3PL)

    4. Enter FORMAT of "FROM" METRIC file or (0) to accept (32x,2f12.6,12x,f12.6) ->
      If your item parameter files were modified using PARTO3PL, then enter 0 to accept the default format. The columns corresponding to a, b, and c parameters will be referenced using the f12.6 format descriptors. If your item parameter files are in a different format, you must provide the FORTRAN format statement using standard descriptors (consult a FORTRAN text).

    5. Enter NUMBER of ITEMS in "FROM" TEST (500 max) ->
      Enter the number of items for the focal group.

    6. Enter file name for "TO" (REFERENCE) METRIC ITEM PARAMETERS ->
      Enter the name of the item parameter file for the reference group (e.g., 3PLREF.3PL)

    7. Enter FORMAT of "TO" METRIC file or (0) to accept (32x,2f12.6,12x,f12.6) ->
      If your item parameter files were modified using PARTO3PL, then enter 0 to accept the default format. The columns corresponding to a, b, and c parameters will be referenced using the f12.6 format descriptors. If your item parameter files are in a different format, you must provide the FORTRAN format statement using standard descriptors (consult a FORTRAN text).

    8. Enter NUMBER of ITEMS in "TO" TEST (500 max) ->
      Enter the number of items for the reference group.

    9. Enter FILENAME for TRANSFORMED "FROM" ITEM PARAMETERS ->
      Enter a filename to save the focal group parameters after they have been placed on the metric of the reference group.

    10. TRANSFORM "FROM" THETAS? YES(Y) or NO(N) ->
      In this application, transformed thetas are not needed, so enter N for no.

    11. Enter name of *UNEDITED* BILOG COVARIANCE file for FOCAL group ->
      Enter a filename, such as 3PLFOC.COV. This file must be used exactly as created by BILOG. After responding to this question, ITERLINK will execute.

    12. Enter name of *UNEDITED* BILOG COVARIANCE file for REFERENCE group ->
      Enter a filename, such as 3PLREF.COV. This file must be used exactly as created by BILOG.

    13. Enter CRITICAL P-VALUE or (1) to accept Bonferroni ->
      As explained by the program, a critical p-value must be specified for flagging DIF items. To maintain a family Type I error rate of .05, the default Bonferroni correction should be used. However, you have an option specifying a more liberal or conservative criterion.
      To accept the Bonferroni corrected p-value, enter 1.
Interpreting ITERLINK Results

Using the specifications described above, ITERLINK will create three output files: ITERLINK.DBG
PAIRDIF.DBG
foctran.out


ITERLINK.DBG

At the beginning of this file, you will find a summary of the input specifications. Check this information to be sure that your choices were entered correctly. Next, you will find a summary of the calculations performed on each iteration.

For each iteration, find the header lines beginning with "ROUTINE "PAIRDIF3" CONDUCTS A PAIRWISE (2-GROUP) DIF ANALYSIS for 3-PL model," as shown below.

Note the values of the linking constants used to transform the focal group item parameters and covariances. Note also the Bonferroni corrected p-value obtained by dividing .05 by the number of items.

Output

Next examine the Lord's chi-square results using the corrected p-values. Items identified as having DIF are indicated by a "YES"; items having no DIF are marked "NO". DIF items are excluded from the linking automatically on the next iteration.

Output

Next examine the Lord's chi-square results using the corrected p-values. Items identified as having DIF are indicated by a "YES"; items having no DIF are marked "NO." DIF items are excluded from the linking automatically on the next iteration.

Output

Next you will see the corresponding results for iteration 2. However, additional information for iterative linking follows the Lord's chi-square statistics. Three columns of values are presented. A "0" in the second column, referred to as LINK1(I), indicates that item I had DIF on the previous iteration and was excluded from the most recent calculation of the linking constants. Similarly, a "0" in the first column, referred to as LINK2(I), indicates that item I had DIF on the current iteration. A value of "0" in the third column, ABSDIFF, indicates that the item had the same outcome (DIF or no DIF) on consecutive trials. Iterative linking stops when all items have the ABSDIFF = 0.

Output

Skipping to the last iteration (near the end of the ITERLINK.DBG file), you will find the information for the final linking constants (in the format shown for iteration 1), followed by a table of the final DIF results. These results are shown below. Note, ITERLINK indicates that the same items were identified on consecutive iterations.

Output

PAIRDIF.DBG

This file contains detailed information about the DIF calculations. At the beginning of the file, the names of the covariance files are listed. Next, the details of the computations for the Lord's chi-square statistics are presented for each item, based on the linking constants obtained on the final iteration. A summary of the results for item 1 is shown below (Note that all of the information is clearly labeled.). Check to see if item parameters for the reference (Ar, Br, and Cr) and focal (Af, Bf, and Cf) group, as well as the covariance matrices were read correctly.
Output



foctran.out

This file contains the transformed focal group item parameters in free-field format. (The a parameters are in column 1, b's in column 2, and c's in column 3.)

Back