To hold the problem to a reasonable size, we have set the sample size at 100. This is on the small side in an IRT application (a sample size in excess of 200 is usually considered desirable), but with the default priors assumed for the item parameters, the results of the calibration are reasonable.

Because the sample size is not large enough to attempt estimating the population latent distribution, we have accepted the default normal distribution with mean zero and standard deviation one.

Twenty items is the smallest number for which the chi-square test of item fit can be recommended. Due to the small sample size, the test is not very sensitive, and, as might be expected, none of the items shows any indication of poor fit.

For scoring the examinees, we have chosen the most robust method, EAP (Bays) estimation, on the assumption that the population distribution is normal. We follow the usual testing convention of standardizing the scores in the sample (with mean 0 and standard deviation 1, in this case). Because the scale of measurement for a psychological or educational test almost always has an arbitrary origin and unit, no generality is lost by rescaling in the calibration sample. In Example 2, these EAP score estimates are compared with MI (maximum likelihood) estimates similarly scaled.

To illustrate (in Example 2) how response data can be scored from the file of item-parameter estimates from an earlier BILOG calibration, we make use of the SAVE option in the GLOBAL command, followed by the SAVE command, to save the item parameter file as EXAMPL01.PAR. The item parameters are saved after being adjusted to account fr the rescaling of the scores in the sample. If they are used to score other data, the resulting scores will be expressed in the scale metric of the original sample. They can however, be saved after rescaling in other data, in which case they will be readjusted to the scale of that sample.

THREE - PARAMETER MODEL RANDOM DATA EAP SCORE ESTIMATION

>COMMENTS

This example illustrates the use of the GLOBAL "OMITS" option with the 3-PL model. Omitted responses are scored fractionally correct in an amount equal to the reciprocal of the number of multiple-choice alternatives (NALT = 5)

The data for this example have been randomly generated to fit the 3-PL model, but some of the responses with probability of being correct less than 0.3 have been randomly omitted. So that the true estimated scores can be compared, the generating values of the respondents' scores are used as the case ID's.

The data are in the file EXAMPL01.DAT of the BLGDAT directory; the answer key (KFNAME) and the omit key (OFNAME) are, respectively, the first and second records of the data file.

The respondents' scores are estimated by the EAP method (default) and rescaled to mean 0 and standard deviation 1 in the sample (RSC=3). The item parameter estimates are saved AFTER rescaling.

>GLOBAL DFNAME='BLGDAT\EXAMPL01.DAT', NPARM=3, OMITS,SAVE;

>SAVE PARM='EXAMPL01.PAR';

>LENGTH NITEMS=20;

>INPUT NTOT=20, NALT=5, NIDC=10, KFNAME='BLDDAT\EXAMPL01.DAT',

OFNAME='BLGDAT\EXAMPL01.DAT';

>TEST TNAME=RANDOM;

>CALIB FLOAT;

>SCORE RSC=3;

SAMPLE PROBLEMS 6-2

Phase 1 output: EXAMPL01.PH1

********** PHASE 1************

 EXAMPLE 01: TEST WITH OMITTED RESPONSES

THREE-PARAMETER MODEL RANDOM DATA EAP SCORE ESTIMATION

>COMMENTS

[Same as above]

>GLOBAL DFNAME='BLGDAT\EXAMPL01.DAT' , NPARM=3, OMITS,save;

 OMITS WILL BE REPLACED BY THE RECIPROCAL OF THE NUMBER OF RESPONSE ALTERNATIVES

• SAVE PARM='EXAMPL01.PAR';
• LENGTH NITEMS = 20;
• SUBTEST LENGTHS: 20
• INPUT NTOT=20, NALT=5, NIDC=10K, KFNAME='BLGDAT\EXAMPL01.DAT',

OFNAME='BLGDAT\EXAMPL01.DAT';

FORMAT CARD FOR INPUT IS (4X,10A1,T17,20A1)

[INPUT FILES]

>TEST TNAME=RANDOM;

ANSWER KEY:

1 RANDOM RRRRRRRRRRRRRRRRRRRR

OMIT KEY:

1 RANDOM 88888888888888888888

OBSERVATION 1 WEIGHT: 1.0000 ID: -0.324

SUBTEST 1 RANDOM

TRIED RIGHT

 [1] The first scratch files are normally deleted at the end of the problem run, but the first three can be saved by assigning them non-default names in the SAVE command. If the data set is large and is to be analyzed more than once, the master binary file, EXAMPL01.MFL, contains all the data and is worth saying. The calibration binary file contains those cases used for item parameter estimation; it may be equal to our smaller than the master file. In route applications, there is very little justification for requesting a calibration sample larger than the default size of 1000; the improvement in precision of item parameter estimation in larger sample sizes is relatively small.

  The EXAMPL01.VFL file contains the estimated sampling variances are convariances of the item parameter estimators. While the square roots of the variances appear as standard errors in the output, the covariances between parameters (obtained from the inverse of the estimated information matrix) are available only in this file. Covariances between items are not saved, but those between the two or three parameters of the separate items appear in this file. (See Chapter 5.)

[2] The presented/not-presented codes (TRIED: 1.0 presented, -1.0 not-presented) and item scores of the first two cases (RIGHT: 1.0 if correct, 0 if incorrect, -1 if omitted) are output as check on the data format and scoring keys.

[3] The classical items statistics provide the starting values for the iterative estimation of the item parameters (see Chapter 1, pp. 3-5, for the relationships between item statistics and item parameters). If the number of items and sample size is large and the classical normal assumptions apply, these starting values are sufficiently accurate that a few EM cycles yield estimates of the item parameters that are quite satisfactory for most practical purposes. For preliminary analyses under these conditions there is little reason to spend the additonal computing time required to obtain fully converged MML or MMAP estimates.

Phase 2 output: EXAMPL01.PH@

*********** PHASE 2 ***********

EXAMPLE 01: TEST WITH OMITTED RESPONSES THREE-PARAMETER MODEL RANDOM DATA EAP SCORE ESTIMATON

>CALIB FLOAT;

*******************************************

CALIBRATIONO F SUBTEST RANDOM

*******************************************

METHOD OF SOLUTION

EM CYCLES (MAXIMUM OF 10)

FOLLOWED BY NEWTON-RAPHSON STEPS (MAXIMUM OF 2)

QUADRATURE POINTS AND PRIOR WEIGHTS:

 PRIOR DISTRIBUTIONS ON ITEM PARAMETERS

(THRESHOLDS, NORMAL; SLOPES, LOG-NORMAL; GUESSING, BETA)

[18 similar lines omitted]

[EM STEP]

-- 2 LOG LIKELIHOOD = 2260.2174

SAMPLE PROBLEMS 6-6

CYCLE 1: LARGEST CHANGE = 2.3714

-- 2 LOG LIKELIHOOD = 2258.8731

UPDATED PRIOR ON ASYMPTOTES; ALPHA & BETA = 5.29157 16.70843

UPDATED PRIOR ON LOG SLOPES; MEAN & SD = -.24391 .50000

CYCLE 2: LARGEST CHANGE = .07377

-- 2 LOG LIKELIHOOD = 2258.4835

UPDATED PRIOR ON LOG ASYMPTOTES; ALPHA & BETA =  5.3610 16.63990

UPDATED PRIOR ON LOG SLOPES; MEAN & SD = -.26662 .50000

CYCLE 3: LARGEST CHANGE = .02392

-- 2 LOG LIKELIHOOD = 2258.3934

UPDATED PRIOR ON ASYMPTOTES; ALPHA & BETA = 5.40416 16.59584

UPDATED PRIOR ON LOG SLOPES; MEAN & SD = -.27645 .50000

CYCLE 4: LARGEST CHANGE = .01786

-- 2 LOG LIKELIHOOD = 2258.3526

UPDATED PRIOR ON ASYMPTOTES; ALPHA & Beta = 5.43811 16.56189

UPDATED PRIOR ON LOG SLOPES; MEAN & SD = -.28489 .50000

 CYCLE 5: LARGEST CHANGE = .00809

[NEWTON STEP]

UPDATED PRIOR ON ASYMPTOTES; ALPHA & BETA = 5.46420 16.53580

UPDATED PRIOR ON LOG SLOPES; MEAN & SD = -.28872 .50000

--2 LOG LIKELIHOOD = 2258.3003

CYCLE 6: LARGEST CHANGE = .00809

SUBTEST RANDOM: ITEM PARAMETERS AFTER CYCLE 6

LARGEST CHANGE = .008

QUADRATURE POINTS AND POSTERIOR WEIGHTS:

[4] The quadrature points and weights are respectively, deviates and normalized probability densities of the assumed prior distribution of ability. The default is a normal prior, but the user can supply other values. (The item-parameter estimation is not very sensitive to moderate departures from a normal prior.)

[5] See the summary for the PRIORS command, p. 4:31, for the relationship between the ALPHA and BETA parameters of the beta distribution and the mean and variance of the asymptote-parameter distribution.

[6] When priors are assumed on the item parameters, the likelihood may not increase during the EM cycles. This occurs because the starting values are closer to point of maximum marginal likelihood than the MAP estimates. But if the number of items and the sample size is large enough to justify the use of the FLOAT option, the means of the priors will be estimated simultaneously with the item parameters and the marginal likelihood will generally increase. The resulting estimates are then intermediate between the MML and the MMAP estimates. The compromise is necessary because there are seldom enough items to estimate the variances and other moments of their prior distributions accurately. Note that during the EM cycles the means of the priors change but not the standard deviations.

[7] If the number of items is 20 or greater, approximate chi-square statistics for the goodness-of-fit of each item are calculated and output. For this purpose, the cases in the calibration sample are sorted into successive intervals of the latent continuum according to the EAP estimates of their ability rescaled to mean 0 and standard deviation 1. Then, neighboring intervals are collapsed until the expected proportion of correct or incorrect responses exceeds 0.05. This gives a reasonable test of fit fi the number of items is large enough to make the assignment of the cases accurate, and if the sample size is large enough to retain three or more intervals. In the present example, the sample size is too small for a useful test; for more suitable data, see Example 8. Note that because the MML estimation procedure the residual frequencies (OBSERVED - EXPEVCTED) are not under linear constraints due to estimation of item parameters, the degrees of freedom are equal to the number of intervals retained.

When the number of items is less than 20, these chi-square statistics are replaced by root-mean-square standardized posterior residuals at the quadrature points. (See Examples 2, 4, 5 and 6)

[8] At the end of the calibration, normalized probability densities of the latent distribution are estimated at the quadrature points by the a posteriori weights shown in the output. In the present example, their agreement with the prior normal weights is very good, but this is to be expeced because the simulated data were generated from a normal distribution.

EXAMPLE 01: TEST WITH OMITTED RESPONSES

THREE PARAMETER MODEL RANDOM DATA EAP SCORE ESTIMATION

CORRELATIONS AMONG SUBTEST SCORE ESTIMATES:

RANDOM = 1.000

MEANS AND STANDARD DEVIATIONS OF SCORE ESTIMATE:

TEST: Random

MEAN: .015

S.D.: .868

MARGINAL LATENT DISTRIBUTION FOR SUBTEST RANDOM

MEAN = .015 S.D. = .968

[11]

RESCALING DONE WITH REPECT TO SUBJECT DISTRIBUTION [12]

SCALING LOCATION

SUBTEST CONSTANT

RANDOM 1.153 -.017

[13]

[14]

SUBJECT IDENTIFICATION

[OUTPUT FOR 80 CASES OMITTED HERE]