Simulating a CAT using catR

This document is a practical exercise illustrating a Computerized Adaptive Testing (CAT) simulation. CAT is grounded in Item Response Theory (IRT) and requires prior estimation of item parameters. Its purpose is to deliver short, tailored questionnaires by selecting items from a pool in real time based on each respondent’s estimated ability. For example, if a participant answers a difficult item correctly, the algorithm skips easier items and instead presents a harder one to narrow down the ability estimate more efficiently. As a result, participants do not need to respond to every item in the pool. This simulation uses an item pool of 200 items with a maximum of 40 items administered per respondent. The catR package handles the simulation, while foreach and doSNOW are used for parallel computation.

Code 
library(catR)
library(foreach)
library(doSNOW)
library(ggplot2)
library(patchwork)

Simulate item pool

This simulation uses the 2PL (two-parameter logistic) model, which assigns two parameters to each item: a discrimination parameter (a), reflecting how well the item distinguishes between ability levels, and a difficulty parameter (b), reflecting how hard the item is. The genDichoMatrix function from catR generates item parameters for dichotomous (binary) items — items scored as either correct (1) or incorrect (0), such as multiple-choice questions.

The 200-item bank is generated with the following priors:

  • Parameter a follows a normal distribution with mean 2 and SD 0.1, ensuring all discrimination values are positive.
  • Parameter b follows a normal distribution with mean 0 and SD 4, producing items that span a wide range of difficulty — from very easy (large negative b) to very hard (large positive b).

A high a value means the item sharply differentiates between respondents near its difficulty level; a high b value means only respondents with high ability are likely to answer correctly.

genDichoMatrix also includes c (pseudo-guessing) and d (inattention) parameters. Since their values default to 0 and 1 respectively, they have no practical effect and the model reduces to a standard 2PL formulation.

Code 
itemBank<-genDichoMatrix(items=200,model="2PL",aPrior=c("norm",2,.1),bPrior=c("norm",0,4))
head(itemBank)
         a          b c d
1 2.040940 -2.5058152 0 1
2 2.168887  0.7345733 0 1
3 2.158659 -3.3425144 0 1
4 1.966909  6.3811232 0 1
5 1.771476  1.3180311 0 1
6 2.249766 -3.2818735 0 1
Code 
summary(itemBank)
       a               b                 c           d    
 Min.   :1.711   Min.   :-8.8588   Min.   :0   Min.   :1  
 1st Qu.:1.950   1st Qu.:-2.4553   1st Qu.:0   1st Qu.:1  
 Median :2.000   Median :-0.1975   Median :0   Median :1  
 Mean   :2.004   Mean   : 0.1422   Mean   :0   Mean   :1  
 3rd Qu.:2.073   3rd Qu.: 2.4520   3rd Qu.:0   3rd Qu.:1  
 Max.   :2.265   Max.   : 9.6065   Max.   :0   Max.   :1

Simulate item responses

catR provides the genPattern function, which generates simulated item responses for a given set of ability values and an item bank. Here, 1,000 respondents with abilities uniformly spaced between −10 and +10 are simulated. Descriptive statistics are shown for the first 9 items to verify the generated response patterns.

Code 
data<-genPattern(th=seq(-10,10,length=1000),it=itemBank,model=NULL,seed=1)
summary(data[,1:9])
       V1              V2              V3              V4              V5              V6              V7              V8              V9       
 Min.   :0.000   Min.   :0.000   Min.   :0.000   Min.   :0.000   Min.   :0.000   Min.   :0.000   Min.   :0.000   Min.   :0.000   Min.   :0.000  
 1st Qu.:0.000   1st Qu.:0.000   1st Qu.:0.000   1st Qu.:0.000   1st Qu.:0.000   1st Qu.:0.000   1st Qu.:0.000   1st Qu.:0.000   1st Qu.:0.000  
 Median :1.000   Median :0.000   Median :1.000   Median :0.000   Median :0.000   Median :1.000   Median :0.000   Median :0.000   Median :0.000  
 Mean   :0.629   Mean   :0.462   Mean   :0.674   Mean   :0.182   Mean   :0.434   Mean   :0.663   Mean   :0.405   Mean   :0.361   Mean   :0.379  
 3rd Qu.:1.000   3rd Qu.:1.000   3rd Qu.:1.000   3rd Qu.:0.000   3rd Qu.:1.000   3rd Qu.:1.000   3rd Qu.:1.000   3rd Qu.:1.000   3rd Qu.:1.000  
 Max.   :1.000   Max.   :1.000   Max.   :1.000   Max.   :1.000   Max.   :1.000   Max.   :1.000   Max.   :1.000   Max.   :1.000   Max.   :1.000

Estimate full-information scores with mirt

Treating the simulated response matrix as real participant data (one row per respondent), the mirt package is used to estimate item parameters and respondent ability scores under the assumption that all items were administered. These full-information scores serve as a benchmark against the CAT estimates produced in the next step.

Code 
mirt_model<-mirt::mirt(data=data.frame(data),model=1,itemtype="2PL",technical=list(theta_lim=c(-6,6),NCYCLES=10000))

Simulate CAT administration and scoring

Each respondent is administered a CAT drawn from the 200-item pool, with a maximum of 40 items. The simulation is configured through four control lists:

  • start — defines first-item selection. The initial item is chosen to have a difficulty closest to θ = 0 (startSelect = "bOpt"), a neutral starting point before any ability information is available.
  • test — controls item selection and scoring during the adaptive phase. Ability is estimated using Bayesian Modal (BM) estimation (Birnbaum, 1969). EAP (the default of mirt::fscores) is a common choice; here BM, catR’s default, produced scores that correlate highly with the mirt EAP benchmark (see Plot 1). BM uses catR’s default standard normal prior N(0, 1), which pulls extreme estimates toward 0; this shows up in the conditional bias reported by simulateRespondents at the end of the page. Items are selected by bOpt (the item whose difficulty b is closest to the current ability estimate), and randomesque = 5 adds slight randomisation by drawing the next item at random from the 5 items with b closest to the current estimate. The estimation range is widened from the default [−4, 4] to [−20, 20] so that estimates are not cut off at ±4.
  • final — specifies the scoring method applied at test termination (BM, same range).
  • stop — defines termination criteria. The test ends when either 40 items have been administered or the standard error of the provisional ability estimate drops to 0.3 or below — whichever occurs first.
Code 
start<-list(theta=0,nrItems=1,startSelect="bOpt")
test<-list(method="BM",itemSelect="bOpt",randomesque=5,range=c(-20,20))
final<-list(method="BM",range=c(-20,20))
stop<-list(rule=c("length","precision"),thr=c(40,0.3))
cl<-makeCluster(parallel::detectCores())
registerDoSNOW(cl)
result<-foreach(participant=1:nrow(data),.combine=rbind,.packages=c("catR")) %dopar% {
  rc_result<-randomCAT(itemBank=itemBank,
                       model=NULL,
                       responses=data[participant,],
                       start=start,
                       test=test,
                       stop=stop,
                       final=final)
  data.frame(questions=length(rc_result$thetaProv),
             est_theta=rc_result$thFinal,
             se=rc_result$seFinal)
  
}
stopCluster(cl)
result<-data.frame(result,mirt_scores=as.numeric(mirt::fscores(mirt_model)))

Results

The results are summarised graphically across six panels:

  • Plot 1 — CAT vs. mirt scores: Scatter plot of CAT ability estimates against mirt EAP scores. CAT respondents answered at most 40 items, while mirt scores are based on all 200 responses. High agreement between the two validates the adaptive procedure.
  • Plot 2 — Items administered vs. CAT theta: Respondents with extreme abilities need more items, because the CAT starts at θ = 0 and the first items, with b near 0, carry little information about them. The estimate has to work its way outward, slowed by the N(0, 1) prior, and the pool also has fewer items in the tails (b ~ N(0, 4)).
  • Plot 3 — Items administered vs. SE: For tests that ended before 40 items the SE sits just below the 0.3 target (between 0.28 and 0.3), because these tests stopped once precision was reached. SE above 0.3 appears only at exactly 40 items: these 279 respondents hit the length cap before reaching the target, which happens for extreme abilities where the pool has few items with matching difficulty.
  • Plot 4 — CAT theta vs. SE: Standard error stays near 0.3 over most of the scale and increases only at the extremes, where the 40-item cap is reached before enough information is collected. Recall that SE is approximately the inverse square root of the test information (for BM, the information plus the prior’s contribution).
  • Plot 5 — Distribution of CAT estimated scores.
  • Plot 6 — Distribution of mirt estimated scores.
Code 
p1 <- ggplot(result, aes(x = est_theta, y = mirt_scores)) +
  geom_point(alpha = 0.5) +
  geom_smooth(method = "lm", se = FALSE, linewidth = 0.7) +
  labs(x     = "CAT Theta",
       y     = "mirt Theta",
       title = paste("Pearson r", round(cor(result$est_theta, result$mirt_scores), 4))
  ) +
  theme_bw(base_size=20)

p2 <- ggplot(result, aes(x = questions, y = est_theta)) +
  geom_point(alpha = 0.5) +
  labs(x = "Questions asked", y = "CAT Theta") +
  theme_bw(base_size=20)

p3 <- ggplot(result, aes(x = questions, y = se)) +
  geom_point(alpha = 0.5) +
  labs(x = "Questions asked", y = "SE") +
  theme_bw(base_size=20)

p4 <- ggplot(result, aes(x = est_theta, y = se)) +
  geom_point(alpha = 0.5) +
  labs(x = "CAT Theta", y = "SE") +
  theme_bw(base_size=20)

p5 <- ggplot(result, aes(x = round(est_theta, 0))) +
  geom_bar() +
  labs(x = "CAT Theta", y = "Frequency") +
  theme_bw(base_size=20)

p6 <- ggplot(result, aes(x = round(mirt_scores, 0))) +
  geom_bar() +
  labs(x = "mirt Theta", y = "Frequency") +
  theme_bw(base_size=20)

(p1 | p2) / (p3 | p4) / (p5 | p6) +
  plot_annotation(theme = theme(plot.background = element_rect(fill = "white")))

Conclusion

The CAT estimates are roughly uniform between about −6 and 6, as expected because the true abilities were evenly spaced, with pile-ups at the ends of their range (-7.3 to 7.9) where the pool runs out and the N(0, 1) prior compresses the most extreme abilities (true θ runs from −10 to 10). The mirt scores are mound-shaped and on a narrower scale because mirt standardises the latent distribution to N(0, 1) when it calibrates the items. In real applications abilities are usually roughly normal, and item banks are built so that information is highest where most respondents are.

catR also provides simulateRespondents, a convenience function that runs a complete CAT simulation and computes summary diagnostics — bias, RMSE, and mean SE — as a function of true ability.

Code 
result_simulate_respondents<-simulateRespondents(thetas=seq(-6,6,.01),itemBank=itemBank,genSeed=NULL,start=start,test=test,stop=stop,final=final)
Simulation process:  0 %
Simulation process:  10 %
Simulation process:  20 %
Simulation process:  30 %
Simulation process:  40 %
Simulation process:  50 %
Simulation process:  60 %
Simulation process:  70 %
Simulation process:  80 %
Simulation process:  90 %
Simulation process:  100 %
Code 
result_simulate_respondents

 ** Simulation of multiple examinees ** 
 
Random seed was not fixed 
 
Simulation time: 10.1146 seconds 
 
Number of simulees: 1201 
Item bank size: 200 items 
IRT model: Two-Parameter Logistic model 
 
Item selection criterion: bOpt 
 Stopping rules: 
    Stopping criterion 1: length of test
       Maximum test length: 40 
    Stopping criterion 2: precision of ability estimate
       Maximum SE value: 0.3 
 rmax: 1 

Mean test length: 18.62614 items 
Correlation(true thetas,estimated thetas): 0.9949 
RMSE: 0.4852 
Bias: -0.0118 
Maximum exposure rate: 1 
Number of item(s) with maximum exposure rate: 1 
Minimum exposure rate: 0 
Number of item(s) with minimum exposure rate: 13 
Item overlap rate: 0.1642 
 
Conditional results 
                        Measure     D1     D2     D3     D4     D5     D6     D7     D8     D9    D10
                     Mean Theta -5.405 -4.205     -3   -1.8 -0.605    0.6  1.805      3  4.195    5.4
                           RMSE  0.635  0.546  0.457  0.344    0.3  0.294  0.344  0.439  0.547  0.732
                      Mean bias  0.531  0.456  0.333  0.148  0.062 -0.096 -0.176 -0.302 -0.444 -0.628
               Mean test length 28.442 21.875 16.818 13.605 11.725 12.091 14.075 17.294 22.175 28.099
            Mean standard error  0.295  0.294  0.293  0.294  0.294  0.294  0.293  0.294  0.293  0.295
 Proportion stop rule satisfied      1      1      1      1      1      1      1      1      1      1
             Number of simulees    120    120    121    119    120    121    120    119    120    121

These results can be saved by setting 'save.output' to TRUE 
  in the 'simulateRespondents' function
Code 
invisible(capture.output(plot(result_simulate_respondents)))

Overall, the simulated CATs used 18.6 items on average, with a correlation of 0.995 between true and estimated abilities and an RMSE of 0.485. The conditional bias shows the pull of the N(0, 1) prior toward 0: it is positive at the low end (0.53 at θ ≈ -5.4) and negative at the high end (-0.63 at θ ≈ 5.4).