Um Enem mais curto, preciso e seguro: desenvolvimento de uma aplicação em formato de Testagem Adaptativa Computadorizada

Abstract

The Brazilian High School Exam (Enem) consists of a written essay and four tests with 45multiple-choice items: Human Sciences and their Technologies (HS); Natural Sciences and their Technologies (NS); Languages, Codes and their Technologies (LC); and Mathematics and their Technologies (MT). The exam is used as a selection process for entry into higher education courses. This use poses challenges for the exam in its format, such as: producing precise scores for a diverse population, minimizing the effect of item position on performance, and constructing equivalent tests. It is possible to advance in these challenges by applying Enem in a Computerized Adaptive Testing (CAT) format. Therefore, the objective of this work was to develop a more efficient, precise, and secure Enem CAT than its current format. We divided the thesis into two Articles and two Products. Article 1 compared sample distributions in the calibration of items in the three-parameter logistic model of Item Response Theory. We used information from the four Enem 2020 tests to simulate the responses of 5,040 participants drawn from three types of sampling designs: random, rectangular, and shifted. There was no significant difference between the designs for the discrimination parameter. The shifted sample recovered the difficulty parameters in HS better than the rectangular sample and in NS better than the random sample. The shifted and random samples recovered the pseudo-guessing parameters better than the rectangular sample in all four tests. The results do not point to the prevalence of one type of sample to calibrate the Enem 2020 items. Product 1 consisted of a statistical package for simulating CAT in an R environment. Article 2 evaluated the method of progressive restricted exposure (PR) control with different acceleration parameters in a CAT in terms of efficiency, precision, and security. We manipulated the item selection method (Random, Maximum Fisher Information - MFI - and PR with two acceleration parameters) and the stopping criterion (Fixed length of 20 and 45 items, Standard Error of 0.30 and Error Reduction of 0.015 with Standard Error of 0.30) and simulated the application of 16 CAT conditions for each test. Finally, we simulated the application of Enem 2020 in a linear format and compared it to the 20-item fixed-length CAT. The test size was larger with MFI. In the fixed-length CATs and with the stopping criterion of standard error, MFI and PR (with both acceleration parameters) had similar results for precision. With the error reduction criterion, PR performed worse. Security increased as the acceleration parameter increased. The adaptive version of Enem had higher precision than the linear version. Product 2 of the thesis was the publication of an Enem CAT application with the algorithm determined in Article 2. We concluded that it is possible to reduce the size of Enem and improve its precision and security with a CAT.