CAAP Methodology

Standardized Test Methodology

Any benefits received from the use of a published nationally normed standardized test are linked directly to our ability to equate the concepts being measured to our rubrics. Because we cannot make curricular improvements from knowing how our students perform compared with a national sample, it is imperative that we understand exactly, the concepts being measured on a standardized test. With this information we can cross walk the skills to the capabilities included on our outcome rubric. In general, standardized tests provide content and prompts developed by external experts and are well supported by validity and reliability studies.

Instrument

Our instrument of choice is ACT’s Collegiate Assessment of Academic Proficiency (CAAP) for critical thinking. By fitting this test to our institutional outcomes, this method enjoys face and content validity internally. We use CAAP to provide evidence that our general education objectives are being met and to document change in students' performance levels from one educational point to another. The test is a 32-item, 40-minute test that measures students' skills in clarifying, analyzing (17-21 items), evaluating (5-9 items), and extending arguments (6 items). An argument is defined as a sequence of statements that includes a claim that one of the statements, the conclusion, follows from the other statements. The test consists of four passages that are representative of the kinds of issues commonly encountered in a postsecondary curriculum. Passages typically presents a series of sub-arguments in support of a more general conclusion and are presented in a variety of formats, such as debates, case studies, or experimental results. Each passage is accompanied by a set of multiple-choice test items.

Process

Once the materials for the CAAP test are received in the Office of Institutional Research, they are held until the two-week testing period begins. Courses are selected in which the test will be dispensed to all students. Instructors from the courses administer the test during class time within the two-week testing period. Members from the Scholarship Action Group provide logistical support. Completed tests are returned to the Office of Institutional Research from which they are sent to ACT for scoring. (Contact the Office of Institutional Research for a more detailed description of administration requirements.)

Sampling

The sampling method is clustered random. A random sample of courses is drawn from the courses that support the critical thinking outcome. All the students that are present on the day of test administration in those courses take the test. The results are then generalizable to all students taking courses supporting the critical thinking outcome. Typically, the number of students taking the test is large enough to be able to generalize to all learner group one students.

Analysis

We use CAAP test results in two ways to suggest areas in our curriculum that require improvement. First, the test content areas can be linked to the capabilities on our critical thinking rubric. Thus, student performance within the content areas can be analyzed in relation to the capabilities in the language internally developed. This is done by mean of item analysis. Second, we can analyze the performance of beginning students and compare that to the performance of students with more earned credit hours or exposures to the critical thinking outcome. In this way, we can assess the effectiveness of our curriculum and whether it has made a difference in the abilities and skills of our students.

Construct validity is guaranteed by ruling out alternative explanations for the measured results. Test score serves as the dependent variable for which we are determining the sources of variance. The two key independent variables are number of NMC credit hours a student has earned and number of exposures a student has had to the general education outcome being measured. Number of exposures is determined by evaluating the courses a student has completed that have been identified by instructors as supporting a specific general education outcome. A student’s placement test scores serve as a control variable to rule out prior skill as an explanation of artifact score. The data are then analyzed by regression analysis.

Results

The results of the regression analysis show the degree to which learning at NMC or exposure to the critical thinking outcome has contributed to students’ achievement on the critical thinking test content areas. With a representative sample of “freshman” students and a representative sample of “senior” students we can document the performance gain in students’ learning over time. Students receive their scores on the test, and those performing at or above the national average receive a certificate of achievement from the college. The results of the item and regression analysis are provided in a report from the Office of Institutional Research.