The Mathematics Placement Test
Since 2007, the Department of Mathematics and Statistics has used the ACCUPLACER mathematics tests to place students into lower division mathematics and statistics courses through the first semester of calculus and introductory statistics courses. All state funded institutions of higher education in Utah use the same ACCUPLACER tests and there is considerable coordination among t he institutions. Much of the quantitative assessment that has been carried out in the Department of Mathematics and Statistics over the last few years has revolved around the mathematics placement test (MPT).
The MPT was introduced at Utah State University and the other institutions of higher education in response to low student achievement in pre-calculus mathematics courses. In some instances students would unsuccessfully repeat the same course 3 or more times. In addition, faculty teaching courses at the first-semester-of-calculus and introductory statistics level s perceived that substantial numbers of students in their courses were ill-prepared mathematically and struggled from the very beginning of the courses.
Courses Subject to Placement Test The courses that the MPT places students into are:
- Math 0900 (Elements of Algebra)
- Math 1010 (Intermediate Algebra) Prerequisite: Math 0900
- Math 1030 (Quantitative Reasoning) Prerequisite: Math 1010
- Math 1050 (College Algebra) Prerequisite: Math 1010
- Math 1060 (Trigonometry) Prerequisite: Math 1010
- Math 1100 (Calculus Techniques) Prerequisite: Math 1050
- Math 1210 (Calculus I) Prerequisites: Math 1050 and 1060
- Math 2020 (Introduction to Logic and Geometry) Prerequisite: Math 1050
- Stat 1040 (Introduction to Statistics) Prerequisite: Math 1010
- Stat 2000 (Statistical Methods) Prerequisite: Math 1050
- Stat 2300 (Business Statistics) Prerequisite: Math 1050
Students Subject to the Placement Test and the Math Prerequisite Acceptability Time Limit All students wishing to register for one of the classes listed above that have not successfully completed the prerequisite course or obtained a high enough score (see the general catalogue for details) on the ACT, SAT, or AP tests within the last calendar year are required to take the MPT. This includes:
- Students at the Logan, Utah campus of USU.
- Students at regional campus and distance educations sites throughout the state of Utah.
- Students in Utah high schools taking Math 1050 and 1060 by concurrent enrollment though Utah State University.
- Students in other USU programs, including the business programs in China (that require Math 1050, Math 1100, and Stat 2300).
The one-year for which prerequisites are valid is the Math Prerequisite Acceptability Time Limit (MPATL). Many of these students are incoming students that have not taken a mathematics class and an assessment such as ACT, SAT, or AP in their senior high school year.
The Elementary Algebra Test and the College Level Mathematics Test There are two ACCUPLACER tests that are used by the Department of placement. The Elementary Algebra (EA) test may be used to place students in Math 0900 or Math 1010 and is generally only taken by a subset of the students that wish to take Math 1010.
The College Level Mathematics (CLM) test may be used to place students in all of the courses for which the MPT may be required. The vast majority of students that take the MPT take the CLM test, and all the analyses that follow are for the CLM.
Taking the Mathematics Placement Test Multiple Times. At this time there is no restriction on the number of times students may attempt the MPT, but students may only take it once per day.
Refresher Course The department offers a refresher course for the content of Math 0900, Math 1010, and Math 1050 & 1060 in the week before classes in fall and spring semesters. The fee for the refresher course includes the fees for two attempts at the MPT, one at the beginning of the refresher course and one at the enbd of the refresher course.
Questions Addressed by Analyses of Mathematics Placement Test Data
- Does the refresher course improve student performance on the MPT?
- Do students that take the refresher course earn higher or lower grades, on average, than students that did not do the refresher course?
- Do students that qualify for admission to a course by taking the MPT obtain, on average, higher or lower grades than students that qualify for admission to the course by some other mechanism, such as passing a prerequisite course?
- Is there any relationship between students’ scores on the MPT and their final grades?
- What is the overall impact of the MPT on D-F-W rates of courses that the MPT is used to place into?
- What are good predictors of a student’s score on the MPT?
1. Does the refresher course improve student performance on the MPT?
Analysis for this question began by creating a scatter plot of pre-test verse post-test scores for individuals participating in the refresher course. A scatter plot was created for each semester that the refresher course has been offered (fall 2007, fall 2008, spring 2009). Then all data for anyone who has taken the refresher course was compiled into one data set and a scatter plot was produced. From observing the graphs below, there is clearly an increasing linear relationship between the pretest and post test for each semester and for the combined data.
A model was created regressing the pre-test score on the post-test score. To determine if inferences could be made on this model, the usual model assumptions were checked. A residual plot was produced (see below). This plot does not suggest any problems with constant error variance.
A four-plot summary of the residuals was produced to observe if model assumptions were met. The symmetry plot appears to indicate non-symmetry and the box plot suggests there may be outliers. The histogram suggests possible non-normality. Therefore, the model assumptions were not met.
In order to make inferences, the data were transformed. After taking the fourth root of CLM2, model assumptions were met. With this transformation the data are approximately symmetric and normal. The residual plot indicates possible non-constant error variance. However, the potential outliers somewhat create this pattern and must remain in the data because they contain critical information about the relationship between pre and post-test scores.
Since model assumptions were met, a bar-chart was formed for further analysis. This bar-chart summarizes the effect of the refresher course in terms of the number of students placed into specific courses by their placement exam score.
To compare the mean pre-test and post-test scores, a paired t-test was conducted. However, when observing the differences between the pre and post-test scores, model assumptions were not met (see four-plot summary below).
A square root transformation on the differences improved the data, allowing model assumptions to be met (see four-plot summary below).
There was a significant increase of 20.2 points on the Accuplacer college-level mathematics placement exam after taking the refresher course. This difference between pre and post test scores was significant (P<.0001) after transforming the differences to meet model assumptions.
Conclusion: The refresher course is very effective in preparing students for the mathematics placement test.
- Nearly all students that take the refresher course score higher on the MPT after the course than before it.
- The mean difference is about 20 points, which is the difference between admission to say, Math 1010 and Math 1050, or Math 1050 and Math 1100. Thus, on average, after taking the refresher course, students are qualifying for admission to a course one level higher than they were before taking the refresher course.
Analyses carried out by: Cortnie Broadus and Nicole Rupp under the supervision of Dr. John Stevens. 20 February 2009.
2. Do students that take the refresher course earn higher or lower grades, on average, than students that did not do the refresher course?
This question involves comparing the average grades of two groups: The students that placed into a course as a result of their placement exam score after they took the refresher course, and the students that placed into the course without taking the refresher course.
First, the observations for students who received an Incomplete or a Withdraw were dropped because the numbers were so few that they would not have an effect. Then the data for Math 0900 was dropped because students do not receive college credit for this course. Finally, if a student opted to pass/fail the course these observations were also disregarded due to the fact that a “pass” would be equal to a zero on the grading scale which would be equivalent to an F. In addition, each letter grade was assigned a numeric value, A = 4.00, A- = 3.67, . . . D+ = 1.33, D = 1.00, an F = 0.
In order to compare the two groups, model assumptions had to be met so that conclusions could be made based on the analysis. The following box-plot was created in order to check the normality of the two groups. Although the plots are not perfect, the assumption of normality is met.
Since the assumption of normality was met, further analysis on this data could be completed. A two-sample t-test was performed to compare the final grades of the two groups. The results of these tests can be seen in the table below. In Math 1100, the difference in the grades between the two groups was statistically significant (P=.04) with the students who took the refresher course getting higher grades. All of the other classes did not give statistically significant p-values. Overall, the difference between the group that took the refresher course and the groupthat did not take the refresher course was statistically significant (P=.01) with the final grades for the refresher course group being higher.
| No Refresher Course | Refresher Course | |||||||
| Course | Average Grade | Average Letter | Sample Size | Average Grade | Average Letter | Sample Size | P-value (t-test) | |
| Math 0900 | NA | NA | NA | NA | NA | NA | NA | |
| Math 1010 | 2.22 | C | 1,765 | 2.40 | C+ | 48 | 0.30 | |
| Math 1030 | 3.22 | B | 27 | 2.50 | C+ | 2 | 0.22 | |
| Math 1050 | 2.48 | C+ | 789 | 2.70 | B- | 32 | 0.30 | |
| Math 1060 | 2.72 | B- | 267 | 2.81 | B- | 16 | 0.77 | |
| Math 1100 | 2.18 | C | 241 | 2.91 | B- | 15 | 0.04 | |
| Math 1210 | 2.71 | B- | 234 | 2.75 | B- | 16 | 0.91 | |
| Math 2020 | NA | NA | NA | NA | NA | NA | NA | |
| Stat 1040 | 2.73 | B- | 410 | 2.83 | B- | 6 | 0.83 | |
| Stat 2000 | 2.75 | B- | 15 | 3.66 | B+ | 2 | 0.34 | |
| Stat 2300 | 2.67 | B- | 198 | 3.03 | B | 9 | 0.33 | |
| Overall | 2.42 | C+ | 3,946 | 2.68 | B- | 146 | 0.01 | |
Conclusion: Only in Math 1100 is there evidence that students that took the refresher course perform better, on average, in the course than students that did not take the refresher course. Summing overall courses for which the MPT is used for placement there is a modest effect of about ¼ of a gradepoint in favor of students that took the refresher course.
Analyses carried out by: Cortnie Broadus and Nicole Rupp under the supervision of Dr. John Stevens. 23 March 2009.
3.Do students that qualify for admission to a course by taking the MPT obtain, on average, higher or lower grades than students that qualify for admission to the course by some other mechanism, such as passing a prerequisite course?
This question entails comparing the average grades of two groups: students that placed into a course as a result of their MPT score,and students that placed into a course by some other mechanism, such as successfully completing a prerequisite course in the previous year.First, the observations for students who received an Incomplete or a Withdraw were dropped because the numbers were so few that any analysis on them would be invalid. Then the data for Math 0900 was dropped because students do not receive college credit for this course. Finally, if a student opted to pass/fail the course these observations were also disregarded due to the fact that a “pass” would be equal to a zero on the grading scale which would be equivalent to an F. In addition, each letter grade was assigned a numeric value, A = 4.00, A- = 3.67, . . . D+ = 1.33, D = 1.00, an F = 0.
In order to make valid observations based off comparison of the two groups, model assumptions had to be met. The following box-plots were created in order to check the normality of the two groups. By viewing the graphs below, it is clear that the assumption of normality is met.
Since the assumption of normality was met, further analysis on this data could be completed. A two-sample t-test was performed to compare the final grades of the two groups. The results of these tests can be seen in the table below. Math 1050, 1060, and 1210 as well as Stat 1040 resulted in statistically significant p-values (P = <.0001, <.0001, .03, and <.0001 respectively). This implies those who took the placement exam earned higher grades than those who did not take the placement exam. However, overall there was no evidence of higher grades between the two groups.
| No Placement Exam | Placement Exam | ||||||
| Course | Average GPA | Average Letter | Sample Size | Average GPA | Average Letter | Sample Size | P-value |
| Math 0900 | NA | NA | NA | NA | NA | NA | NA |
| Math 1010 | 2.19 | C | 5,346 | 2.24 | C | 1,562 | 0.13 |
| Math 1030 | 2.74 | B- | 194 | 2.67 | B- | 11 | 0.20 |
| Math 1050 | 2.45 | C+ | 8,647 | 2.79 | B- | 269 | <0.01 |
| Math 1060 | 2.54 | C+ | 4,075 | 2.95 | B- | 164 | <0.01 |
| Math 1100 | 2.23 | C | 3,413 | 2.45 | C+ | 63 | 0.22 |
| Math 1210 | 2.50 | C+ | 2,152 | 2.88 | B- | 58 | 0.03 |
| Math 2020 | 3.07 | B | 755 | 4.00 | A | 2 | 0.18 |
| Stat 1040 | 2.40 | C | 3,752 | 2.94 | B- | 250 | <0.01 |
| Stat 2000 | 2.72 | B- | 379 | 3.5 | B+ | 2 | .035 |
| Stat 2300 | 2.48 | C+ | 3,388 | 2.71 | B- | 62 | 0.13 |
| Overall | 2.42 | C | 32,101 | 2.46 | C | 2,443 | 0.09 |
Conclusion: Overall students that qualified for admission to a course by the MPT have the same grades, on average, as students that qualified for admission by some other mechanism. This suggests that the current cut-offs for the various courses are approximately correct. For some courses—Math 1050, 1060, Math 1210 and Stat 1040—students that obtained admission to the course by the MPT than students that did not. This result suggests that for these courses, the MPT cutoffs are a little high. We shall continue to monitor these results with a view toward an adjustment of cutoff scores in the future.
Analyses carried out by: Cortnie Broadus and Nicole Rupp under the supervision of Dr. John Stevens. 24 March 2009.
4. Is there any relationship between students’ scores on the MPT and their final grades?
The final course grade was plotted against the placement exam score for those students who needed the placement exam to place into the course (plots shown below).
Most of the plots proved to have a significant trend at the alpha = .05 level, but the R-square is quite low for all of them, as shown in the table. Math 1030, Stat 2000, and Stat 2300 are the three plots that were non-significant. The others show that higher scores correspond to a positive slope (higher grade).
It can be seen from these plots that some students were able to take courses despite missing the courses’ formal cut-off scores. The reason for this is not clear, but there may be other pre-course requirements that were met.
| Course | Slope | P-value | R² |
| Math 1010 | 0.032 | <0.001 | 8% |
| Math 1030 | 0.009 | 0.438 | 1% |
| Math 1050 | 0.016 | <0.001 | 4% |
| Math 1060 | 0.024 | <0.001 | 8% |
| Math 1100 | 0.011 | 0.016 | 2% |
| Math 1210 | 0.012 | 0.001 | 4% |
| Math 2020 | 0.028 | 0.002 | 27% |
| Stat 1040 | 0.016 | <0.001 | 4% |
| Stat 2000 | 0.012 | 0.497 | 1% |
| Stat 2300 | 0.006 | 0.134 | 1% |
|
|
|
|
|
Conclusion: There is a weak but discernable positive relationship between score on the MPT and final grade.
Analyses carried out by: Ben Johnson under the supervision of Dr. John Stevens. 7 May 2009.
5. What is the overall impact of the MPT on D-F-W rates of courses that the MPT is used to place into?
Data on the combined D-F-W rate (that is, final grades of D or F or student withdrew from the course) were obtained for Math 0900, 1010, 1050, 1100 and Stat 1040 and 2300 for fall semester 2005 through Spring 2009, and are tabulated below. The numbers in parentheses below the D-F-W rates are the total numbers of students in the respective courses for that semester. Math 1060 and Stat 2000 are deliberately omitted from the table due to small sample sizes.
| Course | Semester | |||||||
| Fall | Spring | Fall | Spring | Fall | Spring | Fall | Spring | |
| 2005 | 2006 | 2006 | 2007 | 2007 | 2008 | 2008 | 2009 | |
| Math | 30.8 | 36.0 | 36.4 | 40.5 | 40.3 | 42.7 | 32.7 | 39.5 |
| 0900 | (383) | (253) | (390) | (289) | (511) | (307) | (621) | (387) |
| Math | 31.0 | 34.0 | 26.6 | 31.5 | 26.5 | 29.0 | 30.7 | 30.5 |
| 1010 | (1,053) | (702) | (1,325) | (670) | (868) | (738) | (1,048) | (891) |
| Math | 17.7% | 21.9% | 16.2% | 21.4% | 18.3% | 12.2% | 13.4% | 15.1% |
| 1050 | (2,008) | (1,569) | (2,239) | (2,006) | (1,958) | (1,826) | (1,484) | (1,609) |
| Math | 26.7% | 30.0% | 26.7% | 33.8% | 22.4% | 31.1% | 25.4% | 30.3% |
| 1100 | (465) | (444) | (603) | (420) | (615) | (302) | (493) | (406) |
| Stat | 25.3% | 26.1% | 17.8% | 27.2% | 19.0% | 15.8% | 16.2% | 13.2% |
| 1040 | (691) | (521) | (714) | (544) | (463) | (329) | (470) | (423) |
| Stat | 22.0% | 22.8% | 23.5% | 29.0% | 13.8% | 12.4% | 11.8% | 16.0% |
| 2300 | (459) | (382) | (533) | (555) | (347) | (581) | (296) | (444) |
Conclusions: There are considerable fluctuations in the D-F-W rates for the courses. In some cases there are clear fall versus spring differences. For example, for Math 1100 the D-F-W rate is about 4%--5% higher in spring than in fall, even though most of the teachers are the same.
In terms of the main question of this section, the MPT appears to be effective in substantially reducing the D-F-W rate in some of the targeted courses. Specifically,
- the D-F-W rate in Stat 2300 has decreased dramatically, and the drop coincides exactly with the full implementation of the MPT in Fall 2007. In the four semesters prior to the implementation of the MPT the D-F-W rate for Stat 2300 ranged from 22% to 29%; since the MPT was implemented the D-F-W rates have ranged from 11.8% to 16%. The same instructor taught the course over that 8 semester period.
- The D-F-W rate has also decreased in Stat 1040. In the last three semesters the D-F-W rate has ranged from 13.2% to 16.1%, which are lower values than for any semester before the implementation of the MPT.
- There seems to be a similar drop in D-F-W rate for Math 1050, which was a high priority target for the placement test. In the last three semesters the D-F-W rate has ranged from 12.2% to 15.1%, which are lower values than for any semester before the implementation of the MPT.
- Over the same period the D-F-W rate has increased in Math 0900. Our supposition is that some students that, in the past, would have taken and failed Math 1050, Stat 1040 and Stat 2300 are now placing into Math 0900 and Math 1010, and some of these students are struggling even in these lower level mathematics courses.
- At this time there is insufficient evidence to conclude that the MPT has had an impact on Math 1010 and Math 1100. In both courses the D-F-W rate seems
Data provided by: Linda Skabelund. Interpretation by: Richard Cutler. 1 September, 2009.
6.What are good predictors of a student’s score on the MPT?
Using data from fall 2007 through spring 2009, a multiple linear regression model was used to predict the placement exam score (on the Accuplacer college level math exam) based on previous experience. With 1,424 observations, the R-square value for the final model was relatively high at 47.9% and model assumptions were verified. The p-value for the model was <.0001. The following are the predictors in the final model: the student’s math act score, the student’s undergraduate overall GPA, the time since the student’s last math class, the student’s enrollment in the college of Engineering or Science, whether or not the student was a sophomore, junior, or senior, and the student’s gender. The following interactions were also significant: gender and math act score, college of Business and gender, college of Engineering and gender, college of HASS and gender, and the undergraduate overall GPA and status as a junior.
After analyzing the data, as a student’s math act score increases, undergraduate overall GPA increases, enrollment in the college of Engineering or Science, and being a sophomore, junior, or senior, his/her placement exam score also increases. On the other hand, the longer a student has gone without taking a math class, the lower his/her predicted placement exam score. The final model also suggests that being a female reduces the predicted placement exam score.
To interpret the interactions, math act score has a stronger influence for females than for males and being in the college of Business and Engineering also has a stronger influence for females than for males. Whereas, being in the college of HASS has a smaller influence for females than for males. These differences among the colleges may be due to the fact that more females are in the college of HASS than in the colleges of Business and Engineering. Lastly, the undergraduate overall GPA of a student has a smaller influence for juniors than for freshman.
The following is the proc reg output from the SAS output window for our final model. The following are the variables: ACTMath is the students math ACT score, UndergradOverallGPA is the students undergraduate overall GPA, time is the time since the students last math class, S2 is an indicator for enrollment in the College of Business, S4 is an indicator for enrollment in the College of Engineering, S5 is an indicator for enrollment in the College of HASS, S7 is an indicator for enrollment in the College of Science, L2 is an indicator for being a Junior, L3 is an indicator for being a Sophomore, L4 is an indicator for being a Senior, and G1 is an indicator for gender (with G1=1 being female). The five interactions can be defined as follows: Gact is the interaction between gender and ACTMath, S2G1 is the interaction between being in the College of Business and gender, S4G1 is the interaction between being in the College of Engineering and gender, S5G1 is the interaction between being in the College of HASS and gender, and L2un is the interaction between undergraduate overall GPA and being a junior.
Analyses carried out by: Cortnie Broadus and Nicole Rupp under the supervision of Dr. John Stevens. 23 March 2009.
| FINAL MODEL | |||||
| The REG Procedure | |||||
| Model: MODEL1 | |||||
| Dependent Variable: logclm | |||||
| Number of Observations Read = 1424 | |||||
| Number of Observations Used = 1424 | |||||
| Analysis of Variance | |||||
| Source | DF | Sum of Squares | Mean Square | F Value | Pr > F |
| Model | 16 | 115.47467 | 7.21717 | 80.90 | <.0001 |
| Error | 1407 | 125.51339 | 0.08921 | ||
| Corrected Total | 1423 | 240.98806 | |||
| Root MSE | 0.29867 | R-Square | 0.4792 | ||
| Dependent Mean | 3.86601 | Adj R-Sq | 0.4732 | ||
| Coeff Var | 7.72566 | ||||
| FINAL MODEL | |||||
| The REG Procedure | |||||
| Model: MODEL1 | |||||
| Dependent Variable: logclm | |||||
| Parameter Estimates | |||||
| Variable | DF | Parameter Estimate | Standard Error | t Value | Pr > |t| |
| Inytercept | 1 | 2.54496 | 0.07405 | 34.37 | <.0001 |
| ACT Math | 1 | 0.04804 | 0.00275 | 17.46 | <.0001 |
| Undergrad Overall GPA | 1 | 0.06045 | 0.01485 | 4.07 | <.0001 |
| time | 1 | -0.07058 | 0.00719 | -9.81 | <.0001 |
| S2 | 1 | 0.02884 | 0.02957 | 0.98 | .03296 |
| S4 | 1 | 0.16531 | 0.02872 | 5.76 | <.0001 |
| S5 | 1 | 0.01876 | 0.04172 | 0.45 | 0.6531 |
| S7 | 1 | 0.06457 | 0.02638 | 2.45 | 0.0145 |
| L2 | 1 | 0.62271 | 0.11913 | 5.23 | <.0001 |
| L3 | 1 | 0.04685 | 0.02248 | 2.08 | 0.0373 |
| L4 | 1 | 0.2249 | 0.03295 | 6.81 | <.0001 |
| G1 | 1 | -0.32312 | 0.10069 | -3.21 | 0.0014 |
| G act | 1 | 0.01048 | 0.00427 | 2.46 | 0.0141 |
| S2G1 | 1 | 0.11068 | 0.04758 | 2.33 | 0.0202 |
| S4G1 | 1 | 0.16118 | 0.07905 | 2.04 | 0.0417 |
| S5G1 | 1 | -0.09791 | 0.05038 | -1.94 | 0.0522 |
| L2un | 1 | -0.14189 | 0.03713 | -3.82 | |