Cover letter from Rob Pangborn

 

  Final Report from University-wide Math Commission, June 2001

 

  IMPLEMENTATION PROCESS from Rodney A. Erickson

 

  Cover letter from Rob Pangborn

PENN STATE

College of Engineering	               Robert N.  Pangborn, Dean 814.863.3750
Undergraduate Studies Dean's Office    Betty A. Mantz,
101 Hammond Building	                 Administrative Assistant
University Park, PA 16802	       Facsimile Number	         814.363.4749



Memorandum

DATE:	July 25,2001
TO:	Rodney A. Erickson
FROM:	Rob Pangborn
RE:	Math Commission Report

Attached are the final report and recommendations of the University-wide
Math Commission that was appointed in January. The Commission met every two
weeks throughout the semester and did substantial research and analysis of
placement data, current processes, and newer, more diagnostic approaches.
There is unanimous agreement among commission members that web-based
diagnostic testing will be of significant benefit to Penn State and its
students. However, the membership was more divided on the prospect of using
the test for binding placement of students in an appropriate level in the
math course sequences. The representatives of the math faculty on the
Commission were most apprehensive about reports from other schools that
taking the test in an unsecured, unproctored environment can result in
students' placement above their abilities. We don't want to inadvertently
jeopardize the strong record of success we have achieved at Penn State, both
in terms of students' performance in the math courses and in overall student
retention. However, all members agreed that, if pilot testing proves to be
reliable, there could be little argument with, or resistance to, utilizing
the web-based format for placement purposes.

Therefore, as indicated in the recommendations, we are encouraging that some
caution be exercised (#1) while a web-based diagnostic test is initially
developed (#2) and implemented, and that this new tool be carefully assessed
to establish consistency with the traditional placement process which has
been remarkably successful across the Penn State system (#3). We do,
however, believe that the web-test could be used as soon as it is available
for the significant cadre of students who do not anticipate taking a formal
sequence of mathematics study, and that computer-based testing could be
accommodated in a number of colleges that have appropriate facilities. As
stated at the conclusion of the report, these experiments with computer and
web-based testing could potentially involve one-third to one- half of all
entering students. Along with corresponding assessment, this pilot
experience should provide confidence in the technology and refinements in
the placement process that will allow extension of the web-based approach to
the broader population of entering students.

I hope you will find this report informative and responsive to the charge 
you gave the Commission. 

------------------------------------------------------------------------

cc: J. Cahir	
J. Cohen

Attachment	

Final Report from University-wide Math Commission, June 2001



               University-wide Math Commission
               Final Report and Recommendations
                         June 2001


                      Executive Summary

The University-wide Math Commission was charged by the provost to recommend
new procedures for mathematics placement and diagnostic testing.  During the
Spring semester, 2001, the Commission reviewed benchmarking information from
within the CIC and from other institutions.  It analyzed data to assess the
effectiveness of math placement and student performance at Penn State.
Members of the Commission researched the relevant literature on placement
methods, retention, math testing, and the like.  The Commission also
reviewed opportunities and issues associated with web-based testing, in
particular examining a pilot web-testing study completed at Penn State
during this semester.  Based on its evaluation, the Commission makes the
following recommendations.

(1) For the present, maintain the existing  pencil-and-paper  test procedure,
or some  comparably   secured option,  for testing    and  placing   incoming
students.

(2) Develop and offer a web-based diagnostic test - to include both enhanced
diagnostic capabilities and an embedded (traditional) placement test - to
all incoming students.

(3) Complete a systematic, comprehensive, data-based evaluation of the
effectiveness of a web-based placement testing prototype at Penn State.

(4) Eliminate testing redundancy or unneeded security, and explore interim
collaborations for computer-based testing options, and pilot a web-based
diagnotic testing/skills assessment instrument for students in academic
tracks that do not require mathematics course sequences or calculus.


                           Charge

The University-wide Math Commission was charged by executive vice
president and provost Rodney A. Erickson on January 31, 2001. The
provost asked the Commission "to recommend new procedures for
mathematics placement which take a diagnostic approach, preferably
using the World-Wide Web. The procedures should be aimed at clearly
discriminating student preparation for the various levels of beginning
mathematics, including preparation of students who do not contemplate
taking calculus." The Commission was further asked to provide
suggestions on how to deal with related implementation issues and to
take a Universitywide approach, so that the new system will apply at
all campuses.

In carrying out its charge, the Commission was encouraged to consider
diagnostic approaches, updating of cut-off scores, updating of methods
used for calculus- and non-calculus bound student populations, and
strategies for reassignment of students who may have been incorrectly
placed.

                            Membership
                                                                       
Maria-Carme Calderer, Co-Chair      (Department of Mathematics)        
Robert N. Pangborn, Co-Chair        (College of Engineering)           
George E. Andrews            	    (Department of Mathematics)        
Douglas K. Brown		    (Penn State Altoona College)       
Michael J. Dooris		    (Center for Quality and Planning)  
Paul J. Hutta			    (Penn State Abington College)      
Janet Fay Jester		    (Department of Mathematics)        
Shyam S. Sethuraman		    (College of Communications)        
Eric R. White			    (Division of Undergraduate Studies)
Dawn M. Zimmaro			    (University Testing Services)      

                           Process

The Commission met first as a group on February 27, 2001. It organized
itself almost immediately into four informal working groups to
investigate and report out in various key areas:

* Benchmarking - with CIC and other institutions 
* Data analvsis - assessment of placement and student 
  success at Penn State
* Benchmarking - with CIC and other institutions
* Data analysis - assessment of placement and student success at Penn State
* Research - the literature on placement methods, retention, math testing,
 and related matters
* Web-based approaches - evaluating opportunities and issues associated with
 web-based testing.

                          Benchmarking

The Commission gathered information from CIC and other peer institutions,
as well as from schools dissimilar to Penn State, such as smaller
universities and private or community colleges.  The Commission especially
sought information on computer-based testing.

Benchmarking results were mixed, probably because testing is only one of
many factors - such as the range of incoming student abilities, the nature
of advising, the sequencing of math courses - that make the total package
of math placement and education at any particular college or university
unique.

In the Big Ten, Ohio State, Michigan, Michigan State, and Minnesota are
doing placement on the web.  Although the programs are new enough that
little information on their success is available, the early reports from
Minnesota and Michigan State are generally positive, while Ohio State has
experienced scheduling difficulty because more students are placing into
calculus with web testing than with paper-and-pencil testing.

Other Big Ten schools are experimenting with web-based testing for
placement or early diagnostics, including Wisconsin, Northwestern, and
Purdue.  Nebraska is preparing to go to web-based testing, while Illinois,
Texas at Austin, MIT and others including Pitt and Temple still use
paper-and-pencil tests.  The purposes of the on-line and/or computer-based
tests vary from school to school, as does whether the tests are freely
accessible on the web or taken only at proctored sites.  There are other
differences as well: time limits, opportunity for retakes, the use of
calculators, and so on.

Web-based testing is available at some schools for diagnostic purposes but
not for placement. At others, proctored computer-based testing is available
as an optional alternative to pencil-and-paper.  Some institutions compare
the results of placement testing to math SAT scores or other measures to
judge validity and reliability.  At some schools, placement is binding; at
others, the results are used only to provide recommendations on course
selection.

Most placement tests have been created in-house by the individual
university and tailored to its courses, although some schools are using
tests purchased from ETS or ACT.  The commercial products tend to be used
by community colleges, in particular; they offer adaptive capabilities and
often cover a number of subjects.

Virginia Tech reported a pre-calculus (probably equivalent to Penn State's
Math 22) success rate -- defined as the proportion of students earning a C
or better -- of 54 percent in 1996.  Virginia Tech subsequently moved to an
"emporium format" and improved the rate to 73 percent by 1998.  (Very
briefly, the Virginia Tech math emporium is a technology-intensive approach
that emphasizes 24/7 access to learning resources, active learning, team
teaching, modular design of courses and materials, and continuous
improvement.) Penn State's data will be explored in the following section,
but the comparable success rate for Math 22 students at University Park
(the best Penn State comparator to Blacksburg) is 57 percent.

                           Data Analysis

Members of the Commission reviewed considerable data in some detail, so
generalizations are tricky.  Nonetheless, it is fair to say that many
aspects of the placement procedures and math sequences basically work well.
The Math Department's small-section calculus initiative has been very
successful, improving student success rates from about 45 percent to about
66 percent.

The drop-in-sequence option, through which a student struggling in, say,
Math 22 is allowed to drop to Math 21 early in the semester, also is
valuable; about 50-60 students take advantage of this option in a given
semester for all math courses.  On the other hand, a situation in which two
to three percent of Math 22 and Math 140 students need to drop in sequence
also highlights an opportunity for improvement.

Course-Taking Patterns

The Commission examined course-taking patterns for three different
categories of courses: calculus (primarily Math 110, 111, 140, 141),
pre-calculus (primarily Math 21, 22, 26) and general math (primarily Math
17, 35).  Table 1 shows the course-taking patterns for 5,763 Spring 2000
baccalaureate degree graduates (based on Registrar's data for these
students, going back ten years).


baccalaureate degree graduates (based on Registrar's data for these
students, going back ten years).

                               Table 1.
      Type of Math Courses Completed by Spring 2000 Graduates

University-wide         University Park         Campus colleges                
Calc	3157  (54.8%)	Calc    1731  (61.4%)   Calc    1426  (48.4%)     
Precalc 2356  (40.9%)	Precalc  759  (26.9%)	Precalc 1597  (54.2%)
General	690   (12.0%)	General  260  ( 9.2%)	General 430   (14.6%)


Because of data such as those in Table 1, the Math Commission's
analyses have dealt with the full range of math courses. In other
words, the Commission members' concerns include but are not limited to
the more advanced calculus sequences.

Not all Penn State students have an interest in math and not all are
in mathintensive majors. Table 2 shows the proportion of these 5,763
students taking calculus or pre-calculus, and those who satisfied the
general education quantification requirement in some other way (for
example, with logic or computer science or statistics courses).

                               Table 2.
         Spring 2000 Graduates: GQ Courses Completed

Only non-calculus/non-pre-calc	  Some other mix
	17%	                         74%
Only calculus/pre-calc	          None (e.g., transfer students)
	6% 	                          2%

The 17 percent figure illustrates that math placement is, in effect, a 
nonissue for a sizable minority of Penn State undergraduates.

Placement Testing and Success Rates

The Commission analyzed in particular detail five years of data
(1995-96 through 1999-2000) on students' math grades in relation to
FTCAP test scores. The Commission defined "success" as a C or better,
and included students who withdrew from math courses in the analvsis.

the data show that scoring and placement work reasonably well on the
whole.  For example, the overall success rate was 68 percent for Math
140 and 61 percent for Math 22. Although peer data are not extensive
nor exactly comparable, these overall Penn State rates seem more or
less consistent with the experience of other schools.

                              Table 3.
    Test Subscores and Related Success Rates for 1999-00

	Math 140 -	Earned C or	Math 22 -	Earned C or
	Relevant	Better in	Relevant	Better in
	Subscore	Math 140	Subscore	Math 22
	0-11	        49%
	12-13	        55%	        0-6	        41%
	14-25	        68%	        7-12	        56%
	26-34	        92%	        13-19	        69%
	Total	        68%	        Total	        61%

Table 3 illustrates, at a highly summarized level, several threads
that the Commission also observed in more detailed data. Placement
testing works especially well for students who are strong in math; it
does not work as well for weaker students. Placement testing works
better for upper-level calculus courses than lower- level math
courses. While Table 3 does not reveal as much about remediation as
about testing, it suggests that remediation may work less well than
placement. Other data support that idea; for example, the success rate
for students who place directly into Math 140 is 78 percent, but the
success rate for Math 140 is only 46 percent for students who first
completed Math 22. (This is also consistent with findings from the
research literature, reviewed later in this report.)

The Commission also examined information for all of the main pre-calculus
and calculus courses, and for all of Penn State's campus colleges, and there
are significantly different patterns in the data for University Park and
other locations.  For example, in direct placement situations, University
Park students are more successful than Commonwealth College students in Math
140 (73 percent versus 61 percent) and Math 110 (71 percent versus 63
percent). However, direct placement students are less successful at
University Park in Math 22 (57 percent versus 61 percent). Remediated
students are generally better off at a campus college than at University
Park in terms of eventual success rates in targeted, higher-level courses.

Remediation works less and less well as students need more and more
remediation.  At University Park, the Math 140 success rate for students who
started with Math 22 is 41 percent; the Math 140 success rate for students
who started with Math 21 is 22 percent.  The relationship is similar for
other course sequences and other locations.  In short, placement and test
scoring seem to work reasonably well; remediation is not a completely
successful substitute for strong high school math preparation.

                             Research

Testing

The current test administered by Penn State is based on the model developed
by the Mathematics Association of America (MAA) in the early 1990's. The MAA
examinations have been administered by many academic institutions
nationally.  Test design focused on measuring the probability of success -
defined as the probability of earning a C or better in calculus.  The MAA
examinations are in paper-and-pencil format, and are administered in
secured, proctored settings.  Observers agree that these tests predict
calculus success fairly well, and that they have not been as effective in
predicting success in college algebra and trigonometry.  Adequate studies
are lacking, however; in any case, in the year 2000 the MAA discontinued
production of the tests.  The decision was in part motivated by the
acknowledgement that placement testing should be as sensitive as possible to
the academic situation at each institution.  It is difficult for a single
test format to serve unique, diverse institutional needs.

Remediation

Nationally, college and university pre-calculus curricula have been
implemented - reluctantly in some cases - to remediate what colleges
perceived as an instructional deficiency for which high schools were
primarily responsible.  Because of this perception, the nature and structure
of these efforts vary widely across colleges and universities. For example,
in some universities of the SUNY system, the remedial curriculum has been
discontinued, effectively shifting this instructional responsibility to high
schools.

Pragmatically, effectiveness in remedial mathematics is highly linked with
courses having a very specific and small set of goals.  This means that
courses attempting to address a myriad of deficiencies, or poorly defined
deficiencies, are more likely to fail (as measured by student success rates)
than courses that target clear and focused goals ("Remedial Education in
Colleges and Universities: What's Really Going On?" Jamie P. Merisotis and
Ronald A. Phipps, The Review of Higher Education, Fall 2000, Volume 24, No.
1, pp. 67-85).

                          Web-based Approaches

University Testing Services, the Division of Undergraduate Studies, and the
Department of Mathematics collaborated to develop and offer a practice
version of the math placement exam via the web to incoming first-year
students.  Letters invited 1,850 students who were Fall 2001 paid accepts as
of December 2000 to participate in the web exam in early Spring 2001. 350 of
those incoming students actually attempted the web-based test.

The test required students to log in with their social security number.  The
test had 72 items parallel (that is, similar in content and difficulty) to
the items on the paper-and-pencil placement test.  The test was timed, to
allow 70 minutes in total.  Students received subset scores and other
feedback (such as a list of suggested topics where more study was indicated,
based on the FTCAP advising cutoffs and the FTCAP advising guide).  These
students were still required to take the paper-and-pencil version of the
FTCAP test.

In the pilot study results, all four math subscores (basic math, algebra I,
algebra II, and trigonometry) and the total math test score were positively
and significantly correlated with math SAT scores.  High school grade point
average did not correlate as well with the math test scores, but this should
be expected; a strong high school g.p.a. does not necessarily mean that a
student ever took trigonometry or calculus.  The test did not discriminate
well between University Park and campus college admits and again, this is as
expected; high school g.p.a. is the primary variable used in the prediction
model which refers students to campuses and, as already noted, g.p.a. is not
necessarily a valid measure of high school math preparation.  Math
Commission members reached the following conclusions from the pilot
web-testing program.

1. Based on comparing the web results with a control group and with student
   scores on the FTCAP test, it does not appear that students cheated on the
   web test.

2. There were strong correlations between the web test and the FTCAP test
   scores, suggesting that the web test is a valid measure of performance on
   the FTCAP test.

3. Students who took the web test generally did better on the FTCAP test
   than students who did not take the web test.  This might indicate that
   the web test may have helped students to study and review material before
   taking the FTCAP test.

4 Students who had low to average SAT math scores were most likely to show
  an improvement from the web test to the FTCAP test; for these students,
  there was a statistically significant (although small) gain.

5. Students who had a high SAT score did well on both the web and FTCAP
   tests.

6. The web test scores were generally lower, and consequently placement
   action based on the web test scores would lead to placement in lower math
   courses. 

7. The greatest variation between the web and FTCAP test scores in terms of
   placement was in the mid-range math courses such as Math 21, 22, and 26.

There are obvious limitations to interpreting the results of this pilot test
- for example, the participating students self-selected into web-based
testing, and the students choosing to participate tended to have relatively
high math SAT scores - so the Math Commission hesitates to reach premature
conclusions.  Nonetheless, at this point the pilot provides an informative,
positive basis for continued consideration and evaluation of web-based math
testing at Penn State.

                        Guiding Principles

The members of the Math Commission agree in principle on many aspects of
desirable future directions for math placement at Penn State.  The processes
in place certainly work, and while there may be opportunities for
improvement, any changes should build upon, and not damage, the current
system.

The lack of security (whether through proctoring, authentication, or some
other means) inherent in on-line placement testing is a main reason why this
method of testing has not been embraced by the mathematics community.
Security is an important issue for placement testing, especially in
circumstances in which placement is binding.

It would be desirable for placement testing to be more diagnostic - that is,
to provide feedback to incoming students on their mastery of various math
subjects (algebra, geometry, trigonometry, and calculus), so that they can
better address any weaknesses, possibly even before enrolling at Penn
State.

Non-secure web-based testing can be valuable and appropriate as a diagnostic
tool that provides students or prospective students with fast, useful
feedback.  Security is not an issue for diagnostic testing.

It would be desirable for testing to be more adaptive - that is, to be more
responsive to the ability of individual students taking the test, and adjust
questions presented to each student based on prior correct or incorrect
answers.

Placement testing should be considered as one element that works in
combination with a strong advising system and an array of math courses and
sequences.  The goal for the system is to help inform and support students
in their choice of courses, majors, and career options (including those that
do not require calculus), and to help them to do well.

Any changes should preserve the beneficial aspects of the drop-in-sequence
and other measures that help students to succeed in their first Penn State
math course.

Testing should produce reliable, accurate, and useful measures, while taking
advantage of opportunities to improve existing processes, possibly reducing
cost and time commitments.



                  Specific Recommendations

(1) For the present, maintain the existing pencil-and-paper test procedure,
    or some comparably secured option, for testing and placing incoming
    students.

Most of the impacts, both positive and negative, of performing placement
testing via an on-line (unsecured) test, are speculative.  It is only
prudent to maintain a system that works.  In the meantime, alternative
processes can be developed, prototyped, and thoroughly evaluated.

(2) Develop and offer a web-based diagnostic test - to include both enhanced
    diagnostic capabilities and an embedded (traditional) placement test -
    to all incoming students.

The members of the Math Commission all agree that the ability to provide
early, formative feedback to prospective students is a worthwhile and
compelling goal.  This recommendation allows Penn State to move forward with
web-based testing without abandoning the validity of the existing
instrument, while providing an opportunity to further evaluate online
testing.

By embedding the conventional 50-question placement test in the new,
diagnostic instrument, Penn State will be able to expand pilot testing,
compare the results and cutoffs for the web-based test with those from
pencil-and-paper testing, and identify comparable scores/cutoffs for
placement using the new, expanded question bank.

The cost of developing the test for diagnostic purposes is independent of
the test's eventual or potential use for placement.  Providing appropriate
technical service to users and collecting and compiling the test results
would probably be done for informational purposes in any case.

Issues of when the test would be taken and by whom (by applicants, by
offers, by paid-accepts) would be determined as part of implementation, as
would details about how often the test could be taken, at what intervals,
with what imposed time constraints, and so on.

(3) Complete a systematic, comprehensive, data-based evaluation of the
    effectiveness of a web-based placement testing prototype at Penn
    State.

It can be argued that existing helpful interventions such as the
drop-in-sequence are founded on controlled enrollment based on binding
placement, and hence on secure testing.  This line of reasoning assumes that
cheating will likely occur in an unsecured setting, causing high numbers of
misplaced students and overwhelming the Math Department's ability to
reassign them.  These are legitimate concerns.  On the other hand, the need
for a drop-in-sequence option is itself evidence that current placement
practices are not perfect.

This recommendation is offered because the Math Commission wants to preserve
the strengths of the existing system, while remaining open to potential
improvements.  Therefore, it would be prudent to have a better idea of what
is likely to happen by having traditional results available during some
initial trial period.  This will insure that comparative data are available
to feed a comprehensive assessment process and answer a wide variety of
questions.

	Can problems with technology (incompatibility of home computers,
	etc.) be effectively overcome?

	Is the distribution of scores for the unsecured web test
	significantly different from the proctored, pencil-and-paper
	version? What is the incidence of significantly different
	results for individual students?

	To what extent does student performance improve as a result of
	early feedback from the web-based tests?

        Can the ability to add an adaptive element to testing make placement
	decision even more accurate?

(4) Eliminate testing redundancy or unneeded security, and explore interim
    collaborations for computer-based testing options.

In a typical year, about 13,000 incoming students take the placement test.
Of those, 15 to 20 percent satisfy the quantification category of general
education with courses such as logic, probability, and introductory computer
programming, or general math having no prerequisites.  Thus, for some 3,000
incoming students taking the placement test each year, whether it is binding
or not is of little consequence.  For students very committed to entering
certain career paths, further testing beyond the web-based version is
unnecessary, or could be accomplished later if their aspirations changed.

Another 10 to 20 percent of students might be accommodated in University
Park or Campus College computer labs to take the exam under secure
conditions, while agreements might be struck with some larger high schools
to offer the exam to students there.  This approach would also provide data
by which to compare the results for individual students taking the same test
under both proctored and un-monitored conditions.

Conservatively, between one-third and one-half of the current
paper-and-pencil testing could be eliminated without affecting the process
of placing students in the math sequences at all. 

Finally, the Commission encourages that au the academic units continue to
evaluate the needs of their majors for study in mathematics, statistics,
computer science and other competencies falling under the broad term of
quantification. Input from the various disciplines will be essential to the
career success of graduates and the design of curricular offerings
appropriate for students on different career tracks. It will also be crucial
to the development of formative feedback to students emanating from the
proposed diagnostic testing and to the effectiveness of the advising system.

---------------------------------------------------------------------------

IMPLEMENTATION PROCESS from Rodney A. Erickson

Date:      November 15, 2001
From:      Rodney A. Erickson
To: 	   University Academic Measurement Committee
Subject:   Math Placement

I recently appointed a nine-member, University-wide commission to recommend
new procedures for math placement that take a diagnostic approach,
preferably using the web. While the Commission endorsed the ideas associated
with that approach, they asked that it be done in parallel with the existing
system. I also asked them to provide suggestions on how to deal with the
related implementation issues. The commission's analyses and four
accompanying recommendations contribute to a general understanding of math
placement for first year Penn State students, but are silent on the
implementation path I believe that we should take. Accordingly, I would like
us to move forward by taking the following steps.

Oversight: I am asking the UAMC to provide active oversight to the
implementation process, which the Office of Undergraduate Education
(University Testing and Division of Undergraduate Studies) and the Office of
Administrative Systems will carry out.

Pilot Test on the Web: Following the Math Commission's recommendation, an
immediate pilot should be implemented in which all out-of-state students,
inclusive of international students, take their fall 2002 placement exams in
math, English, and chemistry, on the Web. This exam should eliminate the
need for the tested students to take parallel paper and pencil exams or to
feel pressured by not having them completed when other students do.
Moreover, it will eliminate long trips for the few who now come to
Pennsylvania for testing.

Useful Advising: The report indicates that we possess the potential to
better incorporate placement testing into academic advising. With early
feedback, students may be able to address remediation needs prior to first
year enrollment, or early in their college experience. Appropriate feedback
also enables greater accuracy in student academic planning. Accordingly, I
am asking DUS to begin work on diagnosis and advising messages and to give
particular attention to the assessment of the Web-based placement pilot and
its implications for enhanced advising and student success.

In sum, the report indicates that appropriate placement, productive academic
advising, self-remediation, and useful diagnostics benefit significantly
from timely feedback. Web-based testing will be an increasingly valuable
tool to achieve these benefits, and your active involvement in seeing that
this happens this year will be respected and appreciated for many years to
come.

The recommendations of the Math Commission are much appreciated, and I look
forward to reviewing the implementation of the protocols discussed above. I
want to thank members of UAMC for their participation in this important
project.

cc:	Math Commission Members
	J. Gary Augustson
	Daniel J. Larson
	Susan Welch
	John J. Cahir