ETS Proficiency Profile 

Introduction

 

The ETS Proficiency Profile (ETS-PP) tests college level general education skills in reading, critical thinking, writing, and mathematics.  When administered to incoming first year students and graduating seniors, it provides a measure of students’ learning in college.  Thus the results inform improvement in curriculum and instruction that go beyond learning in individual majors to encompass the entire undergraduate experience.  

 

One use of the ETS-PP is to measure learning in college, compared across institutions. “Learning Gains” are a relative measure of improvement in critical thinking and writing skills during college, computed by comparing the differences in scores of first year and senior students across institutions. Learning Gains in the areas of critical thinking and writing are reported on our College Portrait.  Reporting Learning Gains from a test like the ETS-PP is a requirement of participation in the Voluntary System of Accountability.

 

What We Did

 

In Fall 2010 and again in Fall 2012, we offered the online unproctored version of the ETS-PP to students enrolled in courses that are typically taken by first year students. In Spring 2011 and again in Spring 2013, we offered the same test to students enrolled in senior capstone courses. In almost all cases, instructors of those courses offered grade incentives to students to complete the test, as required or bonus points. We provided instructors with sample syllabus language (approved by UNF’s IRB), and we offered instructors a class presentation to explain the test to their students. We emailed the students directly about how to access the test itself, providing them with a web link and instructions. Students had a two week window for test completion, during which they received multiple reminders.

 

Results

 

The key results are as follows:

  • The great news: in both 2010-11 and 2012-13, UNF’s Learning Gain was “well above expected” (the highest possible score) for critical thinking, and “above expected” for writing.
  • Our students, both first year and senior, scored higher than the national norms for institutions of our type (Masters [Comprehensive] Colleges and Universities I & II, in the old Carnegie classification).
  • Performance of our seniors varied widely by college; possible explanations include proficiency definitions, student self-selection, specific selection of courses within colleges that participated, or differences in curriculum between colleges. 

Things to Know in Interpreting the Results 

  • Learning Gains in the areas of critical thinking and writing are computed by administering the same test to incoming first year students and graduating seniors; regressing the scores of the first year and graduating students on their entering SAT or ACT scores, based upon regression equations computed by ETS using institutions with a similar SAT/ACT profile as UNF's; and then computing a relative "Learning Gain" in each area. A number of students who were enrolled in first year or senior courses were not, in fact, first year or senior students, and were not included in the calculation of Learning Gains.
  • Transfer students are not included in the Learning Gains calculation.
  • ETS defines proficiency classifications that are tested at each level.  

Where We Go From Here

  • As a condition of participation in the Voluntary System of Accountability, UNF must update its learning gains scores at least once every three years. We anticipate administering the ETS-PP again in 2015-2016.
  • As a first point for attention, we aspire to have a majority of our graduating seniors be proficient in writing and critical thinking. Conversations are currently underway to determine how UNF can improve its scores. Specifically, we are discussing how the definitions of writing and critical thinking in the Academic Learning Compacts compare to the ETS definitions, and whether it would be appropriate to bring more consistency to definitions of writing and critical thinking competencies across units.