ETS Proficiency Profile 

Introduction 

 

The ETS Proficiency Profile (ETS-PP) tests college level general education skills in reading, critical thinking, writing, and mathematics.  When administered to incoming first year students and graduating seniors, it provides a measure of students’ learning in college.  Thus the results inform improvement in curriculum and instruction that go beyond learning in individual majors to encompass the entire undergraduate experience.  

 

One use of the ETS-PP is to measure learning in college, compared across institutions. “Learning Gains” are a relative measure of improvement in critical thinking and writing skills during college, computed by comparing scores of first year and senior students, normalized to their SAT scores, across institutions. Learning Gains in the areas of critical thinking and writing are reported on our College Portrait.  Reporting Learning Gains from a test like the ETS-PP is a requirement of participation in the Voluntary System of Accountability.

 

What We Did 

 

In fall 2010, we offered the online unproctored version of the ETS-PP to about 800 students enrolled in courses that are typically taken by first year students. In spring 2011, we offered the same test to over 900 students enrolled in senior capstone courses. In almost all cases, instructors of those courses offered grade incentives to students to complete the test, either in the form required or bonus points. We provided instructors with sample syllabus language (approved by UNF’s IRB), and we repeatedly offered instructors a class presentation to explain the test to their students. We emailed the students directly about how to access the test itself, providing them with a web link and instructions. Students had a two week window for test completion, during which they received multiple reminders.

 

Results 

 

The key results are as follows:

  • The great news: UNF’s Learning Gain is “well above expected” (the highest possible score) for critical thinking, and “above expected” for writing.
  • Additional great news: UNF’s Learning Gains in both critical thinking and writing are higher than those of 90% of our Carnegie peers who participated in the test.
  • About 95% of students enrolled in eligible first year courses completed the test; and about 75% of students enrolled in eligible senior capstone courses completed the test.
  • Our students, both first year and senior, scored higher than the national norms for institutions of our type (Masters [Comprehensive] Colleges and Universities I & II, in the old Carnegie classification).
  • Performance of our first year students varied widely by whether they were enrolled in Jacksonville Commitment, the Honors Program, Venture Studies First Year Seminars, or introductory English composition courses.
  • Performance of our seniors varied widely by college; possible explanations include proficiency definitions, student self-selection, specific selection of courses within colleges that participated, or differences in curriculum between colleges. 

Things to Know in Interpreting the Results  

  • Learning Gains in the areas of critical thinking and writing are computed by administering the same test to incoming first year students and graduating seniors,  normalizing the scores of the first year and graduating students to their entering SAT scores, and computing a relative “Learning Gain” in each area.
  • A number of students who were enrolled in first year or senior courses were not, in fact, first year or senior students, and were not included in the calculation of Learning Gains.
  • Transfer students are not included in the Learning Gains calculation.
  • ETS defines proficiency classifications that are tested at each level.  

Where We Go From Here 

  • As a condition of participation in the Voluntary System of Accountability, UNF must update its learning gains scores at least once every three years. We anticipate administering the ETS-PP again in 2014-2015.

  • As a first point for attention, we aspire to have a majority of our graduating seniors be proficient in writing and critical thinking. Conversations are currently underway to determine how UNF can improve its scores. Specifically, we are discussing how the definitions of writing and critical thinking in the Academic Learning Compacts compare to the ETS definitions, and whether it would be appropriate to bring more consistency to definitions of writing and critical thinking competencies across units.