DESIGNING A UNIT ASSESSMENT SYSTEM
One Institution’s Quest to Design a Unit Assessment System:
Operationalizing What We Believe About Assessment of Teacher Candidates
Larry G. Daniel, Cathy A. O’Farrell, & Terence Cavanaugh
University of North Florida
Paper presented at the annual meeting of the American Association of Colleges for Teacher Education, February 7-10, 2004, Chicago, IL.
The purpose of the present paper is to present work done to date in developing and implementing a teacher education unit assessment system in the College of Education and Human Services at the University of North Florida. Specifically, we describe the usefulness of the system in informing teacher educators about the quality of teacher education candidates, developing plans for remediation of candidates and improvement of programs, and making decisions about the operation of the teacher education unit.
Institution’s Quest to Design a Unit Assessment System:
With the release of its NCATE 2000 Standards, the National Council for Accreditation of Teacher Education imposed the expectation (Standard 2) that each institution seeking accreditation would develop a unit assessment system “that collects and analyzes data on applicant qualifications, candidate and graduate performance, and unit operations to evaluate and improve the unit and its programs” (NCATE, 2002, p. 21). As outlined by NCATE, the assessment system should reflect the unit’s conceptual framework and incorporate candidate proficiencies per professional and state standards. To aid in the development of unit assessment systems, NCATE created a five-year “transition plan” for phasing in the system. By the time of the 2004 AACTE annual meeting, institutions should be in year four of implementation of their assessment system. For institutions undergoing NCATE review in 2003-2004, assessments and criteria/rubrics for scoring each assessment should be fully developed, the assessment system should be fully operational, data from the assessments should be collected, and data analysis should be in progress.
Fulfilling the Standard 2 expectations for unit assessment systems has been a major challenge for institutions undergoing initial or continuing accreditation review since implementation of the NCATE 2000 standards. Additionally, many states have created their own expectations for teacher education accountability compliance, leaving institutions with the responsibility of simultaneously implementing assessment procedures mandated by both state agencies and NCATE. In light of the challenges of addressing Standard 2 expectations, it is important that institutions work effectively together to share their ideas, challenges, and successes in developing their unit assessment systems.
Teacher education program and curricula have become increasingly aligned with state and professional standards and benchmarks for teacher and student performance (Ambach, 1996; Holbein, 1998; Weisenbach, 2000). Concomitantly, professional accrediting bodies have raised standards and implemented assessment measures and procedures for assuring teacher candidate proficiency vis-a-vis these standards (e.g., NCATE, 2002). In many instances, the ability to comply with the many assessment procedures mandated by these various agencies and organizations creates a daunting task for those endeavoring to track data on candidate outcomes (Tomei, 2002).
Teacher education assessment systems must reflect the complexities of teaching. Teachers must “go far beyond dispensing information, giving a test, and assigning a grade. . . . They must understand how students thing as well as what they know in order to create experiences that produce learning” (National Commission on Teaching & America’s Future, 1996, p. 8). Although traditional measures such as class examinations and standardized teacher tests have their place in these systems, it is important that teacher candidates be assessed on tasks related to teacher competencies related to effective teaching and learning. Compiling of work samples from teacher candidates is one important way that teacher educators can document candidate competencies consistent with standards (Fredman, 2002). These “teacher work samples” (Schalock, 1998) are realistic measures of important teaching outcomes in that they require
teacher education candidates to document their ability to plan, deliver, and assess a standards-based instructional sequence, to profile student learning that occurred during the instructional sequence, and then to reflect on student learning in order to improve teaching practice. . . . Candidates [must] show how they have adapted their instruction to meet the needs of all students and to demonstrate the impacts of their teaching on student learning. (Denner, Salzman, & Harris, 2002, p. 3).
Individual teacher work samples may be compiled into a broader assessment system via development of a teacher candidate portfolio (Campbell, Cignetti, Melenyzer, Nettles, & Wyman, 2000), or the samples may be tied together more loosely via a set of courses within a teacher education program in which the work samples are progressively linked to form a continuum of documentation of the teacher candidates’ learning (Harris, Salzman, Frantz, Newsome, & Martin, 2000). Within either organizational schema, it is the teacher education faculty’s responsibility to develop rubrics or scoring guides to rate the work samples. As Denner et al. (2002) have noted, in a high-accountability system, it is important that the scoring guides/rubrics be aligned with the language of the standards by which candidate performance is to be assessed.
Beyond tracking of candidate work samples, the teacher education unit assessment system should (a) maintain data from summative candidate performance measures and graduate and employer follow-up surveys; (b) utilize information technology to store, retrieve, and generate summaries of unit data; (c) provide information useful for planning remediation activities for students not meeting standards; and (d) define specific decision points at which decisions are made about candidates (NCATE, 2002). Once an institution develops a comprehensive unit assessment system, the system can yield useful data “to build a convincing case for the quality of their programs. . .[and] for monitoring and improving those programs and for advising students (Scannell, 2000, p. 4).
The purpose of the present paper is to present work done to date in developing and implementing a teacher education unit assessment system in the College of Education and Human Services at the University of North Florida (UNF). Specifically, we describe the usefulness of the system in:
1. informing teacher educators about the quality of teacher education candidates in terms of professional and state standards, including benchmarks for assuring candidate performance in terms of the appropriate knowledge, skills, and dispositions needed by candidates to positively impact P-12 student learning;
2. developing plans for remediation of candidates and improvement of programs; and
3. making decisions about the operation of the teacher education unit.
It is our desire that our work will provide ideas for other institutions seeking to develop or modify their unit assessment systems.
UNF is one of 12 state supported universities in the state of Florida. The College of Education and Human Services (COEHS) is the academic unit with responsibility for all educator preparation programs at UNF. The College awards three degrees. At the undergraduate level the College awards the Bachelor of Arts in Education (BAE) degree. Graduate degrees awarded are the Master of Education (MED) and the Doctor of Education (EdD).
The Florida Legislature and Florida Department of Education have developed statutory requirements and rules relative to the approval of teacher education programs in the state. Students from all programs must meet state requirements on a variety of teacher competency measures, including mastery of general studies curriculum (as measured by GPA of 2.5 or higher and acceptable scores on the College Level Academic Skills Test) at point of entry as well as mastery of professional knowledge and subject area knowledge (as measured by scores on the Florida Teacher Certification Exam subtests) upon program completion. In addition, students must demonstrate mastery in the Florida Educator Accomplished Practices, a set of general competencies required of all teachers in the state. Specifically, the 12 practices include competencies of teacher performance in the areas of Assessment, Communication, Continuous Improvement, Critical Thinking, Diversity, Ethics, Human Development and Learning, Knowledge of Subject Matter, Learning Environments, Planning, Role of the Teacher, and Technology.
To gain and maintain approval by the state, teacher education programs in Florida must be able to provide evidence that its program completers have demonstrated proficiency in each Accomplished Practice. In addressing this state requirement, the teacher education programs at UNF have developed a series of “critical task” assignments that exemplify each practice. Faculty who teach courses that include critical tasks assess student proficiency on each one and record these data for inclusion in the unit assessment system’s electronic database. These data serve to inform the College and program decision makers about individual candidates’ progress and provide useful feedback to program faculty about the efficacy of the program curriculum and activities.
Per NCATE Standard 2, the unit assessment system should include time lines for data collection and analysis related to candidate performance and unit operation. Development of the UNF assessment system involved creating of assessment tools, designing of the system, and making of decisions about ways to utilize the system data. Figure 1 illustrates the design of the system. The system is useful in that it links performance of candidates to the unit’s conceptual framework, national and state standards, professional organizational standards and directives, and K-12 student learning.
The unit assesses candidate performance and evaluates all educator preparation programs on a continuous basis. In addition to assessing whether candidates have developed competencies needed to meet unit standards, the process provides an empirical basis for evaluating and continuously improving the unit’s educator preparation programs. The assessment system also includes a unit evaluation process, which provides a basis for incrementally improving unit operations.
Figure 2 illustrates the process of candidate assessment across the span of a program of study, from the point of admission to the point of program completion. Program faculty assess candidate content and professional and pedagogical knowledge, skills, and dispositions through course-based assessments, as well as through decision-point assessments. Data from these assessments are used to make decisions about candidate performance at the pre-admission, developmental, and program completion stages. As candidates progress through the educator preparation programs, they are expected to demonstrate increasingly higher levels of knowledge, skills, and dispositions as identified in the unit’s conceptual framework and program knowledge bases. As feedback is given to candidates following assessments, growth is expected in the candidate’s planning and delivery of instruction. The feedback given to candidates includes a review of strengths observed, concerns, and specific suggestions for developing knowledge, skills, and dispositions relative to professional and unit standards.
Decision Point Assessments
In addition to course-level assessments, the candidate assessment process for each program includes decision point assessments that occur at the pre-admission (program entry), developmental, and program completion stages. These decision point assessments are used to determine whether the candidate meets the standards required to enter the program, continue within in the program, and complete their program. At the pre-admission stage, information on candidate potential is examined. A candidate is not admitted to a particular program unless he/she meets that program’s entry requirements. At the developmental level, faculty assess candidates’ progress in developing the necessary knowledge, skills, and dispositions about the Florida Teacher Accomplished Practices. This information assists faculty and candidates in making decisions about candidates’ developmental needs and about their continuation within the program. Developmental or remediation opportunities will be provided for candidates who exhibit deficiencies, but candidates who still cannot satisfactorily meet the specified standard(s) at the intermediate level will not be permitted to move to the next level of the program. At the program completion stage, faculty use the collected data to assess candidates’ readiness for completing the program and assuming professional responsibilities. Intermediate and program completion assessment criteria are aligned with SPA standards, the Florida Teacher Accomplished Practices and with the unit’s conceptual framework to ensure that these assessments consistently measure candidates’ performance in relation to professional and unit standards. Program critical tasks and assessment instruments will be reviewed for fairness, consistency, accuracy, and avoidance of bias. Candidates who cannot, even with developmental opportunities, meet program completion requirements will not be granted a degree in the program.
Program Entry Assessments
Program admission requirements assess the quality and potential of candidates at the entry of educator preparation programs. Minimum required GPAs exist for initial undergraduate and graduate admissions. Assessments used for teacher education program entry include standardized test results such as CLAST or Praxis I for the initial level and Graduate Record Examination (GRE) for the graduate level.
Intermediate and Program Completion Assessments
At the intermediate level, candidates’ progress in developing the necessary knowledge, skills, and dispositions is assessed in order to make decisions about their developmental needs and, therefore, their continuation in the program. Program completion assessments are used to evaluate candidates’ growth in the knowledge, skills, and dispositions identified by the program and, therefore, their potential for assuming professional responsibilities.
Assessments used at the intermediate and program completion stages include portfolios, which are used in all programs; written essays and journals, which provide evidence of candidate reflection; videotapes, which are used to assess initial candidates’ instructional skills prior to student teaching and to assess counseling skills of counselor education candidates; observations by faculty, which are used in assessing teaching performance; ratings by clinical instructors, which are used to assess teaching performance demonstrated during field experiences; course evaluations; and course and program projects. Candidates’ demonstration of the knowledge, skills, and dispositions needed to impact K-12 student learning is an important component of each program’s assessment procedure.
Candidate Tracking Electronic Database
The unit has developed and is now refining a technology based database to better facilitate the program and candidate evaluation. The electronic database is focused around the 12 Educator Accomplished Practices and consists of all program/track critical assessments identified for courses within each program course objectives and related. Each program critical task or objective is keyed to the Accomplished Practice to which the objective and assessment are most directly related. The database also includes fields showing the type of learning addressed by each objective (knowledge/awareness, skills, or dispositions) as well as the general content and form of the assessment, including reference to the scoring tools and in the future the relevant specialized professional association (SPA) standards with associated objectives.
This database was constructed under the guidance of a college technology committee and was developed with the cooperation of program groups and unit administration. The database houses information about the student’s abilities at various levels: entry, developmental, and program completion. At the entry level, the database has information concerning the student’s entry GPA, required entry tests, and previous schooling. The exit information includes the professional teacher exams (FTCE), completion of required courses and portfolio items, successful completion of internship with associated portfolio, exit GPA, and program completion eligibility. The bulk of the database is occurring at the development level, with the focus of the database being on the Twelve Teacher Accomplished Practices as determined by the Florida Department of Education: Assessment, Communication, Continuous Improvement, Critical Thinking, Diversity, Ethics, Human Development and Learning, Knowledge of Subject Matter, Learning Environments, Planning, Role of the Teacher, and Technology.
Each of the Accomplished Practices is repeatedly checked by the programs concerning levels of ability (knowledge, development, accomplished) and is evaluated by instructors and programs in critical task and portfolio development (unsatisfactory, satisfactory, exemplary). Critical task and portfolio items are developed and checked within program courses, preinternship field labs, and again during student teaching internship. Faculty report to the Office of the Dean what critical tasks will be included in the course at the beginning of each term, assess each student on any critical tasks included in a given course, and complete a report of student progress to be entered into the database (see Figures 3 and 4). The database provides a comprehensive method for documentation from various divisions within the unit and is designed to allow for access by faculty with query and report tools and a separate access being designed for students to track their progress.
An additional component of the database is a remediation tracking section. In this section, students who have been identified as “unsatisfactory” in their work concerning critical tasks or portfolio items will, together with a supervising instructor, develop a remediation plan (see Figure 5) concerning how the student will achieve satisfactory accomplishment of the critical task.
Curriculum Alignment and Refinement Process
Using this database, faculty members within each program are able to examine the alignment of their curricula and critical tasks with specialized professional associations (SPAs) and unit standards. In addition, faculty can analyze the curriculum in total by examining the program’s focus on knowledge, skills, and dispositions as developed by the students within the programs and documented in their portfolios; by reviewing the various types of assessments used in the curriculum; and by examining the extent of a program’s efforts to prepare professionals who are competent and reflective. Curriculum audit and refinement is an ongoing process designed to ensure that the curriculum has the capacity to produce the desired outcomes, and the database will be a tool which will support that kind of program and unit evaluation. Elements of this process for each program include (a) a review of the alignment of each program track with the intended outcomes, and (b) identification of curriculum changes or additions needed to produce the desired outcomes.
The College’s technology committee and database designers have collaborated with program faculty and college and unit administration to ensure that this information is appropriate and focused on criteria derived from the unit’s conceptual framework. In an initial curriculum audit, unit personnel, by program, reviewed the alignment of each program curriculum with the unit’s outcome standards, then programs made course adjustment recommendations and identified alignments between courses and accomplished practices. Programs then developed critical tasks from their program track which will be required for inclusion in student portfolios.
Assessment Validation Process
The validation process is being designed to improve assessment tools and procedures used in all programs to ensure fairness, consistency, accuracy, and avoidance of bias. The assessment validation process also guides faculty in ensuring validity and alignment of course assessments. Priority is given to ensuring the quality and credibility of assessment tools and critical tasks used at major candidate decision points (entry, developmental, and program completion).
The validation process begins with the identification of each program’s critical tasks and development of associated rubrics. Next will be a general assessment of the assessments. The goal of this process is to establish the reliability and validity of all key assessments and establish the predictive validity entry assessments toward performance within the program and in professional positions.
Use of Candidate Assessment Data
At the individual candidate level, information on the candidate’s performance at each decision point will assist the candidate in assessing his or her level of performance in relation to program standards. Faculty review data on each candidate’s performance to determine whether he or she meets criteria for admission to, continuation in, and completion of the program. Many of these criteria currently are used in decision-making at the individual candidate level.
In addition to providing information on individual candidate performance, the design of the database also supports the creation of standard and custom reports for use in evaluation at the program and unit levels, thereby permitting aggregation of these data for use in identifying program strengths and weaknesses. This aggregation of candidate performance data is combined with other unit internal data, such as summaries of candidate complaints and their resolution and external data, first year principal evaluations, for purposes of program evaluation and improvement. UNF’s model for program and unit evaluation is presented in Figure 6.
To thoroughly review each program on an annual basis, program faculty and department chairs examine findings developed through curriculum alignment audits, as well as aggregated internal data on candidate competencies and information from external sources, such as follow-up studies, candidate performance on licensure examinations, employer reports, and state program reviews. Central to this process is a review of aggregated candidate data collected at the pre-admission stage (number and qualifications of all applicants by admission status) and at the intermediate and completion stages (including number of program graduates and graduation rates). Faculty and department chairs also will use results of periodic research studies conducted by the unit using a value-added approach to compare K-12 student achievement gains of classes taught by (and schools administered by) unit graduates with statewide average gains.
This program evaluation process results in development of an action agenda for revising the program curriculum (see curriculum alignment audit below), for improving instruction, for revising field experiences, and for redesigning other components of the program to promote high levels of performance by all candidates. Program-level action agendas are developed by program faculty and the department chair with input from the professional community.
In addition to reviewing program curricula using the curriculum audit process, each program conducts a comprehensive program review on an annual basis. Data analyzed in this review include reports presenting aggregated summaries of candidate strengths and weaknesses compiled using the candidate decision point data described above. Program reviews also are based on summaries of candidate complaints and their resolution.
The program evaluation process includes the collection and examination of external data assessing program performance. Data are collected from state licensure examinations, surveys of principals supervising program graduates, follow-up surveys of graduates, research on student outcomes, and other external sources. Faculty and school partners associated with each program then develop a program improvement plan with a timetable for implementation. This plan guides faculty and school partners in collaboratively working toward improved program outcomes.
Just as candidate data are aggregated for use in evaluating programs, comprehensive analyses of program strengths and weaknesses are aggregated for use in evaluating the unit’s effectiveness. In the unit evaluation process, a unit evaluation committee uses aggregated analyses generated through analysis of evaluation data from all programs. The unit evaluation process also includes analysis of program recruitment and retention data, as well as unit-wide data in the form of faculty evaluations; information on student, staff, and faculty diversity; and unit leadership assessments to identify changes needed to improve unit performance.
Ambach, G. (1996). Standards for teachers: Potential for improving practice. Phi Delta Kappan, 78(3), 207-210.
Campbell, D., Cignetti, P., Melenyzer, Nettles, & Wyman, R. (2000). Portfolio and performance assessment in teacher education. Boston: Allyn and Bacon.
Denner, P. R., Salzman, S. A., & Harris, L. B. (2002, February). Teacher work sample assessment: An accountability method that moves beyond teacher testing to the impact of teacher performance and student learning. Paper presented at the annual meeting of the American Association of Colleges for Teacher Education, New York.
Fredman, T. (2002, February). The TWSM: An essential component in the assessment of teacher performance and student learning. Paper presented at the annual meeting of the American Association of Colleges for Teacher Education, New York. (ERIC Document Reproduction Service No. ED464046).
Harris, L. B., Salzman, S., Frantz, A., Newsome, J., & Martin, M. (2000, February). Using accountability measures in the preparation of preservice teachers to make a difference in the learning of all students. Paper presented at the annual meeting of the American Association of Colleges for Teacher Education, Chicago. (ERIC Document Reproduction Service No. ED440926)
Holbein, M. F. (1998). Will standards improve student achievement? Education, 188, 559-564.
National Commission on Teaching & America’s Future. (1996). What matters most: Teaching for America’s future. Report of the National Commission on Teaching & America’s Future. New York: Author.
National Council for Accreditation of Teacher Education (2002). Professional standards for the accreditation of schools, colleges, and departments of education (2002 ed.). Washington, DC: Author.
Scannell, D. P. (2000). Aggregating data for NCATE 2000. Retrieved May 28, 2003, from http://www.ncate.org/resources/commissioned%20papers/dscannell.pdf.
Schalock, M. (1998). Accountability, student learning, and the preparation and licensure of teachers: Oregon’s teacher work sample methodology. Journal of Personnel Evaluation in Education, 12, 269-285.
Tomei, L. J. (2002, February). Negotiating the standards maze: A model for teacher education programs: White paper. Paper presented at the annual meeting of the American Association of Colleges for Teacher Education, New York. (ERIC Document Reproduction Service No. ED 463263)
Weisenbach, E. L. (2002). Myth 2: There is no connection between standards and the assessment of beginning teachers. In G. Morine-Dershimer & G. Huffman-Joley (Eds.), Dispelling myths about teacher education (pp. 25-32). Washington, DC: American Association of Colleges for Teacher Education.
Figure 1: The University of North Florida Unit Assessment System Change
Figure 2: Assessment of Candidates
Figure 3: Critical Task Class Reporting Form
Figure 4: Faculty Critical Task Verification Form
Figure 5: Remediation Plan Form
Figure 6: UNF’s COEHS Program and Unit Evaluation Model