Session 3: Overview

Structures and Processes Essential to the Development
Of a Performance Assessment and Unit Accountability System:
What Six Renaissance Partnership Institutions Have Learned in
Their Quest to Address NCATE Standard II

SECTION I - CONTENT
This symposium will feature the results and lessons learned at six Renaissance
Partnership Project institutions that have worked together and helped each other
develop unit accountability and performance data management systems. These
systems address NCATE Standards and show evidence of teacher graduate impact on
P-12 student learning.

Background and Context
In their fall conference of 1997, presidents, provosts and deans of education of the
Renaissance Group launched an initiative to become more accountable for the impact
of their graduates on P-12 student learning. This effort was prompted by the standards
and principles proposed by the New Teacher Assessment and Support Consortium
(1992), the report of the National Commission for Teaching and America's Future
(1996), drafts of NCATE standards then being proposed for Year 2000 (NCATE 2000)
and new demands from state and local constituency groups requiring higher test scores
for all P-12 students.

Beginning in 1998, representatives often Renaissance Group institutions began
meeting twice each year to plan specific strategies to achieve their accountability goal.
A study was conducted by Pankratz and McLean (1998) to assess the potential and
feasibility to collect and report performance data. While most institutions were collecting
traditional test scores, course grades and student teaching evaluators, performance
data that could be linked to P-12 student learning was rare.

In 1999, the ten institutions received a $5.8 million five-year Title II Partnership grant to
develop and implement key strategies that would enable them to show evidence that
their graduates could facilitate learning of all P-12 students. These included: (1) the
development of accountability data management systems; (2) use of teacher work
samples to report teacher performance; and (3) mentoring teams consisting of teacher
educators, arts and science faculty and school practitioners that developed shared
accountability for teacher performance.

While the development of teacher work samples as an assessment tool for performance
of teacher candidates has moved ahead as the driving force for improvement of teacher
quality, the development of accountability/data management systems has gone at a
slower pace. Over the past two years, three institutions have led the charge and have
developed electronic performance data management systems that have served as
exemplars and prototypes for other institutions in the five-year Title II project.

Also, over the past two years processes of establishing validity based on content
representative (i.e., Realism, Criticality, Necessity and Frequency [Crocker, 1997] and
Inter-Rater Reliability [Shavelson & Webb, 1991]) for teacher work samples has
heightened the need to collect, store, analyze and report performance data on teacher
candidates and graduates. Thus, the building of accountability/data management
systems in Renaissance Partnership institutions has been a slow but deliberate
development process with lead institutions helping other institutions and providing
mutual support. This symposium is designed to tell that story and share the process
with other teacher educators.

Statement of the Problem
The ten Renaissance Group institutions studied by Pankratz and McLean (1998) like
most teacher preparation institutions. They will require a major development effort to
meet are not unlike NCATE Standard II (2000) Assessment System and Unit
Evaluation:

The unit has an assessment system that collects and analyzes data on applicant
qualifications, candidate and graduate performance and units of operation to
evaluate and improve the unit and its program.

Few have fully developed systems of unit accountability and data management. What
are the barriers to the development of systems that meet NCATE Standard II
requirements? What are successful models? How can teacher preparation institutions
work together to overcome barriers and develop systems of performance accountability
that work for them and meet accreditation requirements? These are the
questions/problems this symposium will attempt to answer.

Content of the Symposium
One dean, two associate deans and three teacher education faculty each representing
their respective institutions will share their stories and struggles to develop unit
accountability/data management systems. This will be done in a unique manner. First,
the six presenters from six institutions will collaborate to develop one paper that
communicates to the teacher education community their individual and collaborative
efforts to develop accountability systems. Second, the six presenters will collaborate to
develop a 15-20 minute PowerPoint presentation that shows the structures and
processes common to all institutions and the featured components that are strengths at
each respective institution. All six institutions represented are state supported teacher
preparation institutions that range in size from graduating 200 teachers per year to over
1,000 teachers per year.

All Renaissance Partnership institutions have agreed to collect and report performance
data at four or more points in the development of teacher candidates; admission to the
program, pre-student teaching, exit from the program and in the first year of teaching.
Common data on candidate characteristics, behaviors and performance relative to
teaching standards are collected, analyzed and reported at all institutions.

Over the past year, Renaissance Partnership institutions have identified five essential
elements of a functional unit accountability/data management system. These are:

an institutional commitment to the systematic collection, analysis, use and reporting
of performance data;
the designation and support of an assessment and accountability coordinator
(usually at least half-time);
an ongoing process of development and maintenance of a system of performance
assessments;
alignment with state and INTASC Standards
indicators of performance
clear structured teaching performance tasks (prompts)
scoring rubrics for all standards and performance
benchmarking of levels of performance
maintenance of scorer reliability across programs
an electronic data management system designed for existing and future data
management and reporting needs; and
a systematic and regular process analyzing and reporting performance data on
candidates, courses, faculty and programs to different audiences (i.e., candidates,
graduates, faculty, accrediting agencies and the public).

Both the collaboratively-developed paper and the summary PowerPoint presentation will
show the common key concepts, elements and processes of the accountability systems
and the common processes of development of systems at each institution. However,
the unique features and strengths of each institution will also be presented.

A second and most important part of the content of the symposium will be strategies for
overcoming barriers to developing assessment systems, unit evaluation and
accountability systems. While successful strategies for development and overcoming
barriers will be included in the jointly-prepared paper and the PowerPoint presentation,
the most helpful and specific information about overcoming barriers will be shared and
discussed with colleagues in a panel discussion of the six institution representatives.

Significance of this Symposium
First, the context and processes to be shared in the proposed symposium are very
timely and of utmost importance to every teacher education unit that must be
accountable for performance to state and national accrediting agencies. Second, the
content of this symposium contains successful strategies for breaking through some of
the old barriers to developing and supporting the collection and use of performance data
for program improvement and accountability. Third, the accountability systems
developed in the Renaissance Project provide data on teacher candidate impact on
student learning which is also a new requirement for NCATE. And, finally, this
symposium will show how institutions working together can do what they could not have
done individually without mutual support.

Conclusion
Developing sound performance assessment and accountability systems that can be
used for program improvement and unit evaluation is very important but costly in terms
of startup and operational resources. Also, connecting those who collect and report the
data and those who use the data is unnatural and often difficult. (However, with a
sound conceptual base, some new structures and strategies and local leadership user-
friendly systems that are valid, reliable and possible.) Collaboration with other
institutions and teacher education colleagues is one of the strategies highly
recommended.

SECTION II - PARTICIPANT OUTCOMES / METHODS

Outcomes/Expectations
Participants attending this symposium will have the opportunity to:

learn about the essential elements of unit accountability/data management systems
that address NCATE Standard II;
identify and discuss the barriers to developing and supporting sound data
management systems that can be used for program improvement and
accountability;
evaluate structures and strategies that could be used back home to design and
develop unit accountability/data management systems;
learn about strategies and techniques for collecting data that provide evidence of
teacher candidate impact on P-12 student learning; and
learn about strategies for collaborating with colleagues and networking with the
Renaissance Partnership.

Methods of Presentation
After a brief introduction to the panel of presenters and the purpose of the symposium,
one of the institution representatives will show the 15-20 minute PowerPoint
presentation that summarizes the key concepts, structures, strategies and results of the
Renaissance Partnership accountability effort in the six institutions. This will be
followed by a 20-minute panel discussion by the six institution representatives (one
dean, two associate deans and three teacher education faculty) with respect to the
challenges of developing and sustaining comprehensive data management systems.
Panel members from each institution will be asked to focus on what worked for them
and what has been most successful. After the panel discussion, audience participants
will have about 30 minutes to ask questions, evaluate the Renaissance Partnership
accountability effort and describe what assistance and resources the Renaissance
Partnership Project might provide other institutions interested in developing
accountability/data management systems.

As a final activity of the symposium, a discussant who is a member of the NCATE staff
and has the responsibility for accountability systems will summarize and critique the
work of the Renaissance Partnership and the symposium.

References

Crocker, L. (1997). Assessing content representativeness of performance assessment
       exercises. Applied Measure in Education. 10. 83-95.
Interstate New Teacher Assessment and Support Consortium. (1992). Model standards
       for beginning teacher licensing and development: A resource for state dialogue.
       Washington, DC: Council of Chief State School Officers.
National Commission on Teaching and America's. (1996). What matters most:
       Teaching and America's Future. New York: Teachers College Press.
National Council for Accreditation of Teacher Education. (2000). NCATE 2000 unit
       standards. Washington, DC: Author.
Pankratz, R., & McLean, J. (1998). A study of the potential for measuring the impact of
       Renaissance Group graduates on the students they teach. Unpublished report to
       the Renaissance Group.
Shavelson, R. J., &Webb, N.M. (1991). Generalizability theory: A primer. Newbury
       Park, CA: Sage.