AN EVALUATION OF THE IMPACT OF
THE WATERWAYS/TEN PENNY PLAYERS PROJECT
ON STUDENTS 'AT RISK'
IN TERMS OF KNOWLEDGE, SKILLS
AND ATTITUDES WITH SPECIFIC REFERENCE TO
OUTCOMES AS RELATED TO
THE NEW YORK STATE LEARNING STANDARDS


SEPTEMBER, 1998 - JUNE, 1999

Waterways Project/Ten Penny Players Staff
Richard Alan Spiegel, Co-Director, Administrator/Poet/Publication & Web Site Designer
Barbara Fisher, Co-Director, Administrator/Writer/Illustrator/Printer
Thomas Perry, Photographic and Videographic Documentation
Molly Barker, Visual Artist who Teaches

Consulting Artists, 1998-1999
Lance Axt, actor/director/playwright; Bread & Puppet Theatre; Magie Dominic, visual artist; Desirae Foston, visual artist; Magdalena Gomez, performance/poet; Great Small Works, puppeteers - Susie Dennison, Michael Romanyshyn, Roberto Rossi, Mark Sussman; Helen Grosso, distance technology; Matt Jones, folk musician; Barry Joseph, web design/documentation; Donald Lev, poet; LRW. Net, computer/internet technology - Dr. Lucia Ruedenberg Wright, Randy Wright; Ellen Lytle, poet; Michael Mao Dance; Dennis Nurkse, poet; Louis Reyes Rivera, poet; Ray Shah, computer/internet graphic design; Frank Stearns, photographer; Thelma Thomas, story teller; Pamela White, visual artist; Alison Zadro, actor/director

Frederick Douglass Literacy Center
Lois Rekosh Goldberg, Program Director; Janet Mann, Center Administrator; participating teachers: Bob Budnick, Salvatore Canale, Mary Ann Cornell, Donna Farnum, John Georges, Roslyn Kaye, Myrtle Liburd, Oneita Murray, S. Dolores Russo, Dr. Madeline Slovenz-Low, Gail Tuch, Tyona Washington; guidance counselors: Jeanne Kotuby and Susan Salkin

NYC Vocational Training Center
Alan Werner, Principal; Loretta White, Assistant Principal; participating teachers: Norman Benjamin, Charles Brandwein, David Casey, Chantal Firstman, Jack Giordano, George Goldschlag, Maxwell Mendes, Nina Morris-Farber, Moli Ntuli, Kisoonlall Ramkaran, Paul Rotondo

Education Consultants Norma Gluck, Regent Emeritus & Associates

  • 1.0 Introduction/Background:
    While we believe that the creative work (ie. chap books, poems, puppets, drawings etc.) produced by the students participating in the Waterways Project is important, and that discussing and describing these are a necessary part of this evaluation, we have come to understand that this approach alone is not sufficient to give a complete picture of the achievements of the students and the Project. This evaluation reflects the introduction of a Structural-Functional Analysis Evaluation Research Design. This design was developed by our Education Consultant whom we engaged to work with us over this past year. The design allowed us to prospectively and longitudinally measure student progress during their exposure to our project over the course of the academic year 1998-99, both qualitatively and quantitatively. The proto-type was introduced in September, 1998 and was used and modified during the year to reflect feedback from administrators (principals, assistant principals, program directors), teachers, guidance counselors and Waterways Project's artists and technology experts who used the evaluation forms over the course of the year. In this way, we were able not only to use this new research design, but to evaluate its methodology and the research instruments while we were in the process of using them.
  • 2.0 Design of the Evaluation ResearchIn using this methodology, one looks at three phases within the year's program: the Structure, the Process and the Outcome. Each stage is measured by instruments and/or guides specifically created by or for the Waterways Project and each has specific objectives.
  • 2.1 The Structure Phase: Structure represents a baseline - all of the components which form the underpinnings or plan for this program. It included:
  • the planning of meetings with staff;
  • the staff development that would take place over the year;
  • an analysis of the 28 New York State Learning Standards and use of guides for teachers' and administrators' use;
  • the testing of a prototype of an Observer Evaluation Form for use during classroom sessions led by the visiting artist;
  • a list of questions to be used as a guide for data collection;
  • 2.2 The Process Phase: What happens during this program? How is it carried out?
  • the actual staff development workshops and meetings;
  • a classroom observation sheet for the collection of qualitative data during this phase
    the collection of both qualitative and quantitative data to be used as a basis in evaluating the Outcomes of this project;
  • the work done in helping the students to write poetry for inclusion in the chapbooks;
  • the work done in developing the on-line technology programs;
  • any testing done over the course of the year;
  • the keeping of records of student attendance and progress in various areas, including results of literacy testing.
  • 2.3 The Outcome Phase: Did you accomplish what you set out to do? How are you going to show this? The Outcomes are measured in terms of qualitative and quantitative data.
  • Qualitative data such as:
  • students' creative work (chap books, poetry, dance presentations, etc.)
  • production of videos showing aspects of the program;
  • verbal and personal evaluations given by teachers and administrators, artists, and students as to what this project meant to them and what they felt they learned.
  • Quantitative measures include:
  • attendance records;
  • academic records, i.e. gains in literacy as reflected on testing;
  • graduation records;
  • reports of which of the students in the Waterways Project received GED's, moved to other programs, received special awards at honors assemblies;
  • the Classroom Observation Forms completed by teacher-observers during artist sessions describing what was done and showing how this relates to the New York State Learning Standards.
  • 3.0 Results
    The data collected during this year, using the Structural-Functional analysis reflects that the Waterways Project has had a positive impact upon students, teachers, administrators and artists.
  • 3.1 Student Outcomes
    Data from the Classroom Observation forms shows the development, over the year, of student learning, confidence, engagement, creativity, motivation and participation. This is reflected in teachers' evaluative observations. Teachers completed classroom observation forms during different parts of the year reflecting what they saw with relation to their students' growth and learning over the year. There were a total of 352 comments, with 13 additional, personal remarks. These comments were reviewed and analyzed and provided extremely important and ongoing feedback to the Project over the year. Their observations were full, detailed and positive in nature. They described students' growth in vocabulary understanding and usage, growth in reading and writing, growing ability to use computers for writing their poems, stories and plays, understanding of technical aspects of computer usage, increased reading and writing, more acceptance and understanding of fellow students, creativity in expression and translating this into writing by some students who have had little or no formal instruction in English or writing. Teachers can immediately see the relationship of all the comments they write on the Observation Forms to specific New York State Learning Standards (i.e. those in English Language Arts, the Arts, Math, Science and Technology, Career Skills, Social Studies, etc.)
    378 students participated in the program over the l998/99 school year. 15 anthologies were published; 29 students created their own books of poetry/expressive writing; 98 students published picture books; 18 students published books containing their photographic art. Eight VTC students performed with the Bread and Puppet Theatre at Haym Salomon Nursing Home; 10 FDLC students performed with the Bread and Puppet Theatre at FDLC; 12 FDLC students created puppets and performed with Great Small Works puppet company in performance at FDLC; 10 FDLC students performed poetry and presented their illustrated books to patients at the Concord Baptist Nursing Home in Brooklyn. Students from four classes from FDLC performed their writing at two school wide assemblies; nine FDLC students designed, wrote and used their own digital photographs to illustrate Journeys from Home to School on the FDLC web site; students from all the FDLC classes had an opportunity to begin posting their writing on STREAMS On-Line for comment by teachers, artists and peers; all the published FDLC students exhibited their books in the school hallways and in an art show mounted in the school's cafeteria. Two VTC students represented both schools at a state wide service learning conference where they spoke and performed their poetry to attendees. VTC students attended the monthly professional development workshops held for teachers and artists. VTC and FDLC students participated at the Waterways Project's May arts and technology conference held for teachers. In May VTC and FDLC students performed poetry as well at the Donnell Library Center and at Prospect Park where a group of VTC students also gave a dance concert. Selected examples of student publications are available to researchers.
    The attendance records of students in classes in which they participated in the Waterways Project reflect a somewhat higher attendance rate than those students not in the Project. The range is .8% to 1.5% higher. The FDLC literacy statistics show higher gains made by many of the students in our Project. Gains range from 1 month to 6.5 years depending upon the level of the class. The twelve students from the Frederick Douglass Literacy Center who received GED's all were participants in the Waterways Project. Of the VTC students who worked with the Project, 20 received high school diplomas; 30 GED; 5 IEP diploma. We believe that this data allows us to infer that some of the gains can be ascribed to the students' participation in our Project.
  • 3.2 Teacher Outcomes: The enthusiasm of the teachers participating in this project, whose classes worked with the various artists, has grown over this past year, as their understanding of the importance and relevance of the project to literacy development and to working with the New York State Learning Standards has increased. The feedback from teachers has been positive and their comments on the observation forms highlight the impact these activities are having. The teacher-observation forms show very clearly how the various arts activities are closely related to many of the New York State Learning Standards. Teachers described understanding, for the first time, how the two areas relate. For example, the classroom observation sheet lists various activities that may occur during the time the artist is with the class, and each activity is keyed to specific Learning Standards.
  • 3.3 Administrator Outcomes: Administrators (Principals, Assistant Principals, Program Directors) of the programs and schools involved participated enthusiastically in the staff development sessions and in testing the original observation forms. Feedback from administrators has been positive. They have asked for the continuation of the programs in their schools. The principal of VTC hopes to find a site where the main vocational focus for participating students will be the arts. The Frederick Douglass Literacy Center Comprehensive Educational Plan 1999-2000 in its Needs Assessment section states: 'Writing as well as reading is a vital component of literacy attainment. Public expression as an implicit motivation for writing has led to an increased emphasis on poetry, drama, and puppetry performance. Music and visual arts are also being emphasized as stimuli for writing.
  • 4. Conclusions:
    It is clear to us that this new evaluation design is a 'successful' prototype. It allows us to evaluate all participants in our project using both qualitative and quantitative measures, and in this way to demonstrate how effectively the Waterways-Ten Penny Players Project has successfully met the goals and objectives outlined in our original grant proposal. Because this evaluation method has worked so well for us, we plan to use this design as we move our Project into future years.


    • Ten Penny Players