Table Of Contentseducation.ucdavis.edu/sites/main/files/file-attachments/... · Web viewAll 8th...

Teaching American History (TAH) Year 2 Report

Prepared by (in alphabetical order):

Michele Hamilton, Shani Keller,

Amy Smith, and Theresa Westover

August 20, 2010

TABLE OF CONTENTS

TABLE OF CONTENTS............................................................................................................................2

LIST OF TABLES.....................................................................................................................................3

LIST OF FIGURES...................................................................................................................................4

INTRODUCTION...................................................................................................................................5

TEACHER CONTENT KNOWLEDGE.........................................................................................................8MULTIPLE CHOICE ASSESSMENT RESULTS.........................................................................................................9DBQ ASSESSMENT RESULTS.........................................................................................................................10

TEACHER SURVEY RESULTS.................................................................................................................12TEACHERS’ GOALS, GAINS, AND CLASSROOM PRACTICE.....................................................................................12CLASSROOM PRACTICES...............................................................................................................................14

LESSON STUDY: 8TH GRADE AMERICAN HISTORY TEACHERS................................................................16

STUDENT SURVEY RESULTS................................................................................................................22STUDENTS ATTITUDES AND BELIEFS ABOUT HISTORY.........................................................................................22STUDENTS’ REPORT OF LEARNING ABOUT AMERICAN HISTORY...........................................................................24

STUDENT ACHIEVEMENT ANALYSES...................................................................................................26INFLUENCE OF TEACHER PROGRAM PARTICIPATION ON STUDENT ELA ACHIEVEMENT...............................................36

Student History CST Achievement Pre-TAH vs. Post-TAH...................................................................38STUDENT ACHIEVEMENT BY TAH PARTICIPATION COMPONENTS AND DURATION....................................................39NESTED MODEL ANALYSIS OF STUDENT HISTORY CST SCORES...........................................................................44

SUMMARY AND CONCLUSIONS..........................................................................................................46

Solano TAH Report Year Two

LIST OF TABLES

Table 1. TAH Teacher Descriptive Statistics By Cohort…………………………………………………..6

Table 2. Results of the Multiple Choice Content Knowledge Tests……………………………………….9

Table 3. Results of the DBQ Assessments………………………………………………………………..11

Table 4. Lesson Study Activities……………………………………………………………………….....17

Table 5. 2009 California Standards Test in History Social Science, Scale Score Ranges………………..26

Table 6. Comparison of Mean CST Scores and Performance Levels, TAH Participants’ Students vs. Non-Participants’ Students………………………………………….…………………………………….27

Table 7. Significant Variables from Linear Regression Analysis………………………………………...30

Table 8. Regression Model Predicting 2009 CST History Scale Score…………………………………..30

Table 9. Average CST History Scale Score For TAH Participants’ Students Vs. Non-Participants’ Students, CST ELA Level Far Below Basic……………………………………………………………...33

Table 10. Average CST History Scale Score For TAH Participants’ Students Vs. Non-Participants’ Students, CST ELA Level Below Basic………………………………………………………………….33

Table 11. Average CST History Scale Score For TAH Participants’ Students Vs. Non-Participants’ Students, CST ELA Level Basic………………………………………………………………………....34

Table 12. Average CST History Scale Score For TAH Participants’ Students Vs. Non-Participants’ Students, CST ELA Level Proficient…………………………………………………………………….34

Table 13. Average CST History Scale Score for TAH Participants’ Students Vs. Non-Participants’ Students, CST ELA Level Advanced…………………………………………………………………….35

Table 14. Average CST ELA Scale Scores for TAH Participants’ Students Vs. Non-Participants’ Students by Prior Year ELA Level (T-test results)……………..………………………………………………………..37

Table 15. Regression Model Predicting CST ELA Scale Scores………………………………………...38

Table 16. Change in Mean CST History Performance Level From 2007-08 to 2008-09………………..38

Table 17. Specific TAH Participation vs. Mean CST History Scale Scores, CST ELA Level Far Below Basic……………………………………………………………………………………………………...41

Table 18. Specific TAH Participation vs. Mean CST History Scale Scores, CST ELA Level Below Basic……………………………………………………………………………………………………...41

Table 19. Specific TAH Participation vs. Mean CST History Scale Scores, CST ELA Level Basic…………………………………………………………………………………………………...…42


Table 20. Specific TAH Participation vs. Mean CST History Scale Scores, CST ELA Level Proficient…………………………………………………………………………………………………43

Table 21. Specific TAH Participation vs. Mean CST History Scale Scores, CST ELA Level Advanced………………………………………………………………………………………………….44

LIST OF FIGURES

Figure 1. Participating Teachers' Years of Experience Teaching American History as of the 2009-2010 School Year…………………………………………………………………………………………………7

Figure 2. Number of Sections of American History Taught in the 2009-10 School Year…………………8

Figure 3. Teachers’ Reported Goals (Pre-Survey) and Gains (Post-Survey) ……………………….……13

Figure 4. Teachers’ Reported Classroom Practice at Entry and in Spring 2010………….………………15

Figure 5. 8th Grade Teachers' Report of The Impacts of Lesson Study on Teacher Collaboration………19

Figure 6. 8th Grade Teachers' Report of the Impacts of Lesson Study on Their Knowledge and Use of Instructional Strategies………………………………………………….…………………………………21

Figure 7. 11th And 8th Grade Students’ Attitudes and Beliefs About History……………...……………23

Figure 8. 11th And 8th Grade Student’s Report of Learning About American History……………..……25

Figure 9. 2008 CST ELA Scale Score vs. 2009 CST History Scale Score…………………………….…28

Figure 10. ELA vs. History Scale Score for Participants vs. Non-Participants………………………...…29

Figure 11. Mean CST History Scale Score By Prior Year CST ELA Level, All TAH Participants’ Students vs. Non-Participants’ Students………………………………………………………………..…32

Figure 12. Mean CST History Scale Score By Prior Year CST ELA Level, TAH Participants’ Students Vs. Non-Participants’ Schools at Schools With at Least 40% FRLP-Eligible……………………………36

Figure 13. Specific TAH Participation vs. Mean CST History Scores By Prior Year CST ELA Level………………………………………………………………………………………………………40


INTRODUCTION

Implementation of the Solano County Teaching American History (TAH) Grant began in the summer of 2008 and will continue through the 2010-11 school year. The grant serves 8th and 11th grade teachers of American History in six different districts within Solano County, California. This report presents the findings from the evaluation of first and second years of implementation, including findings related to teachers’ goals, the program’s impacts on teachers’ content knowledge and classroom practices, students’ attitudes about and interest in history, and their reported knowledge gains over the past year in participating teachers’ classes, and the relationship between teachers’ participation in the TAH program and their students’ academic achievement.

Over the course of first two years of implementation, teachers had the opportunity to participate in several components of the TAH program. During the first year of the grant, 8th grade teachers were invited to participate in an initial 3-day workshop held in the summer of 2008, referred to as a “Gear-Up” session and, beginning in Fall 2008 and continuing throughout the 2008-09 school year, a series of professional development workshops. Not all teachers participated in both summer and school year activities but all are included in the 2008-09 Cohort for analytical purposes. The 2009-10, second program year activities began with a ten day Summer Institute in June 2009 which included 8th grade teachers who had previously participated in the program, as well as 8th and 11th grade teachers who joined as new participants (i.e., 2009-10 cohort includes teachers who began in summer or fall of 2009). This was followed by an additional “Gear-Up” session in August, 2009, which was primarily attended by 11th grade teachers, most of whom (73%) had attended the Summer Institute. Finally, throughout the 2009-10 school year, professional development workshops were offered to 8th and 11th grade teachers and 8th grade teachers participated in Lesson Study. Overall, 81 teachers have participated in the TAH program over the two school years. The characteristics of these teachers are provided in Table 1.


Table 1. TAH Teacher Descriptive Statistics By Cohort

2008-09 Cohort 2009-10 Cohort OverallN 39 42 81Grade Taught1 (%)

Eighth 89.5* 21.4* 53.8Eleventh 10.5* 78.6* 46.3

Hours of Participation Mean (SD) 51.4 44.7 47.4 Minimum 16.5 9.0 9.0Maximum 121.5 109.5 121.5

Years Teaching American History (%)

None 3.2 5.0 4.21 - 2 years 0.0* 30.0 16.93 - 5 years 41.9 22.5 31.06 - 10 years 32.3 15.0 22.5More than 10 years 22.6 27.5 25.4

Total American History Courses Taught (%)

None 16.7 20.0 19.21-2 0.0* 47.5 36.53-5 50.0 25.0 30.86 – 10 33.3 7.5 13.5

* Indicates the difference between cohorts is statistically significant (p <.05). Statistical significance levels (p) can be thought of as the likelihood that the pattern of responses is due to chance; thus smaller values of p indicate a higher level of confidence that the results are due to real differences rather than random variation.

Of the 81 teachers who have participated in the TAH program, 39 teachers joined the program in the 2008-09 cohort and 42 teachers joined in the 2009-10 school year. The teachers’ total hours of participation range from 9 to 121.5 hours of TAH training, with an average of approximately 47 hours of participation per teacher. There was no statistically significant difference in the number of hours of participation between the 2008-09 and 2009-10 cohorts. In general, participating teachers were fairly experienced, with almost half of the teachers, both overall and in each cohort, reporting having taught 6 or more years. However, the 2009-10 cohort had a larger proportion of teachers with only 1 to 2 years of experience (30% vs. 0%). Differences in the teachers’ experience teaching American history are displayed in Figure 1.

1 In the 2008-09 school year, the TAH training was only offered to 8th grade teachers; however four teachers who taught 8th grade in the 2008-09 school year, taught 11th grade in the 2009-10 school year.


Figure 1. Participating Teachers' Years of Experience Teaching American History as of the 2009-2010 School Year

None

1 - 2 years

3 - 5 years

6 - 10 years

10+ years

0% 5% 10% 15% 20% 25% 30% 35% 40% 45%

3%

0%

42%

32%

23%

5%

30%

23%

15%

28%

2009-10 Cohort 2008-09 Cohort

In general, teachers in the 2009-10 cohort reported teaching fewer sections of American history than those who entered the program in the 2008-09 cohort. The majority of the 2008-09 teachers (predominantly 8th grade teachers) reported teaching either 3 to 5 (50%) or 6 to 10 (33%) sections; whereas the majority of the 2009-10 cohort (the majority of which are 11th grade teachers) reported teaching either no sections of American history (20%) or 1 to 2 sections (48%). Although the TAH program is designed for teachers who teach American history, a small number of teachers in each cohort reported that they were not teaching any sections of American history in the 2009-10 school year. It is possible that these teachers’ teach different courses in different years, and although they teach American history on occasion and thus participated in the program, they were not teaching it in the 2009-10 school year. Figure 2 shows the number of sections of American history taught in the 2009-10 school year for each cohort of teachers.


Figure 2. Number of Sections of American History Taught in the 2009-10 School Year

None

1-2 sections

3-5 secions

6-10 sections

0% 10% 20% 30% 40% 50% 60%

17%

0%

50%

33%

20%

48%

25%

8%

2009-10 Cohort 2008-09 Cohort

Despite the differences in the teachers’ experience teaching American history and the number of sections taught across the two cohorts, there were no statistically significant differences between 8th and 11th grade teachers in terms of their teaching experience or number of sections taught. Furthermore, there were no statistically significant differences between either 8th and 11th grade teachers or between the two cohorts of teachers in the total number of hours of program participation.

TEACHER CONTENT KNOWLEDGE

Teachers’ content knowledge was evaluated in two ways: multiple choice assessments and Document Based Question (DBQ) assessments, both of which were administered at the beginning and end of each session (i.e., “Gear-Up” sessions, 2008-09 and 2009-10 school years, and 2009 and 2010 summer institutes). Within each session the same assessments were used for pre- and post-testing, but different pre-post multiple choice assessments were administered at each session (e.g. the multiple choice test for a summer institute pre-post was different, covering different material, than the test administered for the fall-spring school year). Each assessment was designed to assess teachers’ content knowledge related to the specific history topics covered in the individual sessions and to the California state standards. In the “Gear-Up” and Summer Institutes, teachers’ were assessed on the first and last days of the workshop. Participants in the 2008-09 and 2009-10 school years where assessed in the fall and again in the spring. Eighth and 11th grade teachers’ multiple choice tests covered different content.


MULTIPLE CHOICE ASSESSMENT RESULTS

As discussed in the previous evaluation report (Solano County Teaching American History Grant Report: Year One) on the first year of implementation of the TAH program, paired-samples t-tests indicated that teachers in both the 2008 “Gear-Up” session, the 2008-09 school year, and the 2009 Summer Institute scored statistically significantly higher on the multiple choice post-test than on the pre-test, indicating that teachers tended to improve their content knowledge over the course of the program. Similarly, paired-samples t-tests indicate that 8th grade teachers who participated in the TAH program over the course of the 2009-10 school year made statistically significant gains between pre- and post-testing. Over the course of the 2009-10 school year, teachers improved their scores by an average of 8 percentage points (Table 2). Please see the Year One report for details on multiple choice test preparation and administration procedures.

Despite the improvements made by teachers in the majority of the TAH sessions, 11th grade teachers who participated during the 2009-10 school year did not make statistically significant improvements between pre- and post-testing. This lack of improvement, as well as differences in the magnitude of teachers’ gains in the different TAH sessions, may be due in part to differences in the content covered and the assessments administered at each of the sessions. Because both the content and the assessments were different in each of the sessions, it is possible that the degree of difficulty varied across assessments, making comparisons across sessions difficult. However, overall these findings are promising, indicating that over the course of all TAH sessions with the exception of the 11th grade 2009-10 session, teachers are showing improvements in their content knowledge. Results from the 2010 Summer Institute content knowledge assessments will be reported in the Year 3 report. Table 2 shows the results of the multiple choice content knowledge tests in each of the sessions.

Table 2. Results of The Multiple Choice Content Knowledge Tests

Participating Teachers (n)

Pre-Test Average Percent

Correct

Post-Test Average

Percent Correct

Average Percent

Gain 3-day “Gear-Up” 2008 21 84 92 8*2008-09 8th Grade 14 45 78 33*Summer Institute 2009 18 60 77 17*2009-10 8th Grade 13 65 74 8*2009-10 11th Grade 29 64 64 0* Indicates the gain is statistically significant (p <.05). Statistical significance levels (p) can be thought of as the likelihood that the pattern of responses is due to chance; thus smaller values of p indicate a higher level of confidence that the results are due to real differences rather than random variation.

In addition to examining teachers’ gains on the multiple choice assessments, the relationships between teachers’ performance on the assessments and the number of hours they attended the TAH program, the cohort in which they entered the study, the number of sections of


American history taught, and their years of experience teaching American history were examined. After controlling for the other variables, number of hours of participation, number of sections taught, and years of experience were not significantly associated with the teachers’ performance on the multiple choice content knowledge test. However, the cohort that the teachers entered the study was significantly related to the gains they made between the pre- and post-tests in the 2009-10 school year. Teachers who participated in the 2008-09 cohort tended to make larger gains, by approximately 2.44 points (12.2 percentage points), than teachers who entered the study in the 2009-10 cohort. This effect is statistically significant even after controlling for the number of hours that the teachers participated in the TAH program and, therefore, is not attributed to teachers in the 2008-09 cohort having the opportunity to participate in more hours of training. It is possible that this finding may indicate that the effects of the TAH training may become evident after the teacher has had time to utilize what they have learned and apply the material in their classroom.

DBQ ASSESSMENT RESULTS

The second way in which teachers’ content knowledge was evaluated was through the use of DBQ assessments. DBQs are commonly used as a student assessment approach in advanced placement (AP) history classes. According to one source:

The document based question (DBQ) is designed to enable students to work like historians, analyzing and synthesizing evidence from a variety of sources and media. Students will be evaluated on their ability to interpret such factors as purpose, source, bias, date, and place of origin, tone, etc. In order to receive a satisfactory score, students must establish and prove their thesis through accurate and sophisticated utilization of the available documents. The DBQ is designed to test the skills a historian uses in interpreting historical material. (http://www.mtholyoke.edu/acad/germ/neh/gruppe2/principles.html)

For the purposes of this evaluation, participating teachers were asked to complete DBQs in response to prompts that focus on how the teacher would use materials provided to develop a lesson for his/her students. The prompts used for both 8th and 11th grade teachers for the 2009-10 school year program is listed below.2

Administration Prompt:

In any written format, please respond to the documents by:a. Demonstrating the theme, topic, and/or content of the following documents.b. Outlining a lesson plan that discusses HOW you would use these documents inyour classroom to explain the content.c. Describing specific activities or strategies to help your students comprehend andthink critically about the documents.

2 Prompts for the Summer 2008 “Gear-Up” session, the 2008-09 school year, and the Summer 2009 Institute can be found the Year 1 Report.


http://www.mtholyoke.edu/acad/germ/neh/gruppe2/principles.html

d. Using at least two of the documents in your answer.

As can be seen in Table 3, the only TAH session in which teachers did not make statistically significant gains on the DBQ assessments between pre- and post-testing was the 2008 “Gear-Up” session. As discussed in the previous report, this may be because only 3 days separated the pre- and post-testing. In addition, this initial session was seen as a “pilot” exercise for the DBQ assessment: after the “Gear-Up” session the DBQ assessment prompts were refined and a more sensitive and rigorous rubric for grading was developed. Please see the Year One report for full description of the DBQ assessment development and scoring.

Over the course of the 2009-10 school year, both 8th and 11th grade teachers made statistically significant gains on the DBQ assessment (paired samples t-test). All 8th grade teachers improved their scores, with an average gain of 6.4 points and improvements ranging from 1 to 15 points. Similarly, 11th grade teachers had average gain of 2.7 points, ranging from a 9 point improvement to a decrease by 4 points. Among the 11th grade teachers, 43% showed improvements from pre- to post-testing. Teachers’ DBQ scores were not significantly correlated with years of experience teaching American history, sections taught, or hours of TAH participation.

Table 3. Results of The DBQ Assessments

Teachers (n) Average Pre-Score

Average Post-Score

Average Point Gain

“Gear-Up” 2008 22 10.2 10.5 0.52008-09 8th Grade 15 12.9 18.9 5.7*

Summer Institute 2009 18 12.3 16.7 4.2*2009-10 8th Grade 16 11.3 17.7 6.4*2009-10 11th Grade 30 12.1 14.8 2.7*

* Indicates the gain is statistically significant (p <.05). Statistical significance levels (p) can be thought of as the likelihood that the pattern of responses is due to chance; thus smaller values of p indicate a higher level of confidence that the results are due to real differences rather than random variation.


TEACHER SURVEY RESULTS

Upon entry into the TAH program and in the Spring of each year, teachers were asked to complete a survey reporting on their goals, what they learned/wanted to learn in the program, and their classroom practices. Teachers were asked what they were most interested in learning about (pre-surveys) and where they felt they had gained in expertise/skills (post-survey). In addition, teachers were asked about the frequency in which they engaged in classroom practices that are considered to be effective in order to examine whether the TAH program is associated with changes in teachers’ practice. Finally, on the post-survey administered to 8th grade teachers in the spring of 2010, teachers were also asked about how participation in lesson study impacted their knowledge of instructional practices and views on collaborating with other teachers. To date, 71 teachers have completed the pre-survey and 53 teachers completed the post-survey in the Spring of 2010.

TEACHERS’ GOALS, GAINS, AND CLASSROOM PRACTICE

On the pre-survey, teachers were asked to rate how important each of nine program content areas were to them on a scale of 1-5, with a 1 being “less important” and a 5 being “very important/useful.” As in the 2008-09 school year, the two areas which the teachers rated as being the most important were “Increasing my knowledge of effective instructional strategies to help my students read and understand the textbook and other historical documents” (45.7% rated as a 4 or 5) and “Increasing my knowledge of effective instructional strategies for teaching American history” (42.3% rated as a 4 or 5). The areas that the smallest proportion of teachers rated important included (a) increasing skills in grounding history in current research (31.0%) and (b) collaborating with other American history teachers (32.4%).

On the post-survey, teachers were asked to rate the same program content areas in terms of how much they felt they had increased their knowledge and abilities. Teachers were asked to rate the areas on a scale of 1-5, with a 1 indicating “not at all” and a 5 indicating “very much.” As in the 2008-09 school year, the majority of teachers reported gaining skills in each of the skill/knowledge areas addressed by the survey. The areas that the largest proportion of teachers rated as improved (i.e., rating of a 4 or 5) were (a) “Increased my interest in collaborating with other teachers in Solano County to share my ideas about curriculum and teaching practices” (92.5%) and (b) “Increased my knowledge of effective instructional strategies for teaching American History” (84.9%). The areas in which the smallest proportion of teachers reported having improved their skills (i.e., rating of a 4 or 5) were (a) “Learned more about how to locate appropriate primary sources for my students” (56.6%) and (b) “Increased my skill in grounding my American history lessons in current historical research” (62.3%). Results of the pre- and post-survey are displayed in Figure 3.


Figure 3. Teachers’ Reported Goals (Pre-Survey) and Gains (Post-Survey)

Learn/learned to located primary sources

Increase/Increased skill in grounding history lessons in current research

Help students express their historical understanding in writing

Increasing/Increased familiarity with current historical research

Help students see history as a process and interpret historical events

Increased my content knowledge about American history

Learn/learned strategies to help students understand text and historical documents

Learn/learned effective American history instructional strategies

Collaborating with other American History Teachers

0.0% 20.0% 40.0% 60.0% 80.0%100.0%

39.4%

31.0%

40.8%

36.6%

35.7%

40.8%

45.7%

42.3%

32.4%

56.6%

62.3%

66.0%

73.1%

75.5%

75.5%

75.5%

84.9%

92.5%

Post survey scored 4 or 5 where 5 = "Very Much" (n = 53)Pre survey scored 4 or 5 where 5 = "Very Important or Useful" (n = 71)

In addition to looking at teachers’ responses to the individual survey items, the relationships between the sum of the survey items for both the pre- and post-survey and the teachers’ hours of TAH participation, teaching experience, grade level taught, sections taught, and assessment scores were examined. The sum of the pre-survey items, which can be thought of as a measure of the teachers’ goals or motivation, was not significantly correlated with the total number of hours the teachers had participated in the program, the number of years of experience teaching American history, the number of sections of American history taught in the 2009-10 school year, or the teachers’ multiple choice or DBQ scores. However, eleventh grade teachers tended to have higher survey responses than eighth grade teachers (p < .001), indicating that they tended to view increasing their content knowledge and instructional strategies and skills as more important or useful.


Similarly, the sum of the post-survey items, which can be thought of as a measure of the teachers’ perceived learning, was not significantly correlated with the teachers’ grade level, the number of years of experience teaching American history, or the number of sections of American history taught in the 2009-10 school year. However, there was a statistically significant correlation between the number of hours of participation in the TAH program and the teachers’ perceived learning (r = .322, p = .01), indicating that teachers who have attended more hours of TAH training tended to perceive gaining more from the TAH program.

Despite the correlation between perceived learning and the number of hours of participation, there was no statistically significant correlation between teachers’ perceived learning and their performance on either the multiple choice test or the DBQ assessment. This finding mirrors what was found in the evaluation of the 2008-09 teacher surveys and assessments. As was discussed in the Year 1 report, this may be because program providers and the evaluator have stressed to participants that all assessments are intended to measure program effectiveness, not individual teacher growth; therefore teachers are not provided with information about their assessment scores and are unaware of their performance. Similarly, it is possible that teachers are not applying their best efforts on the assessments because performance on the assessments does not impact them personally or because they do not address learning that the teachers consider relevant. Finally, it is possible that teachers’ perceptions of their learning and the assessments are not be correlated due to a “disconnect” between what teachers believe they are learning and what the multiple choice and DBQ tests are assessing. Teachers may be reflecting on learning in a more holistic sense than is captured on the assessments.

CLASSROOM PRACTICES

The next area addressed on the teacher surveys was teachers’ classroom practices. On both the pre- and post-surveys, teachers were asked the extent to which they (a) have students examine and analyze primary sources, (b) develop lessons based on historical questions that students answer by drawing conclusions from historical evidence, (c) teach specific literacy strategies to help students read and understand textbooks and other documents, (d) teach writing skills to help students increase and express their understanding of history, and (e) collaborate with other teachers in Solano County when planning units/lessons on American history. Respondents rated each of these items on a five point scale ranging from 1, indicating “never,” to 5, indicating “nearly always or in the vast majority of your lessons.” Figure 4 shows the percentage of teachers at pre- and post-survey who scored each item as either a 4 or a 5.

In general, teachers reported engaging in these classroom practices more frequently after the TAH training. Although teachers’ frequency of teaching writing skills and literacy strategies in their classes did not change very much, each of the other three pedagogical practices showed statistically significant increases. Frequency of collaborating with other teachers on lesson design, having students examine primary sources, and developing lessons based on historical questions each increased by at least 20 percentage points (Figure 4). This suggests that over the course of the school year, teachers who participated in the TAH training tended to increase their frequency of using these classroom practices; however caution is warranted when interpreted this finding. Because the there is no comparison or control group, these findings are correlational and


therefore, changes in teachers’ reported classroom practices cannot necessarily be attributed exclusively to the TAH training.

Figure 4. Teachers’ Reported Classroom Practice at Entry and in Spring 2010

Teach writing skills

Have students examine primary sources*

Teach specific literacy strategies

Collaborate with other teachers in lesson design*

Develop lessons based on historical questions*

0.0% 20.0% 40.0% 60.0% 80.0%

50.0%

29.4%

40.0%

26.0%

36.0%

50.9%

52.8%

52.8%

54.7%

58.5%

Post-survey scored 4 or 5 where 5 = "Nearly always" (n = 53)Pre-survey scored 4 or 5 where 5 = "Nearly always" (n = 50)

* Statistically significant difference (p < .05) between the percentage of teachers who responded with a 4 or 5 on the pre- and post-surveys

These findings contrast with what was found in the evaluation of the first year of implementation: in the 2008-09 school year, teachers tended to report engaging in these classroom practices more frequently prior to the TAH training than after the training. It was hypothesized that teachers may have changed their interpretation of what these activities actually mean after they were introduced to the concepts as a part of the TAH training. For example, a teacher may have reported that they “almost always” use primary sources on their initial survey, but after attending the training, they may have realized that what they had previously considered “primary sources” did not actually fit the TAH definition. To test whether this was the case, teachers were administered the survey items at the beginning of the 2009 Summer Institute and again as a part of the multiple choice assessment administered at the start of the 2009-10 school year. Because the teachers did not teach between administration of the surveys, differences in the teachers’ responses could be attributed to changes in teachers’ interpretation of the meaning of the survey items resulting from exposure to the TAH training, rather than changes in their actual


classroom practices. However, there was no evidence indicating that teachers rated their frequency of engaging in these activities as less after attending the training; therefore, the pre-survey items that were administered as a part of the survey continued to be used in the evaluation.

In addition to examining teachers’ responses to the individual survey items, the sum of teachers’ responses to survey items, which can be thought of as an indicator of teachers’ frequency of engaging in classroom practices targeted by the TAH program was examined. However, we found that teachers’ report of their frequency of engaging in these classroom practices was not correlated with the teachers’ years of experience teaching American history, the number of sections they taught in the 2009-10 school year, the grade taught, the number of hours of TAH training they have received, or the teachers’ multiple choice or DBQ assessment scores.

LESSON STUDY: 8T H GRADE AMERICAN HISTORY TEACHERS

Lesson study is a practice-based professional development activity that provides teachers an opportunity to collaboratively construct a lesson on a given topic or instructional skill, teach that lesson, observe the results, and then revise and, when possible, re-teach the revised lesson and observe the results. Within the lesson study process, one teacher teaches the collaboratively designed lesson while the other group members observe, noting levels of student engagement and response, fidelity to objectives/goals, and other aspects of the lesson. After the lesson, the observing teachers provide the presenting teacher with feedback and all group members decide what elements of the lesson need to be revised. In traditional lesson study, the lesson is re-taught and observed again by group members. However, due to pacing schedules and constraints on paying for substitute teachers, often, as in TAH, the observing instructors return to their classrooms and teach the revised lesson to their own students and then debrief with their group members, rather than completing a full re-teach and observation cycle. As a result of participating in lesson study, teachers are able to apply their newly acquired American history content knowledge and pedagogy in a systematic, reflective and authentic way.

In year two of the Solano TAH project, participating 8th grade teachers were divided into three lesson study groups for collaborative work over the course of the school year. The lesson study groups met for at least eight hours during three of the five TAH Supper Seminars (conducted November 2010 to February 2011) to develop a standards-based American history lesson. Each lesson study group consisted of four or five 8th grade American history teachers from different schools within Solano County. Facilitators provided feedback to all groups during the TAH supper seminars. Some groups also met outside of the designated TAH supper seminar time to refine their lessons. During the planning sessions, each lesson study group chose a historical investigation question to guide the development of their lesson (Table 4). In addition to selecting an investigative question, teachers determined the appropriate primary sources needed to support their lesson. All groups utilized at least one of the following primary sources for students to analyze during the lesson: images (pictures, advertisements, flyers), quotes, letters,


excerpts from speeches and/or audio recordings. Within their groups, teachers discussed strategies for synthesizing the investigative question, shortening or modifying selected speeches or quotes, how students would analyze the primary sources and how to make the material accessible to the students given the constraints of short class periods. In all groups, teachers determined how much material could reasonably be covered in the time available and developed a final product or culminating activity.

Table 4: Lesson Study Activities

Group-Topic

Investigative Question

Primary Sources/

Documents

Lesson Activity Final Product

Group 1 -Suffrage

What were the views of Americans regarding suffrage for women during the 1800’s?

Quotes

Images

Students analyzed and compared/contrasted two suffrage and two anti-suffrage documents

Written or Oral Essay –Students will present findings and supporting evidence

Group 2- Civil War

Why did men fight in the Civil War?

Quotes

Letter

Audio Recording

Students analyzed four quotes: two individuals in support of the Union and two individuals in support of the Confederacy.

Venn Diagram-Students will show similarities and differences between the Union and the Confederacy.

Group 3- Freedom

When, if ever, does Frederick Douglas experience freedom?

Images

Quotes

Speech

Background Info

Students analyzed three quotes from Frederick Douglass.

No final product noted by evaluator.

Evaluators participated in three lesson study observations (all observations occurred in March) during the spring 2010 semester. In each case, this was the first time the collaboratively designed lesson was taught. Each observation lasted for a full class period. Prior to the lesson study observation, observation group members met briefly to review the intended goals for the lesson and were provided a seating chart. The TAH facilitator provided structured forms for lesson observation and debriefing to all observers. Observers were also reminded by the TAH facilitator to focus on student responses, reactions and understanding of the lesson during their observations. Debriefing sessions occurred immediately following the lesson study observation and were conducted off-campus. During the debriefing sessions, the group analyzed the extent


to which the intended goals for the lesson were met, how students responded to the lesson, student work, and discussed possible revisions. Evaluators were observers only, they did not participate in lesson revision discussion.

Because the evaluators are neither historians nor professional development providers, they were not evaluating the quality of the lessons or teaching approaches, but rather participating in order to gain a better understanding of the lesson study process and how teachers use it to improve their practice and to better engage their students. In all observed classes, teachers appeared to draw connections between previous lessons and concepts and the current lesson to help students see the lesson in historical context. For example, the presenting instructor (group one) referred to the Declaration of Independence as a foundation for discussing women rights. As a result, students were able to draw upon their previous knowledge as a foundation for analyzing society’s historical views of suffrage during the 1800s. The observed lessons used primary sources as an instructional “hook” to engage students with each other and the documents. Although teachers, in the debrief, felt that the use of primary sources served to better engage students, they expressed some concerns that students had not effectively maximized the use of these resources. Students tended to use the primary sources for the purpose of gaining background knowledge, but not to justify their arguments or provide evidence for their conclusions. Likewise, they seemed to feel that the primary sources helped students understand the basic concepts of the lesson, but students were less likely to comprehend the more complex notions of the lesson. For example, during the group three debriefing session, teachers observed through the analysis of written reflections that students understood the concept of personal freedom but were unable to fully grasp the more complex notion of what it means to truly be free. In this case, teachers agreed that the scope of the lesson was too broad and needed to be narrowed to analyze and track how students’ thinking of freedom changed throughout the lesson.

In all of the debriefing sessions, teachers were able to analyze student work and identify how students were responding to the lesson. For example, the presenting teacher (group two) observed that students had completed the required assignment and gained factual information concerning individuals’ decision to fight in the civil war. However, the teacher was unable to determine if students had actually gained historical empathy, which was a goal of the lesson. During the debriefing sessions, the group agreed that adding a category labeled personal connection to the chart would allow students to draw upon personal experiences and demonstrate if they had developed a sense of empathy. Teachers also suggested that the lesson be revised to have students’ role-play by pretending to become the individuals discussed, then write a letter from that individual’s point of view.

Throughout the lesson study debriefing observations, evaluators observed teachers’ working to think critically about how students conceptualize historical events. The process of collaboratively designing a lesson provided teachers with an opportunity to develop a context for historical investigations through the use of primary sources. Teachers were able to assist students in comprehending the basic tenets of an issue as well as developing students’ ability to critically analyze multiple perspectives relative to a specific historical period. The lesson study process appeared to augment teachers’ ability to examine student work and modify their pedagogical


skills as needed. All teachers were concerned with helping students construct historical explanations and identify supporting evidence to support their conclusions. While the evaluators cannot determine from their limited observations if the lesson study process encouraged teachers to step outside their personal teaching style and explore multiple instructional strategies, it was evident that teachers were focused on ensuring that students were engaged and able to assess/comprehend the curriculum.

On teacher post-surveys, conducted in the spring of 2010, the fourteen 8th grade teachers (who had participated in lesson study) responded to an additional survey section which asked them to report on how participation in lesson study impacted their views about collaborating with other teachers (Figure 5) and about their knowledge of instructional practices (Figure 6). Of the teachers who responded to the survey, the majority reported that, as a result of lesson study, they were more likely to collaborate with other history teachers, both in their school (87.5%) and in other schools (81.3%), that they enjoy collaborating with other teachers (87.5%), and that they increased their appreciation of the value of teacher collaboration (81.3%). Furthermore, 62.5% of teachers reported that they plan to collaborate with teachers from their lesson study group in the future.

Figure 5. 8th Grade Teachers' Report of the Impacts of Lesson Study on Teacher Collaboration

I plan to collaborate with teachers from my lesson study group in the future

I increased my appreciation of teacher collaboration

I am more likely to collaborate with history teachers in other schools in the future

I enjoyed collaborting

I am more likely to collaborate with history teachers at my school in the future

0.0%

20.0%

40.0%

60.0%

80.0%

100.0%

62.5%

81.3%

81.3%

87.5%

87.5%

Post-Survey Scored 4 or 5 where 5 = "Strongly Agree" (n = 14)


In addition, the majority of teachers reported that the lesson study experience had an impact on their knowledge and use of instructional strategies. The majority of teachers reported that, as a result of participation in lesson study, they learned (62.5%) and tried (81.3%) new instructional approaches. Similarly, the majority reported that, after the lesson study, they had a better understanding of how to improve their own instructional practices (81.3%), guide students in historical investigations (75.0%), examine student work for evidence of lesson effectiveness (68.8%), and gauge students’ responses to and understanding of lessons (62.5%). Furthermore, teachers reported learning more about a specific topic in history (62.5%) and to better see how students understand and learn history (50%). In contrast, only 37.5% of the surveyed teachers reported that, as a result of the lesson study, they better understand how to tailor their teaching to the specific needs of their students.


Figure 6. 8th Grade Teachers' Report of the Impacts of Lesson Study on Their Knowledge and Use of Instructional Strategies

I better understand how to tailor my teaching to the specific needs of students

I better see how students understand and learn historical concepts

I am better able to gauge my students' repsonses to and understanding of lessons

I learned about a specific historical topic

I better understand how to assess my students' historical knowledge and understanding

I better understand how to examine student work for evidence of lesson effectiveness

I better understand how to guide my students in historical investigations

I learned about different instructional approaches

I better understand how to go about improving my own instructional practices

I tried out new instructional approaches

0.0% 20.0% 40.0% 60.0% 80.0%100.0%

37.5%

50.0%

62.5%

62.5%

68.8%

68.8%

75.0%

75.0%

81.3%

81.3%

Post-Survey Scored 4 or 5 where 5 = "Strongly Agree" (n = 14)


STUDENT SURVEY RESULTS

Students of teachers who participated in the TAH training were asked to respond to a survey in the spring of 2010, reporting on their attitudes and beliefs about history and reporting on what they had learned in their American history course in the 2009-2010 school year. Students were asked to rate items related to their attitudes, beliefs, and learning on a scale of 1-4, with 1 indicating that they “totally agree” with the statement and a 4 indicating that they “totally disagree.” Overall, 687 8th grade students and 271 11th grade students responded to the survey.

Participating teachers volunteered to administer the student survey, there was no oversight to determine which or how many of their class sections were surveyed, thus these results should not be regarded as necessarily representative of all students in all participating teachers’ classes. A total of 14 (33%) of the 2009-2010 teachers administered the survey to their students. A total of 957 students responded to the survey for an average of 74 student responses per teacher. Of the teachers who administered the survey, eight taught 8th grade and six taught 11th grade. Correspondingly, the students who responded to the survey were predominately 8th graders (78%).

STUDENTS ATTITUDES AND BELIEFS ABOUT HISTORY

In general, students’ responses to the items related to their attitudes and beliefs about history were positive, with the majority of students agreeing with the positive statements on the survey. The item on which the largest proportion of students reported that they either agreed or totally agreed was “I like working with other students to learn history” and correspondingly, the item on which the smallest proportion of students reported they either agreed or totally agreed was “I like working by myself to learn history” (Figure 7).

There were several statistically significant differences in the proportions of 8th and 11th grade students who reported totally agreeing/kind of agreeing or totally disagreeing/kind of disagreeing with the items (Figure 7). On seven of the nine survey items, a larger proportion on 11th grade students reported that they either totally agreed or kind of agreed including: (a) liking to work with other students, (b) finding history more important after taking the class, (c) needing to understand history to meet future goals, (d) thinking history is important in life, (e) thinking history is interesting, (f) using history in everyday life, and (g) using history to complete assignments in other classes. It is possible that these differences by grade level may be due, in part, to the difference in the age of the students, with 11th graders being more mature, having more life experience that may help provide context for issues discussed in their American history courses, and being capable of understanding more complex issues.


Figure 7. 11th And 8th Grade Students’ Attitudes and Beliefs About History

I like working by myself to learn history

I used history to complete assignments in other classes

I look forward to taking history classes in college

I have used my knowledge of history in my everyday life *

I will need to understand history to meet my goals in the future

History is important in my life *

History is interesting *

I find history more important now than before I took this class *

I get good grades in history

I like working with other students to learn history

0% 20% 40% 60% 80% 100%

53%

57%

59%

73%

77%

78%

79%

79%

79%

82%

48%

51%

55%

53%

73%

69%

71%

63%

82%

77%

Percentage of 8th grade students (n = 687) who totally agree/kind of agreePercentage of 11th grade students (n = 271) who totally agree/kind of agree

* Statistically significant difference (p < .05) between the proportions of 8th and 11th grade students who totally agreed/kind of agreed


STUDENTS’ REPORT OF LEARNING ABOUT AMERICAN HISTORY

The survey also asked students to report on their learning in their American history course in the 2009-10 school year, both in terms of what they learned and how they learned. The vast majority of 8th (85%) and 11th (96%) grade students reported that their teachers knew how to explain history so they could understand (Figure 8). Correspondingly, the majority of the students reported that they either “totally agreed” or “kind of agreed” that they learned how to (a) interpret and understand historical items (73.6%), (b) to write historical essays and explanations (60.7%), (c) to construct effective arguments (65.6%), and (d) to be a critical reader and to better understand what they read (63.9%). Finally, the majority of the students reported that they understood and learned history better when they discussed historical events (76.7%) or when they saw how it related to current events (75.9%). Just over half of the students reported that they had a difficult time understanding history when they read the textbook (53.7%).

For all but one of the survey items relating to student learning, there were statistically significant differences in the proportions of 8th and 11th grade students who either totally agreed or kind of agreed and those who either totally disagreed or kind of disagreed (Figure 8). For each of these items, a larger proportion of the 11th grade students reported agreeing or strongly agreeing. This may indicate that 11th grade students felt as though they learned more over the course of the school year, or, as with the survey items related to students’ attitudes and beliefs, it may be due to the difference in the students’ age, with 11th graders being more mature and having more life experience.


Figure 8. 11th And 8th Grade Student’s Report of Learning About American History

I had a hard time understanding history when I read the book *

I learned how to be a critical reader and to better understand what I read

I learned how to write historical essays and explanations *

I learned how to construct effective arguments *

I learned how to interpret and understand historical documents *

History "came alive," I felt that I understood history much better *

I understood history better when I discussed historical events *

I learned history better when I saw how it related to current events *

My teacher knew how to explain history so I could understand *

0% 20% 40% 60% 80% 100%120%

53%

69%

70%

72%

79%

83%

86%

87%

96%

44%

62%

57%

63%

72%

64%

73%

72%

85%

Percentage of 8th grade students (n = 687) who totally agree/kind of agreePercentage of 11th grade students (n = 271) who totally agree/kind of agree

* Statistically significant difference (p < .05) between the proportions of 8th and 11th grade students who totally agreed/kind of agreed

Finally, to determine whether students’ report of learning was related to the number of hours their teacher participated in the TAH training, the correlation between the sum of the four student survey items related to learning and the number of hours their teacher participated in the


TAH training was examined.3 The four “learning” items included: (a) “I feel that I learned a lot about how to interpret and understand historical documents in this class,” (b) “I learned how to be a critical reader and to better understand what I read, from this class,” (c) “I learned about how to write historical essays and explanations in this class,” and (d) “I learned how to construct effective arguments about historical events in this class.” The sum of these items can be seen as a composite variable representing the students’ report of their overall learning in the class. Results indicate that the correlation between student’s report of their learning and the number of hours their teacher participated in the TAH training was not statistically significant.

STUDENT ACHIEVEMENT ANALYSES

A major goal of the Teaching American History (TAH) program is to improve the instructional practices and content knowledge of classroom history teachers. One of the intended outcomes of improved instruction is improved student achievement on standardized history tests. To determine whether students of TAH participants benefitted from their teacher’s participation in the program, this portion of the evaluation of TAH focuses on the differences in achievement between students of TAH teachers and students of non-participating teachers. For this report year we report only grade 8 students’ achievement results from the 2008-09 school year (the first year of the grant). Current (2009-10) achievement data were not available until after the due date of this report. They will be reported next year, and will include both grades 8 and 11.

In order to measure student achievement, the primary outcome variable of interest is students’ spring 2009 California Standards Test (CST) Scale Score on the Grade 8 History Social Science (HSS) test. The test is administered to all grade 8 students enrolled in California public schools. The scale score for all CSTs range between 150 and 600. The California Department of Education also assigns a performance level to student CST scores of Far Below Basic (FBB), Below Basic (BB), Basic (B), Proficient (P), and Advanced (A), with the stated goal of all students achieving at the Proficient or Advanced level. The minimum score required to achieve at the Basic level is 300, and the minimum score to achieve at the Proficient level is 350. Table 5 below lists the scale score ranges for the 2009 Grade 8 CST History test.

Table 5. 2009 California Standards Test in History Social Science, Scale Score Ranges

Performance Level

Far Below Basic

(1)

Below Basic

(2)

Basic(3)

Proficient(4)

Advanced(5)

Scale Score Range

150-270 271-299 300-349 350-395 396-600

3 Due to the nested structure of the data, with students grouped within teachers, hierarchical linear modeling (HLM) was used to examine the relationship between the number of hours of TAH training a teacher received and their students’ report of learning.


Participating districts provided student CST data for the 2008-09 administration. To facilitate comparison with student performance from prior years, 2007-08 and 2006-07 CST data were requested along with student demographic data for the past three years. As student-level data regarding eligibility for free/reduced-price lunch was not readily accessible in many districts, a school-wide proxy variable was used (schoolwide percent of students eligible for free/reduced-price lunch). All participating districts submitted student data for the 2008-09 school year; however, not all districts were able to provide historical data on student performance for prior years. This analysis includes all data that districts were able to provide. Finally, TAH teachers’ participation data was described in terms participation in the specific modules of the program - Summer Gear-Up and/or the school year trainings from summer 2008 through the 2008-09 school year.

The broadest look at results is a comparison of students of TAH teachers vs. those of non-TAH teachers. This is accomplished first by a comparison of mean scale scores on the 2009 History CST.

Table 6. Comparison of Mean CST Scores and Performance Levels, TAH Participants’ Students vs. Non-Participants’ Students

TAHN Mean

Std. Deviation

Std. Error Mean

2009 CST HSS Scale Score

TAH Participant

1964 340.03a 59.057 1.333

Non-Participant

2108 331.09a 59.262 1.291

2009 CST HSS Performance Level

TAH Participant

1964 3.21b 1.266 .029

Non-Participant

2108 3.02b 1.278 .028

a,b Statistically significant different (p≤.05)

Overall, students of TAH teachers did earn statistically significantly higher scale scores than did their peers with non-participating teachers. However, the difference is somewhat small: 9 points on a scale that ranges from 150 to 650. Similarly, the difference between the CST History performance levels was significantly higher for students of TAH teachers compared to students of non-participants; however, the difference of 0.19 performance levels has little real-world significance as performance levels are represented by whole numbers (1-5). In addition to the small magnitude of the differences, it is also important to consider that the difference between students of participating and non-participating teachers does not take into account students’ starting levels in this analysis.

Ideally, a comparison of student growth would use two similar tests as the starting and ending measure, however, there is no grade 7 CST History test. Therefore, the grade 7 CST


ELA test was used as a proxy after first testing the hypothesis that grade 7 CST ELA was highly correlated with the grade 8 CST History test. The relationship between grade 7 CST ELA scale score and performance level with the grade 8 CST History scale score and performance level was calculated for each of the most recent three school years (2006-07, 2007-08, and 2008-09). In each year the correlation coefficient between the variables ranged from .707 to .762, all significant at the .0001 level. In other words, there is a significant relationship between performance on the grade 8 CST History test and the prior year’s CST ELA test, whether performance is measure by scale score or performance level in each of the three school years. For the three years of data analyzed, the strongest relationship was between the grade 8 CST History scale score and the prior year ELA scale score (r=.757 for 2008-09). The following graph illustrates the relationship between 2009 CST History scale score and prior year CST ELA scale score.

Figure 9. 2008 CST ELA Scale Score vs. 2009 CST History Scale Score

0 100 200 300 400 500 6000

50

100

150

200

250

300

350

400

450

500

550

600

650

f(x) = 0.784827496557525 x + 60.2493298048476R² = 0.573084800272466

CST History Scale ScoreLinear (CST History Scale Score)

2008 CST ELA Scale Score

2009

CST

Hist

ory

Sca

le S

core

The regression line indicates a positive relationship between Grade 8 History CST scale score and the prior year ELA scale score, in support of the correlation data reported above. The


R2 indicates that more than half (57.3%) of the variance is explained by the linear model. In order to compare students of TAH teachers to those of non-participating teachers, student scores in 2009 CST History and the prior year CST ELA were plotted with separate regression lines. See Figure 10.

Figure 10. ELA vs. History Scale Score for Participants vs. Non-Participants

100 200 300 400 500 600 7000

100

200

300

400

500

600

700

f(x) = 0.781135588848386 x + 58.4770266011315R² = 0.586044118904883f(x) = 0.788214737634844 x + 65.0087354296601R² = 0.555285839006529

History Score - TAHLinear (History Score - TAH)History Score - Non-ParticipantsLinear (History Score - Non-Partic-ipants)


2009

CST

Hisr

oty

Scal

e Sc

ore

The plots for student scores largely overlap, and the regression lines for students of TAH participants and non-participants are very close, however, the students of TAH participants appear to do slightly better, on average, than students of non-participant who entered with similar 2008 CST ELA scores. To determine whether the difference between students of TAH and non-participating teachers is significant and expose other factors that contribute or relate to differential student success, a more thorough regression analysis was run using the stepwise regression method. The analysis included the following independent variables:


Table 7. Significant Variables from Linear Regression Analysis

Variable Description**Prior Year ELA Score


**EL Student English Learner Status – EL or Not ELSWD Student with Disability Status – SWD or Not

SWDAmerican Indian American Indian or Not American Indian**Asian Asian or Not AsianPacific Islander Pacific Islander or Not Pacific IslanderFilipino Filipino or Not FilipinoHispanic Hispanic or Not HispanicAfrican American African American or Not**White White or NotOther Missing/Unknown/Other Ethnicity or Not**Pct FRPL Percent of schoolwide population eligible for

free or reduced price lunch**TAH Participating in TAH – Participant or Non-

Participant**Variable included in final regression analysis using stepwise model (criteria: probability-of-F-to-enter≤.050)

The strongest regression model, which explained approximately 58% of the variance in student data, included the following variables: prior year ELA score, EL, Asian, White, school-wide percentage of students who were free/reduced lunch eligible, and TAH participation. The data in the following tables further describe this regression model.

Table 8. Regression Model Predicting 2009 CST History Scale Score

2009 CST History (predicted)Adj. R square = .573

Variables in Model B Beta

Significance

Prior Year ELA Score

.785 .752 .000

TAH class 5.161 .044 .001Asian (1=yes) 15.983 .059 .000White (1=yes) 6.448 .052 .000EL 6.638 .033 .011Pct_FRPL .118 .033 .021

Placement in a participating TAH teacher’s classroom is confirmed to be a significant variable, with TAH students scoring on average approximately 5 points higher on the CST History test than their peers. Variables positively related to a higher CST History score included:


prior year CST ELA scale score, Asian ethnicity, White ethnicity, and (surprisingly) being EL or in a school with higher percent of student eligible for free or reduced-price lunch.

To examine the difference in CST History achievement between students of participants and non-participants more closely, students were compared by the significant variables within each CST ELA performance level from the prior year. For example, for students falling within the Far Below Basic performance level on their prior year CST, the average CST History scale score for students with participating versus non-participating teachers was compared among (1) all students, (2) Asian students, (3) White students, and (4) EL students. To account for the free/reduced-price lunch (FRPL) variable, students were grouped into high- and low-FRPL school categories. These FRPL categories were created based on the summary descriptive statistics for the data set: with FRPL data for the schools of 3767 students, the average percent of students eligible for Free/Reduced-Price Lunch was 43.08%, and the median was 37.90%. Schools with fewer than 40% of students eligible for Free/Reduced Price Lunch were categorized as “low-FRPL”, while schools with 40% or more of their students eligible for Free/Reduced Price Lunch categorized as “high-FRPL.” The following figure illustrates the differences in achievement between students of TAH participants and non-participants by prior year ELA CST performance level.


Figure 11. Mean CST History Scale Score By Prior Year CST ELA Level, All TAH Participants’ Students vs. Non-Participants’ Students

Far Below Basic

Below Basic

Basic

Proficient

Advanced

0 100 200 300 400 500 600

258

283

321*

356

404

258

280

311*

352

399

Non-ParticipantsTAH Participants

Mean 2009 CST History Scale Score

Prio

r Yea

r CST

ELA

Per

form

ance

Leve

l

* Statistically significant difference, independent samples t-test (p<.05)

Among all students, the only significant difference in achievement was found among students who started the year with a CST ELA performance level of Basic. The following tables compare student performance by prior year CST ELA score for each of the significant variables


Table 9. Average CST History Scale Score For TAH Participants’ Students Vs. Non-Participants’ Students, CST ELA Level Far Below Basic

All Students Asian White EL ≥40% FRPL

Group N

Avg. CST

History Scale Score N

Avg. CST


Avg. CST


Avg. CST


Avg. CST

History Scale Score

TAH Participant 100 258.27 1 --* 16 262.69 33 258.61 80 257.36

Non-Participant 132 257.86 4 --* 15 259.07 25 258.24 51 258.57

*statistics are not reported for groups with n<5

There were no noted differences on the CST History test among TAH participants’ versus non-participants’ students who started the year with a CST ELA level of Far Below Basic, either overall or within any of the significant categories. There were too few Asian students entering at the Far Below Basic level to report results with confidentiality.

Table 10. Average CST History Scale Score For TAH Participants’ Students Vs. Non-Participants’ Students, CST ELA Level Below Basic


Group N

Avg. CST


Avg. CST


Avg. CST


Avg. CST


Avg. CST

History Scale Score

TAH Participan

t182 283.29 6 318.17 39 285.62 33 281.36

141 282.70

Non-Participan

t271 279.94 7 280.43 59 289.97 37 282.00 86 278.85

There were no noted differences on the CST History test among TAH participants’ versus non-participants’ students who started the year with a CST ELA level of Below Basic, either overall or within any of the significant categories.

33

Table 11. Average CST History Scale Score For TAH Participants’ Students Vs. Non-Participants’ Students, CST ELA Level Basic


Group N

Avg. CST History Scale Score N




Avg. CST History Scale Score

TAH Participant

411 321.28a 14 333.07 96 323.10b

53 321.17c

300 322.28d

Non-Participant

508 310.86a 14 339.57

135 311.25b

21 301.10c

192 307.24d

a,b,c,d Statistically significant difference, independent samples t-test (p<.05)

There were several statistically significant differences on the CST History test among TAH participants’ versus non-participants’ students who started the year at Basic on the CST ELA. Among all Basic students, those with a TAH teacher scored, on average, 10.42 points higher on the CST History test than their peers with a non-participating teacher. This difference between students of TAH teachers and non-participants is also seen among White students, EL students, and in schools with 40% or more of their students eligible for free or reduced-price lunch.

Table 12. Average CST History Scale Score For TAH Participants’ Students Vs. Non-Participants’ Students, CST ELA Level Proficient

All Students Asian White EL ≥ 40% FRPL

Group N






TAH Participant 438 355.88 29

376.79a 151 355.21

30 374.07

295

354.94b

Non-Participant 634 352.41 33

357.30a 247 360.91 2 --*

218

342.67b

a,b Statistically significant difference, independent samples t-test (p<.05)*statistics are not reported for groups with n<5

Among students who started the year at Proficient on the CST ELA, significant differences in CST History achievement were seen among Asian students and students at schools with 40% or more of their students eligible for free or reduced price lunch. The success of TAH students within these groups did not, however, translate to a significantly

34

higher score among all students who entered at Proficient.

Table 13. Average CST History Scale Score for TAH Participants’ Students Vs. Non-Participants’ Students, CST ELA Level Advanced


Group N






TAH Participant 248 404.04 17 413.29 109

408.24a

38 420.18

145

406.54b

Non-Participant 367 399.07 25 407.12 182

396.90a 0 --*

145

388.52b

a,b Statistically significant difference, independent samples t-test (p < .05)*statistics are not reported for groups with n<5

Among students who started the year at Advanced on the CST ELA, significant differences in CST History achievement were seen among White students and students at schools with 40% or more of their students eligible for free or reduced price lunch. The success of TAH students within these groups did not, however, translate to a significantly higher score among all students who entered at Proficient. Figure 12 below illustrates the difference in achievement between students of TAH teachers in schools with at least 40% of students eligible for free/reduced-price lunch vs. students of non-participating teachers.

35

Figure 12. Mean CST History Scale Score By Prior Year CST ELA Level, TAH Participants’ Students Vs. Non-Participants’ Schools at Schools With at Least 40% FRLP-Eligible

Far Below Basic

Below Basic

Basic

Proficient

Advanced

0 100 200 300 400 500 600

257

283

322*

355*

407*

259

279

307*

343*

389*

Non-ParticipantsTAH Participants


Prio

r Yea

r CST

ELA

Perf

orm

ance

Le

vel

* Statistically significant difference, independent samples t-test p < .05)

Among students at schools with at least 40% of students eligible for free/reduced-price lunch, students of TAH participants who entered with a CST ELA level of Basic, Proficient, or Advanced scored significantly higher on the History CST than their peers with non-participating teachers. Students with lower levels of proficiency (Far Below/Below Basic) did not appear to gain the same benefits from being in a participating teacher’s classroom.

INFLUENCE OF TEACHER PROGRAM PARTICIPATION ON STUDENT ELA ACHIEVEMENT

In addition to CST History scores, average scale scores for the 2009 CST ELA were also analyzed for significant differences between students of TAH participants vs. those of non-participants. There were no significant differences between participants’ and non-participants’ students’ achievement in English language arts (ELA), when controlling for their prior year CST ELA level. Both overall and when examined by previous year’s performance levels,

36

student ELA scores in participating teachers’ classes did not differ significantly from those in non-participating teachers’ classes (Table 14).

Table 14. Average CST ELA Scale Scores for TAH Participants’ Students Vs. Non-Participants’ Students by Prior Year ELA Level (t-test results)

Prior Year ELA Level N

Mean 2009 CST ELA Scale Score

Far Below Basic TAH Participant 99 269.01Non-Participant 126 265.41

Below Basic TAH Participant 181 291.46Non-Participant 267 290.74

Basic TAH Participant 414 325.62Non-Participant 512 324.77

Proficient TAH Participant 438 366.10Non-Participant 638 368.15

Advanced TAH Participant 248 416.96Non-Participant 368 416.22

Additional analyses, using regression, also failed to demonstrated significant differences between participating and non-participating teachers’ classrooms in student ELA achievement. Neither being placed in a participating teacher’s class (dichotomous variable) nor participating teachers’ hours of participation (continuous variable) resulted in significantly explaining any portion of the variation in student ELA test scores, controlling for their previous year’s ELA score (Table 15).

37

Table 15. Regression Model Predicting CST ELA Scale Scores

2009 CST ELA (predicted)

Adj. R square = .711

Variables in Model 1 B Beta Significance

Prior Year ELA Score .825 .843 .000

TAH (participant/ non-participant)

.276 .002 .795

Variables in Model 2

Prior Year ELA Score .825 .843 .000

TAH teacher hours .043 .011 .240

STUDENT HISTORY CST ACHIEVEMENT PRE-TAH VS. POST-TAH

In order to determine whether TAH teachers were more effective instructors compared to prior years, student scores from 2008-09 were compared to student scores from the prior year (2007-08). In order to account for variation in the starting performance among the student cohorts, data were analyzed by prior year CST ELA level. As CST scale scores are not comparable from year to year, student outcome data is selected for this analysis is CST History performance level, with Far Below Basic = 1, Below Basic = 2, Basic = 3, Proficient = 4, and Advanced = 5. Table 16 shows, for TAH participants and non-participants, the difference between History CST achievement by starting CST ELA level.

Table 16. Change in Mean CST History Performance Level From 2007-08 to 2008-09

Prior Year CST

ELA Level

TAH School

Year

N

Mean

History

CST Level Std. Deviation

Std. Error

Mean

Far Below Basic Non-Participant 2008-09 132 1.37 .558 .049

2007-08 107 1.50 .757 .073

TAH Participant 2008-09 100 1.35 .609 .061

2007-08 78 1.46 .658 .075

Below Basic Non-Participant 2008-09 271 1.88 .857 .052

38

2007-08 273 1.84 .849 .051

TAH Participant 2008-09 182 1.96a .966 .072

2007-08 207 1.73a .758 .053

Basic Non-Participant 2008-09 508 2.64 .962 .043

2007-08 530 2.54 .916 .040

TAH Participant 2008-09 411 2.86b .957 .047

2007-08 404 2.53b .936 .047

Proficient Non-Participant 2008-09 634 3.56 .929 .037

2007-08 609 3.58 .880 .036

TAH Participant 2008-09 438 3.61c .900 .043

2007-08 460 3.46c .995 .046

Advanced Non-Participant 2008-09 367 4.37 .715 .037

2007-08 256 4.43 .727 .045

TAH Participant 2008-09 248 4.47 .725 .046

2007-08 155 4.37 .712 .057a,b,c Statistically significant difference, independent samples t-test (p<.05)

Among TAH participants’ classes, students entering with 2008 CST ELA levels of Below Basic, Basic, and Proficient scored at a significantly higher level on the 2009 CST History test compared to their peers who had entered the same teachers’ classrooms with the same ELA placement the prior year. When analyzed by prior year ELA level, there is no corresponding difference in student performance among non-TAH teachers from one year to the next. This suggests that participating teachers did, in fact, become more effective (as measured by their students’ performance on the CST History test) during the year in which they were participating in the program. Next year’s report will examine the 3 year trend for 8th grade teachers.

STUDENT ACHIEVEMENT BY TAH PARTICIPATION COMPONENTS AND DURATION

In the 2008-09 school year, there were two main components of the program: the summer Gear Up module, and various professional development offerings during the school year. To analyze the impact of each offering, teachers were grouped into the following groups: (1) attended summer Gear Up only; (2) attended various trainings during the school year only; and (3) attended summer Gear Up plus various trainings during the school year. In order to determine which group of students were helped the most, students were grouped according to their prior year CST ELA level. Figure 13 below illustrates the differences in average student achievement among teachers with differing participation in TAH.

39

Figure 13. Specific TAH Participation vs. Mean CST History Scores By Prior Year CST ELA Level

Far Below Basic

Below Basic

Basic

Proficient

Advanced

0 100 200 300 400 500 600

258

274

311

311

387

253

292

307

338

396

260

287

332

368

417

258

280

311

352

399

Non-ParticipantGear Up and School YearSchool Year OnlyGear Up


Prio

r Yea

r CST

ELA

Per

form

ance

Leve

l

The highest mean student achievement at every 2008 ELA performance level was among students of teachers who participated in both Gear Up and school year TAH professional development. Caution must be used in the interpretation of these findings, as it is unclear whether differences in achievement are related to the specific content covered or the total hours of exposure. Tables 17 through 21 compare detail student outcomes based on their prior CST ELA and specific teacher TAH participation.

40

Table 17. Specific TAH Participation vs. Mean CST History Scale Scores, CST ELA Level Far Below Basic

Activity Gear Up Only School Year OnlyGear Up and School Year

Participation N




TAH, Activity Participant 33 257.58 12 253.00 55 259.84TAH, All Others 67 258.61 88 258.00 45 256.36Non-Participants (No TAH) 132 257.86 132 257.86 132 257.86

Among students who started the year with a CST ELA level of Far Below Basic, there were no noted differences on the CST History test scores between those whose teachers went to TAH Gear Up only versus went to other TAH trainings or did not participate in TAH at all.

Table 18. Specific TAH Participation vs. Mean CST History Scale Scores, CST ELA Level Below Basic

Activity Gear Up Only School Year Only

Gear Up and

School Year

Participation N

Avg. CST

History

Scale Score N

Avg. CST

History

Scale Score N

Avg. CST

History

Scale Score

TAH,

Activity

Participant 63 273.67a 28 291.79 91 287.34

TAH, All

Others 119 288.39a 154 281.75 91 279.24

Non-

Participants

(No TAH) 271 279.94 271 279.94 271 279.94

41

a Statistically significant difference, independent samples t-test (p<.05)

Among students who began the year at a CST ELA level of Below Basic, those with teachers who participated in Gear Up only scored significantly lower on the CST History test than students of all other TAH teachers, but not significantly different from students of non-TAH participants.

Table 19. Specific TAH Participation vs. Mean CST History Scale Scores, CST ELA Level Basic


Gear Up and

School Year

Participation N

Avg. CST

History

Scale Score N

Avg. CST

History

Scale Score N

Avg. CST

History

Scale Score

TAH,

Activity

Participant 154 310.87a 46 307.46 211 331.90b,c

TAH, All

Others 257 327.52a 365 323.02 200 310.08b

Non-

Participants

(No TAH) 508 310.86 508 310.86 508 310.86c

a,b,c Statistically significant difference, independent samples t-test (p<.05)

Among students who began the year at a CST ELA level of Basic, those with teachers

who participated in Gear Up only scored significantly lower on the CST History test than did

students of all other TAH teachers, but not significantly different from students of non-TAH

participants. Students with teachers who participated in both Gear Up and the school year

professional development scored significantly higher on the CST History test than did students

of all other TAH teachers, and higher than students of non-participating teachers, as well.

42

Table 20. Specific TAH Participation vs. Mean CST History Scale Scores, CST ELA Level Proficient


Gear Up and

School Year

Participation N

Avg. CST

History

Scale Score N

Avg. CST

History

Scale Score N

Avg. CST

History

Scale Score

TAH,

Activity

Participant 169 310.87a,b 70 338.13c 199 368.27d,e

TAH, All

Others 269 365.63a 368 355.45c 239 345.56d

Non-

Participants

(No TAH) 634 352.41b 634 352.41 634 352.41e

a,b,c,d,e Statistically significant difference, independent samples t-test (p<.05)

Among students who began the year at a CST ELA level of Proficient, those with

teachers who participated in Gear Up only or school year professional development only scored

significantly lower on the CST History test than did both students of all other TAH teachers and

students of non-participating teachers. Students with teachers who participated in both Gear Up

and the school year professional development scored significantly higher on the CST History

test than did students of all other TAH teachers, and higher than students of non-participating

teachers, as well.

Table 21. Specific TAH Participation vs. Mean CST History Scale Scores, CST ELA Level Advanced


Gear Up and

School Year

Participation N

Avg. CST

History

Scale Score N

Avg. CST

History

Scale Score N

Avg. CST

History

Scale Score

TAH, 83 387.41a,b 33 395.55 132 416.62c,d

43

Activity

Participant

TAH, All

Others 165 412.41a 215 405.34 116 389.72c

Non-

Participants

(No TAH) 367 399.07b 367 399.07 367 399.07d

a,b,c,d Statistically significant difference, independent samples t-test (p<.05)

Among students who began the year at a CST ELA level of Advanced, those with teachers who

participated in Gear Up only scored significantly lower on the CST History test than did both

students of all other TAH teachers and students of non-participating teachers. Students with

teachers who participated in both Gear Up and the school year professional development scored

significantly higher on the CST History test than did students of all other TAH teachers, and

higher than students of non-participating teachers, as well.

NESTED MODEL ANALYSIS OF STUDENT HISTORY CST SCORES

To further investigate the effect of the TAH training on students’ history achievement,

we examined the relationship between the number of hours a teacher participated in the TAH

training and their students’ History CST scores, controlling for students’ ethnicity, gender, prior

achievement (i.e., the students’ ELA CST scores from the 2007-08 school year), and the

teacher’s experience teaching American history.4 Students’ History CST scores were

significantly related to the number of hours of TAH training a teacher received (γ = 1.30, p

< .001), indicating that for every hour a teacher participated in the TAH training, their students’

history assessment scores increased by an average of 1.3 points. This finding is particularly

noteworthy, given the wide range in the number of hours teachers participated in the TAH

training, ranging from 16.5 to 39.5 hours in the 2008-09 school year. For example, students who

were taught by teachers who participated for the mean number of hours (27.5 hours) scored

4 Due to the nested structure of the data, with students grouped within teachers, HLM was used to examine the relationship between the number of hours of TAH training a teacher received and their students’ history assessment scores.

44

approximately 15.6 points lower, on average, than students whose teachers participated in the

maximum hours of the training (39.5 hours), and approximately 14.3 points higher, on average,

than students whose teachers participated in the minimum number of hours of training (16.5).

Coupled with the finding that students who were taught by teachers who participated in

the TAH training tended to score higher on the History CSTs than students whose teachers did

not participate in the training; this indicates that not only is the TAH program associated with

increases in students’ history achievement, but more hours of training is associated with larger

student score increases.

However, caution is warranted when interpreting this finding. Because we were not able

to use an experimental design (i.e., teachers chose to participate in the TAH training rather than

being assigned to a treatment or control condition) and because teachers were able to control the

extent of their participation (i.e., the number of hours they participated), it is possible that the

correlation between hours of participation and student achievement may be attributed to a third,

confounding variable rather than to the TAH training. For example, it is possible that highly

motivated teachers tended to attend more hours of TAH training, and due to their increased

level of motivation rather than the effects of the TAH training, these teachers’ students may

have earned higher History CST scores.

45

SUMMARY AND CONCLUSIONS

So far, at the end of the second year of this three-year project (beginning in summer

2008 and ending in spring of 2011), 81 teachers from grades 8 and 11 have participated in the

Teaching American History (TAH) grant. Hours of participation per teacher range from 9 to

slightly more than 120, with an average of 47 hours. Teachers have many opportunities to

participate, both during the summer and school year. Participating teachers’ years of experience

vary widely but nearly 80% have been teaching for at least three years.

The evaluation examines changes in teacher knowledge, classroom practice, attitudes

and response to their participation in the program. It also addresses student response both in

terms of their self-reported interest in and attitudes about their history classes and their

academic achievement. Results indicate significant changes in both teachers and their students

as a result of their participation in this professional development program.

Content knowledge. Participating teachers’ content knowledge is assessed in three

ways: (a) results of pre-post multiple choice tests, (b) results of pre-post document based

exercises (DBQs), and (c) self-reported knowledge gains on the annual teacher survey. In four

out of five multiple exams administered during the first two years of the project teachers have

demonstrated statistically significant growth between pre and post administrations. In all but

the very first DBQ, statistically significant growth has also been documented between pre and

post administrations. This suggests that teachers have gained both historical content knowledge

and pedagogical knowledge, as measured by their descriptions of lessons based on the primary

documents provided in the DBQ assessments. Survey results confirm that teachers believe they

have gained both content and pedagogical knowledge: 85% reporting having learned effective

history instructional strategies, 76% report having both increased their American history content

knowledge and their ability to assist their students in understanding historical documents and

texts, and 73% report greater familiarity with current historical research.

Classroom Practice. Evidence of changed classroom practice includes examination of

response to survey items and some limited observation of the 8th grade teachers’ lesson study

lessons and debriefs. Items on the teacher survey ask about five specific classroom practices

46

that are particularly relevant to the TAH professional development program. Three of these five

practices demonstrate statistically significant growth in reported frequency of practice: (a)

frequency of history-based lessons increased by 22.5%, (b) collaborating with other teachers in

lesson design increased by 29%, and (c) having students examine primary sources increased by

23%. While between 26-36% of the teachers reported frequently employing these strategies

prior to the training, now 53-59% say they do so in their classes. Teaching literacy skills and

writing also increased in frequency but most (40-50%) of the teachers were already engaging in

these practices prior to their TAH participation.

One component of the TAH program in the past year addressed teacher practice by

engaging 8th grade history teachers in lesson study, permitting increased collaboration in lesson

design and encouraging shared reflection on teaching practice and student response. Evaluators

participated in the initial teaching of each lesson study group’s collaboratively designed lesson

and their debrief and also included survey questions in the spring survey asking these teachers

about their experience. While a single observation of the groups and their lessons is certainly

not an adequate picture from which to judge the impact of teachers’ collaboration in this facet of

the program, it did serve to illustrate how the teachers worked to engage their students in a

meaningful historical exercise and reflect on the success of their efforts with their colleagues.

Teacher response on the survey indicated high levels of satisfaction with the collaborative

nature of the work; for example 81-88% of the teachers indicated enjoying collaboration and

expecting to continue to seek opportunities for collaboration in the future. Most (81%) agreed

that lesson study provided an opportunity to try a new instructional approach and reflect on their

own instructional practices. Most (75%) also agreed that they learned about different

instructional approaches and increased their understanding of how to guide their students in

historical investigations as a result of their lesson study participation.

Students’ Self-Reported Response. A subset of the participating teachers (eight from

8th grade and six from 11th grade) administered an evaluator designed student survey in their

classes in the spring of 2010. A total of 957 students responded. Since the survey was a

voluntary activity, evaluators cannot calculate response rates or representativeness of this

student sample. However, with such a large number of students responding, it does provide

some insight into 8th and 11th grade students’ perspectives on their American history class and

47

teacher. Students were generally quite positive about both. Most (60-80%) students generally

expected to get good grades in history, enjoyed working with other students in their class on

history projects/exercises, and reported finding history interesting and important. A large

proportion of the students (63% of 8th grade students and 79% of 11th grade students) report

finding history more important after taking this year’s American history class. Most (over

70%) of the students reported finding their teacher’s explanations useful, liking to have history

related to current events, and liking to discuss historical events. Many (64% of 8th graders and

83% of 11th graders) reported that history had “come alive” for them this year. Generally, the

older 11th students had more positive responses on all survey items, compared to the younger, 8th

grade students

Student Achievement Analysis Results. Due to the release date schedule of statewide

standardized test results data by the California Department of Education (CDE), this year’s

report only examines the student achievement results for the first year of the TAH program, the

2008-09 school year, when only 8th grade teachers were participating. Next year’s report will

address achievement results for both 8th and 11th grade students. In our analyses, 8th grade

History/Social Science (HSS) California Standards Test (CST) and English Language Arts

(ELA) CST results were examined in several ways, including in contrast to the students in all

other non-participating classrooms in the county and as their achievement relates to the level of

their teachers’ participation in the program (e.g. hours of participation). In most analyses, the

prior year’s ELA CST score or proficiency placement level was used as a statistical control.

After comparing the achievement of students of TAH participants and non-participants

on the 2009 History CST, the data seem to point to the following conclusion: Teacher

participation in TAH is associated with higher student achievement on the 2009 History CST,

especially among students who entered with a 2008 CST ELA performance level of Basic or

above.

Students of teachers who participated in both the summer Gear Up and TAH

professional development during the school year outperformed both students in

other TAH teachers’ classrooms and those in non-participating teachers’

classrooms on the 2009 History CST. This difference in achievement was only

48

seen among students who entered with a 2008 CST ELA performance level of

Basic or above

Students of TAH teachers at schools with at least 40% of students eligible for

free or reduced-price meals outperform their peers at similar schools on the 2009

History CST. However, this difference in achievement was seen among students

who entered with a 2008 CST ELA performance level of Basic or above.

Among TAH participants, the average student History CST performance level

significantly increased from the year prior to participating in the program (2007-

08) to the 2008-09 school year. There was no similar increase for the students of

non-participating teachers. This change in average achievement, however, was

only seen among students who entered with a prior year CST ELA performance

of Below Basic, Basic, or Proficient.

Finally, HLM analyses, controlling for the nested structure of the data (e.g.

students within classrooms), finds significant associations between teachers’

hours of program participation and their students’ performance on the History

CST, showing, on average, a 1.3 point scale score increase for every hour of

teacher program participation.

The analysis of student data indicates that teacher participation in TAH may provide

opportunities for increased achievement among students with English-Language Arts skills that

are average or above. The lack of similar gains in achievement among students with lower ELA

proficiency may be due to the additional barriers these students face in accessing historical

documents and text. It is interesting that the lowest rated survey item, among the 8th grade

teachers, was “I better understand how to tailor my teaching to the specific needs of students,”

suggesting that the TAH training may not emphasize techniques to engage lower literacy

students.

Conclusions and Discussion. Program evaluations can seldom control for all the

potential sources of variation in examining or explaining the impact of a given program.

Instead, evaluations rely on triangulation of information from a variety of sources to gain

49

insights into program results. For example, it is impossible to confidently assign causality for

all of the changes in teacher and student knowledge, practice, perceptions, and achievement

discussed in this report solely to teachers’ participation in the TAH professional development

program. However, when we examine the cumulative results of all the data collection and

analyses presented, a fairly compelling picture emerges supporting the proposition that teachers’

participation has had important impacts on both their knowledge and practice, as well as on the

academic achievement in American history among their students. It may be that these teachers,

particularly those who have persisted and accumulated the most hours of participation, may be

particularly motivated to improve and might have found some other set of supports to do so had

TAH not been available.

Nonetheless, it is clear that both the teachers and their students have benefited from the

TAH grant activities. By all available measures, teachers demonstrate improved content and

pedagogical knowledge. Students, particularly those with at least basic level literacy skills,

show improved test scores that are directly associated with the levels of their teachers’

participation in the TAH activities. Further, students self-report increased levels of interest in

and understanding of historical events as a result of their past year’s placement in participating

teachers’ classrooms, suggest that at least for those students who responded to the survey,

teachers’ student engagement skills and strategies have met with some success. Since teachers

and their students are spread across numerous districts it seems unlikely that the improvements

noted are due to some other district-level professional development or instructional

improvement program.

There are some areas of concern. For example, although teachers’ content and

pedagogical knowledge assessments do demonstrate statistically significant growth, for the

most part, these improvements are modest. It is unclear whether this reflects actual relatively

low levels of teacher growth, deficiencies in the assessment instruments, or lack of careful

attention among teachers in completing the assessments. The fact that teachers’ self-reported

increases in knowledge and practice appear fairly strong, coupled with the lack of association

between content/pedagogy assessment improvements and other outcome measures, makes this

an area of some concern. It is encouraging to see positive growth in student achievement

among the 8th grade students of participating teachers, particularly those serving high poverty

50

populations. However, it is concerning to note that these improvements do not appear to reach

those students who enter American history classrooms with lower levels of English

achievement.

Next year’s report, the final report for this project, will include an additional year of data

for both teachers and their students. This will permit evaluation of whether the trends noted in

this report appear to be sustained and/or improved in the final year of the project. Further, the

additional year of data and expansion to include both 8th and 11th grade students will permit

more nuanced data analysis approaches to examine student achievement outcomes.

51

Table Of Contentseducation.ucdavis.edu/sites/main/files/file-attachments/... · Web viewAll 8th...

Documents

Transcript of Table Of Contentseducation.ucdavis.edu/sites/main/files/file-attachments/... · Web viewAll 8th...