The Evaluation of West The Evaluation of West Virginia’s Enhancing Education Virginia’s Enhancing Education

Through Technology Model Through Technology Model Schools Program (PR#: Schools Program (PR#:

S318A030014)S318A030014)Dale Mann, Ph.D.

Jonathan D. Becker, J.D., Ph.D.Charol Shakeshaft, Ph.D.

Interactive, Inc.

September 19, 2005

The TMS StudyThe TMS Study Research Questions:

– Over this school year, how much change in the use/integration of computers (by 4th and 5th grade teachers) was there?

– After the TIS specialists are done (June 2005), what will happen? Will teachers continue to use technology? (Year 2)

Using technology to study technology Experimental/control + pre-post 4-cell sampling framework (treatment/control

x digital/non-digital)

Preview of “Conclusions”Preview of “Conclusions”

Methods Technology Integration Specialists (PD) TMS program schools & teachers TMS students TMS program overall

Conclusions about MethodsConclusions about Methods

Teachers and students cooperated with the use of technology to study technology.

The (classroom) desktop metering and the pager-triggered web-surveys worked reliably.

The “digital data collection” procedures did not affect classroom practice.

Conclusions about the TISsConclusions about the TISs The TIS services were within the boundaries of the

TMS program. But, rather than a single, consistent program, the

variation in the TIS backgrounds and work arrangements delivered several program “interventions,” not a single intervention.

TISs did not become more proficient or more enthusiastic about technology over the year.

Over the course of the first year, TISs learned how to do their job and created helping relations with classroom teachers. They provided less direct instruction to their in-school colleagues, but built the necessary foundations for increased technology integration.

Conclusions about TMS Schools Conclusions about TMS Schools and Teachersand Teachers

TMS teachers had more technology-related PD than others.

TMS teachers were twice as likely to be using computers more for their own work than the control group (web-survey data).

TMS teachers reported more student use of technology and more consistent use over the school year than the control group (web-survey data)

TMS teachers report a broader array of student-related applications than the “reading”-heavy focus of the control group (web-survey + TAM data)

Conclusions about TMS Schools Conclusions about TMS Schools and Teachers (cont’d)and Teachers (cont’d)

TMS teachers reported more positive changes in more areas than the control group.

All teachers (TMS & control) were more likely to direct student use of computers if they had “outside PD”, “one-to-one help”, “release time for tech. PD”, and “access to online PD” – all potential features of the TMS program.

There are a number of areas of teacher proficiencies, attitudes and behaviors that did not change and that may be part of a program revision.

Conclusions about TMS studentsConclusions about TMS students

Students in TMS schools consistently report using computers more frequently than the control group (web-survey data).

Students in TMS schools report more use of the Internet to do reading or research than the control group (web-survey data).

Conclusions about the Conclusions about the TMSTMS program overallprogram overall

This is the first study to analyze more than self-report data about technology use – metered and random-interval web-survey data from teachers and students. Additionally, we do not yet have student achievement data. Therefore, we cannot conclude that the reported patterns are “high”, “exemplary”, “successful” or the opposite.

For the same reasons, we have not made policy or practice conclusions.

Unless the cadre of TISs can be staffed with people who are already expert with technology and with training adults, the first year of a turnkey sequence is more of a learning year for the TISs than a service-delivery year for teachers in schools.

2005-06 Methods2005-06 Methods We will follow 4th and 5th grade teachers who, last

year, had TIS assistance. Some will continue to have TIS services and that

contaminates the “turnkey” logic. Some 4th or 5th grade teachers will move to similar

assignments in other schools; we will not be able to follow them.

We need to document that a Year One “trained” teacher follows through and tries to communicate new capability to colleagues.

Attrition in both treatment and control teachers. Continued use of technology to study technology.

An Unanticipated QuestionAn Unanticipated Question

Following the TISs who work a second year in the same school (or new ones?): Are their second year services more “aggressive”, “direct”, “mature”?

That is central to the turnkey strategy but we did not know the issue would arise.

Similarly, “dependence” vs. “independence”: providing a service vs. building capacity.

Data Collection:Data Collection:

USING TECHNOLOGY TO MEASURE TECHNOLOGY

TREATMENT CONTROL(Blue Team)

DIGITAL

Meters+Pagers (w/ Web surveys)

Cheat Lake ES (Mon)Mason Dixon ES (Mon)Clay ES (Clay)Ellenboro ES (Ritchie)Harrisville ES (Ritchie)Doddridge ES (Doddridge)Burnsville ES (Braxton)Frametown ES (Braxton)Fairdale ES (Raleigh)New Haven ES (Mason)Fairplains ES (Wood)Madison ES (Wood)Franklin EC (Wood)Wilsonburg ES (Harrison)Woodsdale ES (Ohio)

Riverside ES (Mon)

Waverly ES (Wood)Kanawha ES (Wood)Neale ES (Wood)Valley Grove ES (Kanawha)Bridge ES (Kanawha)Clear Fork ES (Raleigh)North Pt. Pleasant ES (Mason)Worthington ES (Wood)Gihon ES (Wood)Lubeck ES (Wood)Van Horn ES (Harrison)Middle Creek ES (Ohio)

NON-DIGITAL

Pre + Post Paper/Pencil surveys only

Cedar Grove Community (Kanawha)Tennerton ES (Upshur)Welch ES (McDowell)Smithville ES (Ritchie)Creed Collins ES (Ritchie)Flatwoods ES (Braxton)Burnsville ES (Braxton)Stratton ES (Raleigh)Beckley ES (Raleigh)Mason ES (Mason)Jefferson ES (Wood)Blennerhassett ES (Wood)Martin ES (Wood)Tucker County ES (Tucker)

Richmond ES (Kanawha)French Creek ES (Upshur)Fall River ES (McDowell)

Emerson ES (Wood)Chamberlain ES (Kanawha)Valley Grove ES (Braxton)

Beale ES (Mason)Mineral Wells ES (Wood)Williamstown ES (Wood)McKinley ES (Wood)Davis Thomas ES (Tucker)

USING TECHNOLOGY TO USING TECHNOLOGY TO MEASURE TECHNOLOGYMEASURE TECHNOLOGY

Meters in desktops– Software runs in the

background 24/7– Records file activity only – Installed on classroom

computers only– Data e-mailed directly to

Interactive, Inc.– ZERO burden on

teachers

Pagers on teachers + web surveys – Free pager rings once per

day, every other week– When pager rings, or as

soon as convenient thereafter, teacher and one randomly selected student take short web-survey

– Pages are randomly scheduled during alternate weeks (not more than 5 per week)

Pagers + Web-SurveysPagers + Web-Surveys From November 22, 2004 through April 15 2005,

the pagers were activated 41 times. In total, Interactive, Inc. received 1,563 usable

pager-triggered web survey responses from 91 different teachers

The range in the number of responses from individual teachers is one to 40 (i.e. a couple of teachers have responded once, and one teacher responded to 40 pages).

We also received 1,311 usable pager-triggered web survey responses from students in 78 different classrooms.

TrueActive MonitorsTrueActive Monitors Starting on October 1, 2004, TAM was

installed on approximately 287 computers.

Interactive, Inc. ultimately received data at one point or another from 205 computers in 24 (of 26) schools.

As of April 1, Interactive, Inc. received over 33,000 e-mails with data from TAM including over 550,000 files activated.

OTHER DATAOTHER DATA Technology Integration Specialists (TIS)

– Each of the 18 technology integration specialists completed a paper-based questionnaire at the beginning of the school year; and 16 completed an end-of-year online QRE.

– Over 42 weeks, Interactive, Inc. received right around 600 activity logs from the TISs.

Teachers– Baseline questionnaires were sent to 227 teachers;

they were returned by 199 teachers, for a response rate of 88%.

– A similar response rate was achieved for the end-of-year QRE.

Preliminary FindingsPreliminary Findings

Documenting the Treatment: Documenting the Treatment: Who are the TISs?Who are the TISs?

10 were classroom-based elementary school teachers and 2 were secondary school teachers

3 were EETT-funded technology integration specialists (serving different schools than in AY 04-05)

2 were Title I literacy specialists and 1 was a librarian

Documenting the Treatment:Documenting the Treatment:A Range of AvailabilityA Range of Availability

ONLY 5 OF THE 18 TISs were contracted to work full-time (designated as at least 35 hours per week) in a single school

The other TISs were all part-time in one or more schools; The range of arrangements is important.

Most of the TISs who worked less than full-time as a TIS worked in another capacity within one of the schools they served as a TIS. For example, one TIS was a ½ time TIS and ½ time librarian in the same school.

Documenting the Treatment:Documenting the Treatment:What did they do?What did they do?

During the early part of the school year, TIS’s devoted much of their time to setting up computers, installing software and trouble shooting. Group sessions were used to introduce software to teachers.

The rest of the year was spent doing mostly small-group work with teachers and individual mentoring/training.

The majority of the professional development occurred during school hours, not before or after school.

Web-survey ResponsesWeb-survey Responses

At the time the pager was activated, were students At the time the pager was activated, were students using computers?using computers?

(% of “Yes” Responses)(% of “Yes” Responses)

Teachers Students

Treatment 24.5 31.1

Control 20.8 25.2

Teacher responses over timeTeacher responses over time

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

0.5

Control

Treatment

Student responses over timeStudent responses over time

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0.45

0.5

Control

Treatment

““At the time the pager was activated, At the time the pager was activated, were YOU (the teacher) using were YOU (the teacher) using

computers?”computers?”

% of “Yes” Responses

Treatment

(weighted n=933)

22.2

Control

(weighted n=630)

9.2

““What were you (the teacher) doing?”What were you (the teacher) doing?”TMS % Control %

Word Processing 28.4 33.3PowerPoint 3.4 0Spreadsheets 9.8 3.5MS Publisher 1.0 0Kidspiration 0 3.5Inspiration 0 0Compass / Riverdeep 3.4 3.5Internet research 26.0 15.8E-mail 3.9 19.3Other 24.0 21.1

SummarySummary When asked at random times throughout the

school year if they were using computers themselves, teachers who had been working with Technology Integration Specialists were much more likely to report that they were doing so (22% vs. 9%).

Furthermore, when using computers, those teachers were much more likely to be doing curriculum-related work (i.e. productivity software).

Students in schools where there was a TIS were only slightly (but statistically significantly) more likely to be using computers.

Teacher QRE DataTeacher QRE Data

Time on Computers (labs)Time on Computers (labs) SCALE: 1= not at all; 2= <15 minutes; 3= 15-45; 4=

46-90; 5=90+

 

 

Teacher End-of-Year Reports of Student Computer Use In Labs:Mean Scores by TMS vs Control and Type of Data Collection

TMS Schools Control Schools

Digital Data Collection

Non-Digital Data Collection

Digital Data Collection

Non-Digital Data Collection

2.96  

2.98 2.54 2.79

Time on Computers (classroom)Time on Computers (classroom) SCALE: 1= not at all; 2= <15 minutes; 3= 15-45; 4=

46-90; 5=90+

 

 

Teacher End-of-Year Reports of Student Computer Use In Labs:Mean Scores by TMS vs Control and Type of Data Collection

TMS Schools Control Schools

Digital Data Collection

Non-Digital Data Collection

Digital Data Collection

Non-Digital Data Collection

2.81  

3.12 2.68 3.04

““Next September, I will be better able to use Next September, I will be better able to use technology in my classroom than I was last technology in my classroom than I was last

SeptemberSeptember””

Scale: 1= strongly agree to 4= strongly disagree

TMS (digital)

TMS (non-digital)

Control (digital)

Control (non-digital)

2.0 2.1 2.5 2.5

Next Sets of AnalysesNext Sets of Analyses

TrueActive meter data Technology use/integration as a

function of school and TIS characteristics (regression analysis and multilevel modeling)

STUDENT ACHIEVEMENT (WESTEST) ANALYSIS (Sept. 05)

QUESTIONS & ANSWERSQUESTIONS & ANSWERS

Contact UsContact Us

Interactive, Inc.61 Green Street

Huntington, New York 11743phone: 631 351 1190

fax: 631 351 1194interinc@aol.com