QUALITATIVE PROGRAM EVALUATION RESEARCH: EMPLOYING THE COMPLEMENTARY ANALYSIS RESEARCH MATRIX APPLICATION (CARMA)

LeAnn Grogan Putney, PhDEducational Psychology & Higher Education

Professor and Department ChairUniversity of Nevada, Las Vegas

Joan Wink, PhDProfessor Emerita

California State University, Stanislaus

joan@joanwink.com

EQRC – Las Vegas, NVFebruary 11, 2014

putneyl@unlv.nevada.edu

YOU DO KNOW THESE HEAPS OF DATA NEED MORE ANALYSIS, RIGHT?

…I TOLD YOU TO USE CARMA TO GET OUT OF THE HEAPS AND OVER TO CONCEPTS!

3

Vygotsky’s Conceptual Development – AKA, THE HEAPS

Heaps Complexes Concepts

Generalized understandingsGroup similar items

Connections among items

4

Critical Action/Complementary Analysis Research Matrix Application (CARMA)

CARMA helps you move from Heaps of Data to Complexes of Relationships toConcepts related to a theoretical

perspective or producing a grounded theory

Putney, L. G., Wink, J & Perkins, P. (2006). Teachers as Researchers: Using the Critical Action Research Matrix Application for Reflexive Classroom Inquiry. Florida Journal of Teacher Education, Vol IX, p. 23 - 35.

NOTE-TAKINGEXPECTATIONSWho are the intended participants?Who is supposed to provide what service?How are participants to be served?What will be produced in the activity?

NOTE-TAKINGEVIDENT IMPLEMENTATIONWho are the actual participants?Who is actually providing what service?How are participants actually served?What is actually being produced?

NOTE-MAKING / ALIGNMENT OR DEPARTURE? Compare/contrast expected with evident using Action Researcher Interpretations For each question do you see alignment or departure?oWhat might be the implications?

NOTE-REMAKING / RECOMMENDATIONS/Modify or maintain program activity?In what ways?For which participants?With what intended outcomes?

6

Complementary Analysis Research Matrix Application [CARMA]

7

Complementary Analysis Research Matrix Application (CARMA)

Content of fieldnotes NoteTaking

Descriptions of setting/people/activities Mapping of physical space Direct quotes or substance of what was said

Note Making Observer comments in margins –

interpretations & questions Note Remaking

What did you learn – maintain or modify? Reflective Cycle – transformative stage

8

Potential Uses for CARMA

Program evaluation Applied classroom research

Self study (action research) or Team approach with classroom teacher

Educational research School wide behavioral practice Single-sex based classes Comparison of several schools – English Language

Learners in high ranking schools for “best practices”Policy analysisCritical incident study

9

Scale of use

Micro

Macro

Lesson Classroom

School District Community

State National

Continental

Global

Unit/Project/Lesson/Guest speaker (Teacher’s, Speaker’s, and Students’ expectations)

Classroom project/Tolerance

School-wide project/Action Research – 111 teachers

District-wide project/Empowerment

Community-wide/Hospice – EPY 716 2008/Critical Incident dissertation

10

Scale of use

State-wide project/Proposition 227 in California- dissertation

National-level project/No child Left Behind policy analysis – dissertation

Continental level/Quality Assurance in European Higher Education Area - policy analysis – dissertation

Global level/recommendations for future research studies

Micro Macro

Lesson Classroom School District Community State National Continental

Global

PARTICIPANT DISCUSSIONFOR YOUR OWN RESEARCH, WHAT WAS YOUR

INTENDED VS ENACTED?

DID YOU ENCOUNTER WHAT YOU EXPECTED?

DO YOU NEED TO COLLECT MORE DATA OR ARE YOU SEEING REPEATING PATTERNS?

11

CARMA APPLIED TO YOU

12

Complementary Analysis Research Matrix Application (CARMA)

WHAT - Tool for critical evaluation, to help the researcher/evaluator to describe and interpret what is being accomplished in the setting, and in what ways the cultural space is being

utilized by the participants, by measuring program expectations, against

evident implementation of practices and outcomes,

from the perspectives of all levels of participants

(Putney, Wink & Perkins, 2006).

13

Complementary Analysis Research Matrix Application (CARMA)

WHY - Roots of CARMA:Vygotskian view of life as a social

construction, steeped in the context of our cultural experiences to evaluate whether the context of a

program is meeting the needs of the constituents, we would have to determine what the constituents and program administrators are socially constructing through the program implementation

.

14

Complementary Analysis Research Matrix Application (CARMA)

WHY - Roots of CARMA:Ethnographic perspective of

learning from people about their cultural patterns of interaction (Agar, SBCDG) people, who work in a common setting

for individual yet related purposes, begin to co-construct norms and expectations for being together in their local time and space

.

15

Complementary Analysis Research Matrix Application (CARMA)

HOW – Observations and Interviews of constituents

Fieldnotes – written account of observations Continuous notes, jottings, head notes,“cooked up”

TEACHER’S REFLECTIONS ON HER OWN PRACTICE

16

Example from action research

Putney, L. G., Wink, J & Perkins, P. (2006). Teachers as Researchers: Using the Critical Action Research Matrix Application for Reflexive Classroom Inquiry. Florida Journal of Teacher Education, Vol IX, p. 23 - 35.

Step 1 – NoteTaking Notations on classroom expectations

Step 2 – Notetaking from observations what is evident

Step 3 – results and conclusions after contrastive analysis

20

Step 4 - Recommendations - decisions