Mosvold-Fauskanger, AERA 2009 paper

8/14/2019 Mosvold-Fauskanger, AERA 2009 paper

1/36

Challenges of translating

CHALLENGES OF TRANSLATING AND ADAPTING THE MKT MEASURES FOR

NORWAY

Reidar Mosvold and Janne Fauskanger

University of Stavanger, Norway

1


2/36


Abstract

This paper reports on the process of translating the U.S. measures of MKT into Norwegian. The

main questions addressed in the paper are (a) what challenges were encountered in the process of

translating the U.S. measures into Norwegian and adapting them to a Norwegian context? (b)

which of these challenges are general, and which appear to be specific to the Norwegian culture?

The PISA Technical Report supports the idea of using double translation, and we have used

double translation with extensive documentation of the changes that were made during the

translation process. These changes were placed in different categories, and we had to put extra

emphasis on issues concerning the translation and adaptation from English to Norwegian.

2


3/36


Challenges of Translating and Adapting the MKT Measures for Norway

In the Learning Mathematics for Teaching (LMT) project, measures were created in order

to analyze teachers Mathematical Knowledge for Teaching (MKT). In this paper, we present

some of the challenges involved in our attempt to translate and adapt these measures for use with

Norwegian teachers. The measures were originally created for use in a U.S. context, and a

number of differences between the two countries contribute to increasing the difficulty of using

the U.S. measures in Norway. At this point, we have carried out the translation as well as a pre-

pilot study. The main focus of this paper is to address the following research questions:

What challenges were encountered in the process of translating the MKT measures into

Norwegian and adapting them for use in Norway?

Which of these challenges are of a general nature, and which appear to be specific to the

Norwegian culture?

We start by presenting the theoretical background for our project, followed by a short

presentation of the Norwegian school system. In the methods section, we present the methods

that were used in the translation process as well as in the pre-pilot. This is followed by a

presentation of results and a discussion. Towards the end of the paper we present some

concluding remarks and suggestions for the road ahead.

Theoretical background

Research from the last 15 years indicates that the mathematical knowledge of many

teachers is dismayingly thin (Ball, Hill, & Bass, 2005, p. 14). When analyzing 700 1st and 3rd

grade teachers (and almost 3000 students), researchers found that the teachers knowledge had an

3


4/36


effect on the students knowledge growth (Hill, Rowan, & Ball, 2005). Stigler and Hiebert

(1999, p. 10) claim that: Although variability in competence is certainly visible in the videos we

collected, such differences are dwarfed by the differences in teaching methods that we see across

cultures. But even though research indicates that teachers knowledge might have a positive

influence on students learning, it is not obvious what the content of this knowledge is. There are

also no clear guidelines for what to focus on in in-service education.

Our study focuses on Norwegian teachers mathematical knowledge for teaching, and it

is closely related to the LMT project

1

. Theoretically, it follows Shulmans (1986) efforts to

define the theories concerning subject matter content knowledge and pedagogical content

knowledge. The categorization of the various components of teacher knowledge has evolved

from Shulmans original proposal, where he distinguished between subject matter knowledge

(SMK), pedagogical content knowledge (PCK), and knowledge of curriculum. In the LMT

project, this model evolved into a model of mathematical knowledge for teaching (MKT).

1 For more info about LMT, see http://sitemaker.umich.edu/lmt/home

4


5/36


Figure 1. Model of MKT (based on Hill, Ball, & Schilling, 2008).

Hill, Ball and Schilling (2008) point to a discussion about how effective teachers have a

unique kind of knowledge. This includes knowledge about students mathematical ideas and

thinking. This domain of teacher knowledge has been identified in the U.S., but there is a

possibility that it might differ from the domain of knowledge that is held by effective teachers in

Norway.

The MKT measures have been developed over several years, and the research team at the

University of Michigan has spent a lot of time and money on this. It would therefore have been

of great interest if we could build on their efforts and use the same material with Norwegian

teachers. Similar attempts have been done in Ireland, and Delaney (2008) points to some

possibilities as well as some problematic issues. The process of translating these items is far from

straightforward, and there are several issues to be aware of when going into this (cf. Delaney et

al., 2008; Mosvold, Fauskanger, Jakobsen, & Melhus, 2009). In our Norwegian project, we do

not aim at comparing the knowledge of U.S. and Norwegian teachers. The measures were not

built for that purpose. Our aim is to learn more about the mathematical knowledge for teaching

that Norwegian mathematics teachers have, and the knowledge they need in order to become

(more) effective teachers. Such information would be useful for pre-service as well as in-service

education.

Although it is not our intention to compare teachers in Norway with teachers in the U.S.,

it is necessary to investigate whether or not our translated and adapted measures work in a way

that will provide meaningful data about Norwegian teachers mathematical knowledge for

5


6/36


teaching. It is therefore necessary to go into an analysis of the scores in order to find out if the

items that were difficult for U.S. teachers are more or less difficult for Norwegian teachers. If

there are significant differences, we have to figure out whether these differences are related to

the translation process, to cultural differences, or to other aspects.

The Norwegian school system

The Norwegian school system is different from that in the U.S., and we point out some

issues here. Norwegian schools are divided into three main categories or levels:

Primary and lower secondary education (children aged 6-15)

Upper secondary education (three years after 10th year of lower secondary education)

Tertiary education (influenced by the Bologna Process)

In 2007, 616,388 children were attending primary and lower secondary education in Norway

(Statistics Norway, 2009). All public education is free of charge, and the Norwegian school

system has some important overall aims and precepts:

Education for all is a basic precept of Norwegian educational policy. Children must

have an equal right to education, regardless of where they live, gender, social and

cultural background or any special needs (MER, 2007, p. 5).

Based on these more overall aims, the goals and frameworks for Norwegian schools are then

defined (and decided) by the Norwegian Parliament and the Government. The Ministry of

Education and Research has a particular responsibility for carrying out the national educational

policy. National standards are ensured through curricula and framework plans along with laws

and regulations (MER, 2007).

When it comes to teacher education, one formerly thought that if teachers knew enough

6


7/36


mathematics, their teaching would be good and their students would learn mathematics. The

content of in-service education was therefore pure mathematics (Cooney, 1999). On the other

extreme, there appeared to be a consensus in Norway that it was possible to become a good

mathematics teacher without knowing any mathematics at all (Haaland & Reikers, 2005).

Nowadays, there are several worldwide attempts to pursue excellence in mathematics classroom

instruction, and the focus is moving away from the traditional in-service courses. Exemplary

lesson development is one example (Huang & Li, 2009), and there have also been recent

examples of studies that make use of an inquiry-based approach in Norway (Jaworski, Fuglestad,

Bjuland, Breiteig, Goodchild, & Grevholm, 2007).

The Ministry of Education and Research (KD, 2006) refers to TIMMS 2003, which

draws attention to the fact that Norwegian teachers education in mathematics and in

mathematics education is below the international average. Another issue is that Norwegian

mathematics teachers rarely participate in relevant in-service education (Pedlex, 2008; Grnmo

et al., 2004). It therefore appears evident that a focus on strengthening in-service as well as pre-

service education of mathematics teachers is necessary.

Mathematics is a compulsory subject for all pupils in Norwegian schools. In years 1-10,

the pupils are supposed to work on the following subject areas (Utdanningsdirektoratet, 2008):

7


8/36


When compared with the content of the MKT items that are used in our studies, it is important to

notice that algebra does not appear as a main subject area in years 1-4. (In our penultimate

curriculum, algebra only appeared in years 8-10.)Functions only appear in years 8-10.

Throughout elementary school (years 1-7), Norwegian pupils should have altogether 812

hours of mathematics. In lower secondary school (years 8-10), they have 313 hours of

mathematics1. Normally, teaching in Norway is organized in a combination of the teacher

addressing the whole class and individual work by the pupils (Haug, 2004; Bachmann & Haug,

2006; Alseth et al., 2003). Norwegian classroom research also indicates that the time that is

actually spent on mathematics in reality is much lower than the hours presented above (Haug,

2006; Skorpen, 2006).

The Norwegian Ministry of Education and Research recently published a strategy for

teachers in-service education: Kompetanse for kvalitet (in English: Competence for quality)

(KD, 2008b). They underline that knowledgeable teachers are important for students learning.

The argument is that high mathematical and pedagogical competence among teachers contribute

to better results among the students. The strategy document does, however, provide little

1 Norwegian lessons are normally organized in 45-minute units, although these numbers are presented in 60-minute

units

8


9/36


10/36


mathematics, which may vary across countries, and it is therefore necessary to include experts of

teaching in the process of translating and adapting the items.

After the translation process, an instrument should continue to measure the same

characteristics it was intended to measure (Geisinger, 1994). An important methodological goal

for translating the MKT measures into Norwegian therefore is to ensure equivalence at the level

of context and opportunity. Various terms are used in cross-cultural research to describe different

aspects of equivalence. According to Johnson (1998), the terms are not always well defined and

considerable overlap exists among them. An attempt to adapt the U.S. measures to an Irish

context (cf. Delaney, 2008) emphasized the need to establish whether the MKT construct is

equivalent in different settings. To focus on construct equivalence is thus an important aspect of

our validation process. Singh (1995) outlines six steps that contribute to construct equivalence,

three of which should be studied even before using the measures to collect data: functional

equivalence, conceptual equivalence and instrument equivalence (Singh, 1995; Delaney, 2008).

We therefore focus on these terms at this point. Functional equivalence relates to whether or not

the MKT construct serves the same function in Norway as in the U.S. In order for students to

acquire knowledge, the teacher must have some kind of knowledge related to teaching (in this

case, MKT). MKT is the mathematical knowledge needed to teach mathematics. This construct

has a universal function, and thus satisfies the requirements of having functional equivalence (cf.

Delaney, 2008).

Conceptual equivalence relates to the question of whether the construct of MKT means

the same in Norway as in the U.S. or not. To answer this question in an Irish context, Delaney

10


11/36


(2008) examined the construct more closely by studying the work of teaching in Ireland. He

compared that work to conceptions of the work of teaching that informed the development of

MKT. Delaney also studied literature about the construct, and he analysed items based on the

construct. He found relatively minor differences in his analysis. One possible explanation might

be that these two countries share a common language. This could make it easier for ideas and

conceptions about teaching to travel back and forth between the U.S. and Ireland. Norway and

the U.S. do not share a common language, so it is possible that more differences may emerge if

the tasks that informed the MKT were compared to tasks of teaching in Norway. Since we have

to take into account the added complexity of a different language, attempts to ensure conceptual

equivalence will be important in our work, but is not included at this point

Instrument equivalence might be related to both the format and the contents of the items.

If the multiple-choice items are equally interpreted in Norway and the U.S., we have instrument

equivalence (cf. Delaney, 2008, referring to Singh, 1995).

When translating a set of items into a different language, it is important to focus on what

Pea (2007) and others call linguistic equivalence. The translation needs to be of high quality. In

our case, there is a strong connection between what some researchers refer to as linguistic

equivalence and what others (cf. Singh, 1995) refer to as functional equivalence, conceptual

equivalence and instrument equivalence. In order to assure linguistic equivalence, we used a

double translation procedure (Adams, 2005). The translation of the items took about half a year.

Towards the end of this period, we had a working seminar where the translations were finished.

In this seminar, we worked in pairs (or sometimes three) and translated all the items. Two groups

11


12/36


13/36


When everyone had finished the test, we had a small break before the focus group interview

started. In the focus group interviews, we asked the participants to give general comments about

the test. Did they find the items difficult? Were there some elements of mathematical knowledge

for teaching (as they have experienced it) that they thought were missing from the test? Did they

have comments regarding the format of the questions? After a round of such more general

questions, we went through every item in the test, one by one, and asked them to make

comments. The focus-group interviews were recorded with a digital video camera and an digital

audio recording device, in order to provide ourselves with a material that could be used for

analyzing more than the discussion alone (e.g. their use of gestures).

One of the groups consisted of three newly certified teachers, who were students in a

masters program in mathematics education. All three were female students. The second group

consisted of two experienced teachers, one man and one woman. They both work in elementary

school, but the woman also has spent some years as a teacher in lower secondary education. The

male teacher finished his teacher education 25 years ago, and he has worked a lot with students

with special needs. The female teacher also has 25 years of experience, and she has been a

teacher in lower secondary school for several years. She only has a small unit of mathematics in

her teacher education (15 ECTS in our terms). The last 10 years, she has worked exclusively

with students in the first years of elementary school.

These teachers were selected because we knew them, and we believed that they could

provide interesting feedback for this phase of our study. We also thought it would be interesting

to see examples on how both experienced and inexperienced teachers might react to the

13


14/36


measures.

Results and discussion

In this part of our paper, we will present and discuss the results from our translation

process as well as from the pre-pilot.

Issues regarding translation

Throughout the translation process, we carefully documented all changes that were made

to the items (other than direct translation from U.S. English to Norwegian). This was done

because we suspect that these changes might influence the teachers responses to the items.

Delaney and colleagues (2008) report on a similar study that was carried out in Ireland, and they

summarized their changes in the following categories:

1. Changes related to the general cultural context

2. Changes related to the school cultural context

3. Changes related to mathematical substance

4. Other changes

Delaney and colleagues (2008) recommended their own results as working guidelines for others

who attempt to adapt the items, so we decided to use these categories in our translation process

as well. They included altering spellings to reflect differences between American and British

English in category 1 above (changes related to the general cultural context), but we decided to

have the translation from U.S. English to Norwegian as a separate category.

The translation and adaptation from American English into Norwegian was far more

complex than the process of adapting the items for use in Ireland (as described by Delaney,

14


15/36


16/36


a) De ignorerer plassverdi/posisjonsverdi.

b) De ignorerer desimalkomma.

c) De gjetter.

d) De har glemt at det fins tall mellom 0 og 1.

e) De gjr alle feilene ovenfor.

We discussed several issues in relation to the translation of this item. First, Norwegian

students are referred to aspupils (oreleverin Norwegian) as long as they are in compulsory

school, andstudents when they enter university. We also changed the name from Mr. Hayes to

Hans, which is a common Norwegian first name. In Norway, it is common for pupils and

colleagues to address teachers with their first name only. This might vary somewhat according to

the teachers age and the level in which they teach, but in lower secondary school the pupils

would normally address their teacher as Hans rather than Mr. (Hans) Hayes. If we decide to keep

the more formal American setting, most Norwegian teachers would find this different from what

they are used to, and they might therefore not experience this as a familiar setting. The result of

this might be that they were distracted from the substance of the question. When making changes

from the American names like Mr. Hayes and Ms. Wilson to more common Norwegian first

names like Hans and Marianne, we are also adding a potential complexity to the item in that it

becomes more difficult to distinguish between the teacher and the pupils in the problem context,

since both are referred to by their first names. We therefore had to change some of the items and

sometimes include some extra information in order to clearly distinguish between the pupils and

the teacher. When making these changes in names, we were conscious about preserving the

teachers gender. When changing the first names (of pupils) to names that are more common in

Norway, we tried to find names that were somewhat similar, but this was not always done.

Another issue is that the original sentence referred to the students in Mr. Hayes class,

16


17/36


and you are not supposed to refer to a class of students in Norway. In 2003, the Norwegian

Education Act (Opplringsloven) 8-2 (1998) was changed into a statement that students can be

organized in groups according to their needs. Prior to this, the law stated that Norwegian students

should be organized in classes. Schools are still allowed to organize their students in traditional

classes, but the Education Act no longer use this term, and most official documents refer to

groups of students rather than classes (KD, 2008a). We therefore decided to translate class into

group, or simply rewrite it somewhat. As described above, we have added a category about

political correctness to Delaneys categories, and changing class into group is one example. This

decision to use the word group instead of class in Norwegian schools might appear trivial, but

there is more to it than what we have described above. It appears that a large number of teachers

continue to use the word class, although the Ministry has decided to avoid it, and for these

teachers the wordgroup as a replacement to class could be both confusing and misleading. If we

decided to go for the traditional term class, which is no longer the officially correct term, we

would probably be faced with a large number of teachers who would argue that our measures

were not up to date, and not in line with the official guidelines. In order to be politically correct,

we have chosen to rewrite the sentences that originally referred to class and usepupils (elever)

instead. We could have usedgroup instead, but that could lead to confusion in some instances,

because the same word is also used when we refer to group work. Sometimes we therefore ended

up rewriting the item somewhat. For example: Mr. Hayess class is translated into Elevene til

Hans (English: Hanss pupils). Another example we have chosen to put in this category is use

of the verb to learn. In a Norwegian context, we normally refer to learning as the outcome rather

17


18/36


than the process. As a result, we find it inappropriate to say that Mr. Alders students are

learning about..., since we cannot know if they have actually learned it. In items that refer to the

learning process, like in the example just mentioned, we therefore decided to rewrite it

somewhat. A Norwegian translation would be: Elevene til Anders arbeider med... (in English:

Andys students are working with...).

The passage about putting decimals in order was also discussed. It is more common to

sort (sortere in Norwegian) numbers than to put them in order. We also talk about decimal

numbers (desimaltall) rather than decimals in Norway. To make this passage sound better in

Norwegian, we moved some of the information from the second sentence to the first. As a result,

it seems as if Mr. Hayes students only worked with ordering decimals from least to greatest,

whereas the original U.S. idea might have been that they worked with ordering decimals in

different ways, although the students in this particular example ordered the numbers from least to

greatest. The first sentences in the stem of this item were difficult to translate directly into

Norwegian, and we decided to rewrite them somewhat. When doing this, there is always a

possibility of interpreting the sentences in a way that has removed or added information to the

item.

In Norway, we use a decimal comma rather than a decimal point, and since comma was

used to separate the different numbers that were presented, we had to change this to avoid

confusion. One possibility could be to represent the numbers like this: 1,1 - 12 - 48 - 102 - 31,3 -

0,676. From a linguistic point of view, this might be a proper solution, but in a mathematical

setting there might be a danger of confusing the - with a subtraction sign. We also discussed the

18


19/36


possibility of using semi-colon instead of comma to distinguish the numbers, but we decided that

this would result in too much clutter. We therefore ended up representing the numbers separated

by extra space. In addition, we had to change .676 into 0,676, because decimal numbers lower

than one are never written without the zero in Norwegian.

In the alternative solutions, we spent some time discussing alternatives a) and d). In a),

there is a reference to place value, and we might use the similar wordplassverdi in Norwegian.

Several teachers would rather prefer to useposisjonsverdi instead, and we decided to include

both alternatives to avoid confusion. Both these words mean the same, and they do not add any

information to the item. Alternative d) was more problematic to translate. When Americans talk

about forgetting your (or their) numbers, this is hard to translate directly into Norwegian. Our

translation therefore had to be an interpretation rather than a direct translation. After some

discussion, we agreed that the meaning of this sentence must be that the pupils have forgotten

that there are numbers between 0 and 1. Another interpretation might be that they did not know

this, and a translation into Norwegian might then be:De kan ikke tallene mellom 0 og 1 (They

dont know the numbers between 0 and 1). Such a translation might, however, indicate that the

pupils have never been taught this, and we believe that this is not the correct understanding of

this alternative solution. We ended up with the following translation:De har glemt at det fins tall

mellom 0 og 1 (They have forgotten that numbers exist between 0 and 1).

The mathematical language used in Norwegian schools of course differs from the

language used in schools in the U.S. In most cases precise translations of the terms were

possible, but the mathematical language that is used in Norwegian schools is often translated into

19


20/36


21/36


It is interesting to notice that this term was so unfamiliar to these masters students, and it was as

a result of their suggestion that we decided to add an explanation to this particular term. This is a

significant change to the original, and it is likely that this item will be somewhat easier for the

Norwegian teachers because of this.

The multiple-choice format

The multiple-choice format has not been widely used in Norwegian schools and might be

unfamiliar to the Norwegian teachers. This is an issue that should be discussed, because it may

cause validity problems. We have seen indications that this might be changing in Norway, and

this appears to be related to the increased use of digital tools in particular. It is conceivable that

in a culture where multiple-choice formats are unfamiliar, one may have to change the format.

But changing the format may be problematic as well, because it could influence the items level

of difficulty, and it might also make scoring the items more difficult (Delaney et al., 2008). It

could also make the items more or less discriminating or change how effectively they measure

the underlying constructs. We have decided to keep the multiple-choice format for now and

evaluate the matter after the pilot study. But our pre-pilot study points our attention to the format,

and we have to be prepared to change it.

When interviewing the three masters students, we started by asking them how they felt

about doing a multiple-choice test. One of them said that the only experience she had with this

format was in connection with the theoretical exam she had to take in order to get a driving

license. She found the multiple-choice format difficult and said:

interview. Likewise, RM are the initials of Reidar Mosvold. When () is used in the transcripts, it indicates that

something has been omitted, and when [something] is marked like this, it represents a comment or addition by us.

21


22/36


You become uncertain, because everything is kind of similar, partly right. And then

you have to choose something. I think its easier when you can simply make your

own answer, instead of having to choose some pre-produced [answer].

One of the other students did not agree. She said:

I have nothing against multiple-choice (...) not in these kinds of tasks and surveys,but if it occurs when I am supposed to sit down and produce something on an exam,

I wouldnt be so happy about it. Because then I feel that I sort out the possible

answers, and I would have thought in a different way, or done things differently

The students were told that the reason why we conducted the pre-pilot study was to test the test

and not to test their knowledge. When we asked them if they would look at the format in a

different way if they were teachers taking the test, they all said Yes. One of them stated the

reason for the Yes by saying: You are kind of forced into someone elses way of thinking.

The other students said mm to indicate their agreement.

The interview continued by discussing each of the items. In this discussion especially one

of the students had statements showing that she was not comfortable with the multiple-choice

format. She said:

Multiple-choice makes me [uncertain], like everything is right (laughter) ... No, I

wont [do that].

What she said also indicates that the format controlled her response. When she was explaining

one of her responses she started to laugh and said: I guess it was because I didnt think there

would only be ones. The idea that there would be a pattern, or that the correct alternatives

would be spread, therefore appeared to have an impact on her response. This is a problematic

issue, and it might indicate that the responses that the participants make not only illustrate their

thinking or knowledge about the issues described, but also their anticipation of how the correct

22


23/36


alternatives are distributed. If they were more familiar with multiple-choice tests, this might not

have been an issue.

Are the MKT measures suitable for the Norwegian tradition?

Both the experienced teachers and the masters students suggested that some of the items

would be too difficult for Norwegian teachers, especially for the teachers working at the lower

grades. One of the masters students repeatedly directed our attention to the items including

algebra. For example she said:

I think that from page 17 and onwards, the tasks become much more difficult, and Iimmediately think [they fit for teachers in] lower secondary school. First there arefractions and calculations and stuff, but from page 17 and onwards there are lots of

theories that belong in lower secondary school. So perhaps questions become

harder because of that. () There are many who teach mathematics in elementaryschool, who might not have so much knowledge concerning the mathematics that

comes after [elementary school].

The two other students agreed, and one of them recommended us to intersperse the algebra items

throughout the form instead of having all towards the end as in the original:

... so that you dont get the cramps in the end. That you feel it becomes harder.Especially if this test is supposed to be for teachers in elementary school.

The experienced teachers also found the items too difficult, especially for teachers that took their

initial teacher education more than 20 years ago. One of the teachers said:

It might be that recently educated [teachers] have been through a teacher education

that enables them to do this, but those who have taught for ten years and more, I am

sure they are not going to feel at home with this or feel competent.

The other teacher agreed with this. Even though the interviews revealed that some of the items

would be difficult for Norwegian teachers, they also indicated that several items were quite easy.

RM: Item 11?

23


24/36


FS2: I think that one was easy enough.

RM: Is it something in particular that made it easy?

FS2: I dont know.

FS1: Pizza is kind of...

FS2: () I think like this: Okay, so I imagine a task, and I ask myself: What is the

answer to this task? And to help me find out if the problem is right or not, then Ithink about what the answer to the problem might be. So there is a little answer

book for my own part, and in addition to just trying: could this work, does it seem

logical, then I think like this.

Although the teachers and the masters students suggested that some of the items would be

difficult for Norwegian teachers, we decided to avoid changes that might influence the difficulty

of the items. We also decided to include the complete set of items in the same order that they

were in the original measures, although the respondents suggested that we might spread the

algebra items to make it appear less difficult towards the end.

Another perspective concerns the relevance of the contexts included in the items. The

masters students found some of the contexts irrelevant for the Norwegian context. Baking

cookies represents a context that is familiar in an American setting, whereas this was not

conceived as a familiar activity in an Irish context. In their article, Delaney and colleagues

(2008) changed this activity into one of baking scones. Neither of these are familiar activities in

a Norwegian setting. The challenge is to find a good alternative for the translation, and at the

same time avoid changing the problem in a way that influences the mathematical challenges

involved. We decided to use the activity of baking chocolate cookies/biscuits in the pre-pilot

study to see how this worked out, although this is an activity that few Norwegians find familiar.

24


25/36


The masters students reacted to this.

FS1: I wonder how many kids bake chocolate cookies.

[...]

FS2: If it had been 3 cups of flour and they were making pancakes, would that beokay?

FS1: Yes, it might be better. Chocolate cookies are so American. There arent many

() who have baked chocolate cookies. Christmas cookies maybe, or sweet bunsor waffles, or ... But chocolate cookies, thats something you buy rather than bake.

After the pre-pilot study we decided to change this context from the chocolate cookies to the

more familiar Norwegian sweet buns with raisins. This is a context that more Norwegians would

find familiar, and we did not have to change any of the numbers or mathematical considerations

that were involved in the item.

The masters students did not always agree on the relevance of the contexts, and the item

that included a context where a paper frog was moved along the number line was particularly

troublesome for some. Since the pre-pilot study did not show agreement among the participants

at this point, we decided to keep the item unchanged.

The item that included a reference to the Tetris video game also confused the master

students:

FS1: () I was annoyed by that one. How did you get that one?

FS2: Because if you rotate, then it is that one.

FS1: Are those the ones youre supposed to...

FS2: Yes, its these figures up here youre supposed to rotate.

FS1: Oh, I rotated these [figures]. Then all of them matched!

25


26/36


FS2: If you rotate that one, anyway, then it will not be similar to that one.

FS1: () I played an awful lot of Tetris when I was little, so I only imagined which

would fall down on that one to make it disappear.

When the item included a reference to Tetris, this masters student made use of her knowledge of

this video game and interpreted the item in a way that was not originally intended. The intention

was that the figures that were rotated were similar to the figures that appear in Tetris, but the

Tetris game strategy was not meant to be applied in the item. This distinction was apparently

unclear to this masters student.

The amount of text included in many of the items was also a problematic issue that

appeared in the interviews in the pre-pilot study. One of the master students described herself as

a slow reader, and she said that the amount of text in some of the items made them difficult for

her.

FS2: In item 3 I had to read the text three or four times before I ()

FS1: There was a lot of text.

FS2: Yes. But I guess that was part of the idea as well, that it should uncover

misconceptions, I dont know

RM: Was it something in particular that resulted in you having to read it severaltimes, or was it simply the amount of information?

FS2: I guess it was just that there was a lot of information in the text compared withwhat you are used to when working with these kinds of tests.

To keep our translated MKT measures as close to the original as possible, we decided to keep the

amount of text unchanged. The amount of text is closely related to the time it takes to complete

the items. The participants did not have a time limit for the test, but we indicated that they would

26


27/36


normally use 60-90 minutes on the entire form. Although there was no time limit, one of the

masters students said that she had felt a time pressure simply because the others had finished

before her, and we were sitting there, waiting for her to finish.

FS1: I had little time. Cause there was too much to read. () I would have

preferred to have another half an hour. () And I can tell you this: I feel it in my

body, even when this wasnt an exam or anything. I don't like to be the last one tofinish these tests. I think it sucks to be the last one, when you two are sitting there

waiting for me to finish. () It stresses me out. I would only wish that I could sit in

peace and quiet, you know.

A last but not least important issue is the concern about how we as researchers are going to use

the results. One of the masters students refer to a test she and her fellow students were given

when they started their own teacher education. This test had no relation to the MKT measures.

The student told us that the test was given without any reason or explanation, and some weeks

later the newspaper headlines proclaimed that the teacher students did not know their

mathematics.

FS1: They [teacher students] didnt know the 10

th

grade curriculum and stuff.

JF: What do you think about that?

FS1: () I do think its sad to be presented like that, because there is a limitation towhat these kinds of tests can say about your knowledge.

One of the masters students also recommended us to change one aspect about the information

that was given as an introduction to the test.

FS1: I would like to say another thing. In the introduction, you shouldnt say thatthose who perform well on this test will have good results in their class. That would

seem like... The teacher who sits there and might not get any answers right, and

who is thinking: What about me then?

FS2: Yes, like we said before, yes.

27


28/36


FS1: At least not if you want them to play along nicely.

We are aware of the fact that the American tradition in which the MKT measures are developed

may be different from the Norwegian tradition. We may experience that teachers are unwilling to

participate in our study as the following transcript from our interview with the experienced

teachers indicates.

FT: [My] first impression is that this was generally quite hard. Lots of [teachers] in

Norwegian schools would be shocked by this, and they would refuse to participate

in a course if they were faced with this beforehand.

Norwegian teachers do not enjoy the feeling of being tested, and this might at least be partly

related to the fact that Norwegian newspapers have been writing lots of negative articles about

the quality of Norwegian teachers. Every time an international test reveals that Norwegian

students performance in mathematics is worse than we would like it to be, newspapers write

about how bad our schools are. There is also an ongoing debate in Norway about the possibility

of paying successful teachers more, and this is another difficult issue that is partly related to the

idea of testing or evaluating the teachers.

The issues presented in this section are some of the aspects we intend to investigate

further in our pilot study.

Conclusion

The questions this paper intended to answer was on the one hand regarding the nature of

the challenges involved in the process of translating and adapting the challenges for use in

Norway, and on the other hand we wanted to find out more about which of these challenges were

general and which were special to the Norwegian case. The latter question is arguably more

28


29/36


difficult to answer than the former. Another issue that has arisen is even more difficult to

approach:

What criteria will guide a decision as to whether or not our translation and adaption has

been successful?

This question will be important to pay careful attention to in relation to our pilot study, and we

are going to present a beginning of this discussion towards the very end of this paper. First, we

are going to pay attention to the initial questions.

Translating and adapting the MKT measures for use in Norway is both complex and

difficult, as the issues we have brought up in this paper indicate. It is not simply a matter of

making a literal translation, it is not even about making a high quality translation, and several

important aspects can be lost in translation. Even changes that appear to be trivial have the

potential of making the items more complicated, easier to misunderstand, or even easier to

understand than was the original intention.

Delaney and colleagues (2008) reported that they changed some of the names in order to

make them more familiar in an Irish setting. Still, they continued to refer to the teachers as Mr.

or Mrs. In Norway, it is more common to address the teacher with his or her first name, and we

had to change the items according to this. Because of this, it was no longer evident which names

belonged to teachers and which belonged to students. In several instances, we therefore had to

rewrite the item somewhat in order to make this clear. Although this might be considered as a

minor issue, it could potentially make the items more complex. This issue is special to the

Norwegian context, but it might also be relevant in some other countries.

29


30/36


An issue that is more special to the Norwegian context is that regarding the use of the

word class when referring to groups of pupils. According to the Norwegian Education Act,

schools are no longer obliged to organize their pupils in traditional classes, and most official

documents refer to groups of pupils, or they use both groups and classes to indicate that

there is no longer a demand that students should be organized in classes. In several schools, they

are very conscious about not using the word class, and in order to avoid this issue we decided

to rewrite the items that referred to this word. This issue is likely to be more related to the

Norwegian context in particular than many of the other issues that we encountered.

In the pre-pilot study, both the experienced teachers and the more inexperienced masters

students agreed that several of the items would be difficult for Norwegian teachers. Some of

them also stressed the fact that several items included a lot of text. These issues might be rooted

in differences in curriculum content or other issues related to school culture. We can only assume

that American teachers did not feel the same way about this, but to our knowledge American

teachers have not been consulted about the items in the same way that we are doing, and we

therefore do not know if these issues are special to the Norwegian context or not.

In our pilot study, we intend to evaluate the success of our translations and adaptations by

using focus group interviews in addition to analyzing the results from the survey. We plan on

asking a selection of the teachers in detail about whether the items appear authentic to them or

not, and whether the mathematical content of the items is of a kind that Norwegian teachers

encounter in their regular teaching practice. The teachers will also be asked to comment on each

particular item in the measures. Data from these interviews might help explain errors or

30


31/36


responses that we would not expect based on previous analysis of the results from the American

teachers. These data might also help us formulate alternative questions or response options in a

future adaptation of the items, if such an adaptation proves to be necessary. In the Irish study, an

analysis of the interview data was fruitful in order to identify which items may cause difficulties

for the teachers, and whether the situations and characters described appeared authentic to

teachers.

After we have carried out and analyzed the results from the pilot study, we are going to

find out if the issues from our translation and from the pre-pilot study that have been pointed out

above are actual problems or not. If some of the items appear to be significantly easier or harder

for Norwegian teachers than for American teachers, we have to go into an analysis of the

interviews to try and figure out why this is. There is a possibility that the MKT construct in the

Norwegian setting differs so much from the American that we have to make more serious

adjustments to the measures, and there is also a possibility that we may have to develop our own

measures from scratch. The really difficult question is still this: When and how can we decide

whether or not the translations and adaptations that we have made are successful? If some items

appear more difficult or easier to Norwegian teachers than to American teachers, we have to

analyze why that is the case. An analysis of the interviews or an analysis of curriculum papers,

textbooks or classroom practice might be necessary in order to do this. Norway and the U.S.

have different types of curriculum frameworks and standards, and such an analysis would

probably be rather complicated to make. Even if the results of the Norwegian teachers are more

or less similar to the American teachers, it does not necessarily imply that the MKT construct is

31


32/36


the same. Since the construct of MKT is so closely related to teaching as a practice, the use of

these measures alone is probably not enough. Our suggestion is that comparative studies are

necessary in order to reach a conclusion about this, and such comparative studies should

probably include approaching the translation, adaptation and testing of the measures in different

countries along with interviews and analysis of classroom practice (e.g. video studies of some

kind).

Although this risk of failure is actual and present, we believe that it is important to try.

By going into such a study with a critical view, we might learn something important about the

constraints and possibilities that are entangled in the process of translating, adapting and using

measures and assessments across language and cultural barriers. The potential rewards from such

an endeavor appear to outweigh the risks that are involved, and we find it important for us as

researchers to shed light on these issues in order to prevent uncritical use (and abuse) of such

measures. On the other hand, it would have been useful for the research community if some

common or more universal construct of MKT could be found. If this is impossible, it would still

be worthwhile to investigate the possible differences between the different cultures.

References

Alseth, B., Brekke, G., & Breiteig, T. (2003).Endring og utvikling ved R97 som bakgrunn for

videre planlegging og justering: matematikkfaget som kasus. Notodden:

Telemarksforsking-Notodden.

Adams, R. (2005).PISA 2003 Technical Report. Organization for Economic Co-operation and

Development.

32


33/36


Bachmann, K., & Haug, P. (2006).Forskning om tilpasset opplring. Forskningsrapport nr. 62.

Volda: Hgskulen i Volda.

Ball, D. L., Hill, H. C., & Bass, H. (2005). Knowing Mathematics for Teaching. Who Knows

Mathematics Well Enough To Teach third Grade, and How Can We Decide?American

Educator(Fall 2005), 14-17+20-22+43-46.

Cooney, T. J. (1999). Conceptualizing teachers ways of knowing.Educational Studies in

Mathematics (38), 163-187.

Delaney, S. (2008).Adapting and using U.S. measures to study Irish teachers mathematical

knowledge for teaching. Unpublished PhD-Thesis.

Delaney, S., Ball, D., Hill, H., Schilling, S., & Zopf, D. (2008). Mathematical knowledge for

teaching: adapting U.S. measures for use in Ireland.Journal of Mathematics Teacher

Education, 11(3), 171-197.

Education Act [Opplringsloven] (1998).Lov om grunnskolen og den vidaregande opplringa

(opplringslova) av 17. juli 1998 nr. 61. Retrieved March 4, 2009, from

http://www.lovdata.no/all/hl-19980717-061.html

Geisinger, K. F. (1994). Cross-Cultural Normative Assessment: Translation and Adaptation

Issues Influencing the Normative Interpretation of Assessment Instruments.

Psychological Assessment, 6, 304-304.

Grnmo, L. S., Bergem, O. K., Kjrnsli, M., Lie, S., & Turmo, A. (2004).Hva i all verden har

skjedd i realfagene: norske elevers prestasjoner i matematikk og naturfag i TIMSS 2003 .

Oslo: Institutt for lrerutdanning og skoleutvikling, Universitetet i Oslo.

33
http://www.lovdata.no/all/hl-19980717-061.htmlhttp://www.lovdata.no/all/hl-19980717-061.html


34/36


35/36


Johnson, T. P. (1998). Approaches to equivalence in cross-cultural and cross-national survey

research. In J.A. Harkness (Ed.), Cross cultural survey equivalence (pp. 1-40). Mannheim,

Germany: Zentrum fr Umfragen, Methoden und Analysen (ZUMA).

KD (2006).Et felles lft for realfagene. Strategi for styrking av realfagene 2006-2009. Oslo:

Kunnskapsdepartementet.

KD (2008a).Kvalitet i skolen. Stortingsmelding nr 31 (2007-2008). Oslo:

Kunnskapsdepartementet.

KD (2008b).Kompetanse for kvalitet. Strategi for videreutdanning av lrere. Midlertidig

versjon. [Electronic Version]. Oslo: Kunnskapsdepartementet. Retrieved December 19,

2008, from http://www.regjeringen.no/upload/KD/Vedlegg/Grunnskole/Kompetanse

%20for%20kvalitet.pdf.

MER (2007).Education - from Kindergarten to Adult Education. Oslo: Ministry of Education

and Research. Retrieved February 10, 2009, from: http://www.regjeringen.no/en/dep/kd/

Documents/Brochures-and-handbooks/2007/Education-in-Norway-2007.html?id=475839

Mosvold, R., Fauskanger, J., Jakobsen, A. & Melhus, K. (2009). Translating test items into

Norwegian - without getting lost in translation? Manuscript submitted for publication.

Pedlex Norsk Skoleinformasjon [Pedlex] (2008).Kunnskapslftet. Fag og lreplaner i

grunnskolen. Oslo: Pedlex Norsk Skoleinformasjon.

Pea, E. D. (2007). Lost in translation: Methodological considerations in cross-cultural research.

Child Development, 78(4), 1255-1264.

Shulman, L. S. (1986). Those who understand: Knowledge growth in teaching.Educational

35
http://www.regjeringen.no/upload/KD/Vedlegg/Grunnskole/Kompetanse%20for%20kvalitet.pdfhttp://www.regjeringen.no/upload/KD/Vedlegg/Grunnskole/Kompetanse%20for%20kvalitet.pdfhttp://www.regjeringen.no/en/dep/kd/Documents/Brochures-and-handbooks/2007/Education-in-Norway-2007.html?id=475839http://www.regjeringen.no/en/dep/kd/Documents/Brochures-and-handbooks/2007/Education-in-Norway-2007.html?id=475839http://www.regjeringen.no/upload/KD/Vedlegg/Grunnskole/Kompetanse%20for%20kvalitet.pdfhttp://www.regjeringen.no/upload/KD/Vedlegg/Grunnskole/Kompetanse%20for%20kvalitet.pdfhttp://www.regjeringen.no/en/dep/kd/Documents/Brochures-and-handbooks/2007/Education-in-Norway-2007.html?id=475839http://www.regjeringen.no/en/dep/kd/Documents/Brochures-and-handbooks/2007/Education-in-Norway-2007.html?id=475839


36/36

Mosvold-Fauskanger, AERA 2009 paper

Documents

Transcript of Mosvold-Fauskanger, AERA 2009 paper