Making More Sense of School Data 分析學校自評數據

Soh KC(2006) Making sense of school data 1

Making More Sense of School Data

分析學校自評數據Dr. Soh Kay Cheng

蘇啟禎博士Consultant (QA)

EMB, Hong Kong SAR

[email protected]


Why do we need data (statistics)?為何要用數據 ( 統計數字 )﹖

Is a special language

Ensures objectivity Avoids

misunderstanding Facilitates

communication

它是特殊的語言它確保客觀它避免誤解它促進溝通


Data for School Self-evaluation學校自評的數據

Describing statuse.g., KPM4 Teacher qualification & experience

Describing viewse.g., KPM 2 Staff’s views on leadership

Describing performancee.g,, KPM22 Student attendance

Describing trendse.g., KPM9 Students’ reading habits of past 3 years

Techniques for comparisonse.g., Compare KPM4 Teacher qualification with Hong Kong’s Reference Data

描述現狀例如 KPM 4 教師的學歷與經驗

描述見解例如 KPM 2 教師對領導的評價

描述表現例如 KPM 22 學生出席率

描述趨勢例如 KPM 9 過去三年學生閱讀

習比較技術

例如 KPM 4 教師學歷與本港參考數據比較


Describing needs reference data描述需要參考數據

A statistic (e.g., %) standing alone has little or no meaning. One or more points are needed for meaningful interpretation of the given statistic.

Reference points may be implicit, assuming the users have common understanding.

Implicit reference points need be made explicit.

孤立的數據（如% ）沒有多大意義甚至毫無意義。數據要有意義﹐必須另有數據作為參考。

參考數據可能是隱含的﹐假設數據使用者已有共識 .

隱含的數據必須明朗化﹐明確指出參考數據的性質。


Good for a laugh

A professor of statistics meets a colleague on his way to the lecture. The colleague heartily greets him, “Good morning, professor!” and then respectfully asks, “How is your wife?”

And the professor absent-mindedly says, “….?”


Describing status描述現狀


How popular are these ECAs?這些課外活動受歡迎嗎

課外活動學會 / 小組的數目學會 / 小組組數學術 1

體育 9

藝術 5

興趣 9

社會服務 12

課外活動學會 / 小組的數目學會 / 小

組組數人數每組平均人數

學術 1 50 50

體育 9 270 30

藝術 5 50 10

興趣 9 90 10

社會服務 12 240 20 The number of groups may not reflect the popularity of each activity. The nature of the activities may have an impact on their popularity. In addition to the number of units, it is good to report also the group

sizes to reflect their populatiry. 每種活動的組別數目不一定反映活動受歡迎的程度。活動是否受歡迎﹐和活動的性質有關。除報告組數外﹐同時也報告各組人數﹐更能反映活動受歡迎的程度。


Box-and-whisker plot as reference 盒鬚圖作為參照

EMB issued Reports on Key Performance Measures Reference Data 2003/2004 for both primary and secondary schools.

The median is the mid-point of a distribution. 50% of the schools are at or above (below) the median.

Top 25% are at or above the 75th percentile. Likewise, bottom 25% are below the 25th percentile. The middle 50% are between the 25th and 75% percentiles.

教統局印發 2003/2004 年度中小學校表現評量參考數據報告。

中位數將學校分為上下兩個群組﹐各有 50% 。

最高的 25% 在 75 百分位數或以上。最低的 25% 在 25百分位數之下。中間的 50%在兩者之間。


已接受專科訓練的三個核心科目教師的百分比% of Subject-trained Teachers

40

50

60

70

80

90

100

中文科教師Subject-trained Chi Teachers

數學科教師Subject-trained Maths Teachers

百份率Percentage (%)

Use of box-and-whisker plot盒鬚圖的運用

The percentage of subject-trained Chinese and Mathematics teachers of school are both 85%. How would you report on these?

For Chinese, the school is at the median of reference data

For Mathematics, the school is among the top quarter.

本校已接受專科訓練的中文和數學教師的百分率均為 85% 。這情況怎麼報告？

中文教師方面，本校位列於參考數據的中位數。數學教師方面，本校位列於最高的四分之一之內。

85% **


Description is not evaluation 描述不等於評估

KPM3 教師專業發展教師參與持續專業發展的平均時數 82.2

校長參與持續專業發展的時數 207.0

教學人員（包括教師與校長）在持續專業發展方面的平均預算支出 $127.3

教學人員（包括教師與校長）在持續專業發展方面的平均實際支出 $96.7

Is it good that the teachers spent 82.2 hours and HK$127.3 on professional development? What about the principal?

Reference points needed could be Hong Kong norm, school’s past records, or pre-determined targets.

教師花費 82.2小時和港幣 127.3元於專業發展，理想嗎？校長方面呢？其他可用的參考資料，如香港常模、本校往年數據、或預定目標。


Good to have another laugh

A young man boasts about his wife and says, “My wife has a perfect figure of 100, it’s 38-24-38.”

His middle-age friend, not wanting to lose face, says calmly, “My wife has a perfect figure adding up to 100, too. It is ….”


Describing views描述見解


KPM02 教職員對學校領導層的觀感校長的領導能力與態度 4.01

副校長的領導能力與態度 4.07

中層領導人員的領導能力與態度 4.01

Five-point scale was used. This fact is implicit and needs be made explicit. Should it be a 10-point scale, the interpretation would be rather different.

The two means of 4.01 give the impression that the Principal and the Middle Management were equally favorably evaluated. What if the SD for Principal is 0.06 and that for the Middle Management 1.12?

Scale and dispersion尺度與變異

報告採用五度量表。尺度應該說明。假如所用的是十度量表，則各階層領導所得的評估便非常不同。

兩個平均數 4.01 似乎顯示校長與中管理層得到同樣的好評。假如校長的標準差 (SD)是 0.06 ，而中管理層的是1.12 ，應作何解釋？


Poverty is relative

A rich American wanted to show his son how rich they were. He took the little boy to a seaside village in the South. When everything was over, the father asked, “Son, what have you learned from this trip?”

The boy said, “Oh, yes! We keep one dog and they have four. Sitting at our patio, the view ends at the gate 50 yards away, but at their, there is no end to the horizon. We built walls to protect ourselves, they have friends to protect them….Thank you father, for showing me how….”


Describing performance描述表現


Absolute and relative standards絕對標準與相對標準

學生從學校圖書館借用資料頻數 ( 中一至中三）1 每周一次或以上 24.2%

2 每兩周一次 22.1%

3 每月一次 36.5%

4 每月少於一次 15.8%

5 從不 1.4%

絕對標準• 預期目標：每兩週借

用資料至少一次的學生有 50% 或以上。

• 實際上﹐每兩週借用資料至少一次的學生有 46.3%﹐接近預期目標 50% 。。相對標準

• 借用層次有五個。其中最高的是“每月一次”。• 因此﹐學生借用資料頻數的眾數 (36.5%) 為“每月一次”。

Absolute standard: 46.3% borrowed at least once in two weeks.

Relative standard:

36.5% is the mode.


The dangerous average

Mr. Dumb Bell jumped into the sea from a jetty and got a big hump on his forehead, because the sign board says:

First 30 meters, average depth 5 meters!


Describing trends描述趨勢


Evaluation varies with reference data 評價隨參考數據而變

Relatively speaking, there was an improvement in 2004 over 2003 in all three subjects, but, there is a retrogression from 2004 to 2005 in the two languages.

Assumption: Papers of the three years are equivalent.

Need HK norms for the three years for proper interpretation.

相對而言， 2004 的成績比 2003 的好，有進步。但是，從 2004 到2005, 兩語文科有退步的現象。

假設：三年的考卷難度相同。正確的解釋需要三年的香港常模作

為參考資料。

0

10

20

30

40

50

60

2003 2004 2005

中文英文數學

校內學科測驗成績學科 2003 2004 2005

中文 41.2 55.0 52.3

英文 22.8 44.7 35.3

數學 13.1 33.9 39.3


Effect Size 效果强度學校數據參考數據比參考數據

課程策劃與組織 3.65 3.54 +0.11*

課程管理 3.70 3.57 +0.21*

教學策略和技巧 3.65 3.45 +0.20*

* 比參考數據 +/-0.1 或以上

Comparison with Reference Data is a good effort. However, +/-0.1 is arbitrary.

If SDs are available, then effect sizes can be calculated for more meaningful interpretation.

將本校的情況和參考數據比較，是好的做法。但是﹐以 +/-0.1 為臨界值﹐似乎武斷。

如有標準差﹐可轉化為效果強度(Effect size), 更有意義。


Conversion to effect size效果強度的轉化

School

Mean

Reference Data

Mean (SD)

Effect Size

課程策劃與組織 3.65 3.54 (0.25) 0.44

課程管理 3.70 3.57 (0.18) 0.72

教學策略和技巧 3.65 3.45 (1.15) 0.20

With reference to the normal distribution, ES can be used to evaluate differences in percentages.

ES = (Mean – Norm) / SD

如參照常態分佈﹐百分率可轉化為效果強度﹐以便檢定百分率差異的意義。

效果強度 = （平均數 – 常模） / 標準差

註﹕參考數據是虛擬的。


Effect size ‘standards’效果強度的‘標準’

Difference in % Effect size Description

Below 8% Below 0.2 Negligible effect

微效果

Between 8 and 18 % Between 0.20 and 0.49

Small effect

小效果

Between 19 and 28% Between 0.50 and 0.79

Moderate effect

中等效果

Between 29 and 34% Between 0.80 and 0.99

Large effect

大效果

35% or more 1.00 and

above

Very large effect

極大效果


Summary 綱要1. 用數據溝通，力求客觀共識。2. 用數據描述現狀﹑見解﹑表現

﹑與趨勢。3. 用參考數據作有意義的詮釋。4. 用百分比作比較。5. 用參考數據來評估。6. 參考的數據可能是常模﹑記錄

﹑或目標。7. 兩個相同的平均數可能有不同

意義。注意變數的大小。8. 用絕對標準或相對標準描述表

現。9. 用曲線圖表達趨勢﹐並注意隱

含的假設。10. 用效果強度的「標準」進行客

觀的評估。

1. Use data to ensure objectivity and common frame of mind.

2. Use data to describe status, views, performance, and trends.

3. Use reference data for meaningful interpretation.

4. Use %’s for comparison.5. Use reference data for evaluation。6. Data for reference may be the norms,

past records, or targets.7. Two means of the same magnitude may

have different meaning. Watch out for difference in variability.

8. Use absolute or relative standard to describe performance

9. Use curves to indicate trends and watch out for tacit assumptions.

10. Use effect size ‘standards’ for objective evaluation of effects.


Statistics are estimates

I asked a statistician for her telephone number and she gave me an estimate.


Comparison Techniques

比較技術


Purpose

School reports always present summary data such as the mean, the SD, etc.

Statistical calculators on the Internet can be used to make some of the needed comparisons.

This part of the seminar introduces such calculators to enable schools to make finer interpretation.


Comparison with expected performance與預期表現比較

N Mean SD

65 3.71 0.35

The result of a survey on KPM2 Teachers’ view on the principal’s leadership. If the expected mean is 3.50, was the principal more favorably evaluated than is expected?This calls for a one-sample t-test and the critical value of p is set at 0.05.

KPM2 教師對校長領導能力的評估，調查結果如表所示。如果預期平均數為 3.50 ，校長所得平均數是否較預期的為高？這需要用單組 t- 測加以鑒定，並以 0.05為 p 的臨界值。


The result shows that the obtained mean (3.71) is statistically greater than the expected mean (3.50). The principal was evaluated higher than the expected.

統計分析結果顯示，實得的平均數（ 3.71 ）的確高於預期的平均數（ 3.50 ）。校長所得評估的確比預期的高。

http://glass.ed.asu.edu/stats/analysis/ttest.html


Comparison of two groups兩群組比較

N Mean SD

Teachers 65 3.52 0.30

Parents 125 3.75 0.65

The table shows the ratings on KPM11 School culture given by teachers and parents. Did the parents evaluate the school more favorably than did the teachers?

This calls for an independent t-test and the critical p is set at 0.05

上表顯示教師與家長對 KPM11 學校文化的評估。兩者對學校的確有不同評價嗎？

這必須用獨立 t- 測加以鑒定，並以 0.05為 p 的臨界值。


The result shows that the difference (0.23) between the two means is unlikely a chance occurrence. Parents did evaluated the school more favourably than did the teachers.

http://glass.ed.asu.edu/stats/analysis/t2test.html

統計結果顯示，兩平均數之差(0.23) 並非機遇現象。家長對學校的評價的確高於教師的評價。


Comparison of many groups多群組比較

Mean SD

Teachers (N=65) 3.51 0.35

Students (N=120) 3.65 0.45

Parents (N=105) 3.83 0.52

The ratings on KPM 11 School culture as given by teachers, students, and parents are shown above. Did the three groups differ in their rating?

This calls for one-way ANOVA (analysis of variance) followed by pair-wise comparisons.

針對 KPM 11 學校文化，教師、學生、與家長作以上的評估。三組的評價的確有差異嗎？

這需要用單向變異分析（ one-way analysis of variance ）鑒定，並再用配對 t- 測。


The analysis shows that p<0.05; there is at least one significant difference among the three means.

分析結果顯示 p<0.05 ，表示至少有一對的平均數有非機遇的差異。

http://statpages.org/anova1sm.html


As there are three pair-wise tests, Bonferroni adjustment is applied to avoid accumulation of error. For three tests, the p-value should be 0.05/3 or 0.0166. To check on these p-values, see the next slide.

有三對平均數作配對比較，必須作 Bonferroni 調整，以避免誤差的累積，而 p- 值應該是 0.05/3 or 0.0166. 要確定這三個 p- 值，請看下頁。



For all three t-values (21.76, 43.82, and 27.84), the corresponding p is 0.0001. The differences among the three means are statistically significance; the differences are very unlikely chance occurrences.

(65 + 120 - 2)

http://department.obg.cuhk.edu.hk/researchsupport/T_Test.asp

三個 t-數 (21.76, 43.82, and 27.84)配對比較的相應 p- 值是 0.0001. 它們之間的差異並非機遇現象；其間的確有差異。


Comparison with reference data (1)與參考數據比較（一）

Professionally trained Not trained

Hong Kong 96% (62) 4% (3)

School 90% (59) 10% (7)

For KPM 4, the school has 90% of its 65 teachers professionally trained. Is this percentage significantly lower than the Hong Kong Reference Data?

This calls for a chi-square test of goodness of fit. The percentages need be converted into frequencies for analysis.

學校的 65 位教師，有 90%受過專業訓練。這和香港參考資料比較，有差異嗎？

這需要用卡方測驗來鑒定。並需先將百分數轉為頻數。


http://www.georgetown.edu/faculty/ballc/webtools/web_chi.html


Since the chi-square is not significant, the school is not different from the Hong Kong reference data.

The chi-square test results show p>0.05; the school’s distribution fits the Hong Kong distribution.

卡方測驗結果， p>0.05; 顯示學校的分配情況和香港參考數據相配。


Comparison with reference data (2)與參考數據比較（二）

Master Bachelor Non-degree

Hong Kong 23% (15) 66% (43) 11% (7)

School 27% (18) 58% (38) 15% (9)

For KPM 4, the school reported the percentages of teachers with different qualification as shown above. Does the school’s distribution differ significantly from the Reference Data?

This calls for a chi-square test.

學校教師學歷分佈如表所示。與參考數據相同嗎？

用卡方測驗鑒定。


Since the chi-square is not significant (p>0.05), the school’s distribution is not different from the Reference Data.

卡方檢查結果， p>0.05 ; 學校的分佈情況和參考數據的沒有不同。


Summary 綱要One-sample t-test for comparing a

observed mean to an expected mean.

Two-sample t-test for comparing the means of two independent groups.

One-way ANOVA for comparing more than two independent means.

Significance of t-value for checking the probability of an obtained t-value.

Chi-square test for ascertaining association between membership and performance.

單組 t-測：比較實際平均數與預期平均數。

兩組 t-測：比較兩組的平均數 .

單向變異分析：比較多過兩組的差異。

t-值的臨界值：檢查所得 t-值是否機遇現象。

卡方測驗：檢定兩個分佈情況之間的差異。


Hyperlinks to calculators

One-sample t-test

http://glass.ed.asu.edu/stats/analysis/ttest.html

Two-sample t-test


One-way ANOVA

http://statpages.org/anova1sm.html

Significance of t-value

http://department.obg.cuhk.edu.hk/researchsupport/T_Test.asp

Chi-square test

http://www.georgetown.edu/faculty/ballc/webtools/web_chi.html


He who laughs last, laughs best最後一笑Q: As a principal, how do you develop your

teachers professionally?A: As a responsible leader, I make sure that

everyone of them is busy and works hard.Q: What are they working hard on?A: That does not really matter, as long as they are

working non-stop.Q: Could you give me an example?A: For instance, ….


Thank youThank you

謝謝謝謝

Making More Sense of School Data 分析學校自評數據

Documents

Transcript of Making More Sense of School Data 分析學校自評數據