Behavioral Pattern Analysis and Quantitative Content Analysis for Online Learning Process

Behavioral Pattern Analysis and Quantitative

Content Analysis for Online Learning Process

國立台灣科技大學工程技術研究所科學教育與數位學習組侯惠澤助理教授

為何探究歷程與行為為何探究歷程與行為 ??

以以 1895-19111895-1911 年清政府的政策為例年清政府的政策為例團體決策中哪裡出錯了團體決策中哪裡出錯了 ??

單看會議記錄是否可以了解單看會議記錄是否可以了解 ??

分析師的工作分析師的工作 -BI-BI 的運用的運用數位學習的運用數位學習的運用 ??

用以診斷學習困難、動機干擾與行為模式用以診斷學習困難、動機干擾與行為模式用以作為自動偵測學習行為並給予適性回用以作為自動偵測學習行為並給予適性回饋的運算饋的運算

現象、文本、詮釋、真相現象、文本、詮釋、真相客觀詮釋學派文本功用客觀詮釋學派文本功用 (Garz, 1994)(Garz, 1994)

導出發現或結果的基本資料導出發現或結果的基本資料各種詮釋分析的基礎各種詮釋分析的基礎發表與溝通結果的核心媒介發表與溝通結果的核心媒介社會科學社會科學 =>=> 文本的科學文本的科學 (Gross, 1981)(Gross, 1981)

再現危機再現危機 (crisis of re-presentation):(crisis of re-presentation): 社會科學可直接捕捉到經驗。當前社會科學可直接捕捉到經驗。當前一般見解偏向於主張生活經驗乃是由研究者書寫的社會文本所創造出之一般見解偏向於主張生活經驗乃是由研究者書寫的社會文本所創造出之產物產物質疑文本與經驗的關連質疑文本與經驗的關連 (Denzin & Lincoln, 2000)(Denzin & Lincoln, 2000)

社會建構論觀點社會建構論觀點 :: 只有經過選擇與詮釋後，事實才具有關連性，拒絕外只有經過選擇與詮釋後，事實才具有關連性，拒絕外在真實假設，參與者藉由附加在特定事件或事物之意義從而主動創造出在真實假設，參與者藉由附加在特定事件或事物之意義從而主動創造出真實，若社會研究要處理社會真時就不可避免必須面對參與者主動附加真實，若社會研究要處理社會真時就不可避免必須面對參與者主動附加意義的運作意義的運作

Mixed MethodMixed Method 哲學基礎哲學基礎實用主義實用主義本體論本體論 :: 實體無用論（實體無用論（ reality of no use)reality of no use) (Tashakkori & Teddlie, 1998)(Tashakkori & Teddlie, 1998) 。。爭辯有無實在，是沒有太大意義的爭辯有無實在，是沒有太大意義的（（ Cherryholmes, 1992; Tashakkori & Cherryholmes, 1992; Tashakkori & Teddlie, 1998Teddlie, 1998 ）。）。知識論知識論 :: 主觀主觀 // 客觀並存論客觀並存論方法論方法論 :: 允許多元研究法並存並用。允許多元研究法並存並用。

QCA ProcedureQCA Procedure

艱難的內容分析範例艱難的內容分析範例 ~~

Building coding systemBuilding coding system

Data collectionData collection Observation Observation

(( 需考慮評分者一致性需考慮評分者一致性 )(Kappa))(Kappa)

Data analysisData analysis

QCAQCA 相關議題相關議題 QCAQCA 為一個有前景的研究方式，但是依然在資料可靠性為一個有前景的研究方式，但是依然在資料可靠性上受到質疑，且最終討論到整個研究的效度上受到質疑，且最終討論到整個研究的效度 (validity)(validity) 問題問題，也就是說，該類研究如何說服讀者運用，也就是說，該類研究如何說服讀者運用 QCAQCA 程序所進程序所進行的推論可靠，且提供實徵的證據與理論的基礎。行的推論可靠，且提供實徵的證據與理論的基礎。舉例而言舉例而言 :: 如果以如果以 QCAQCA來陳述互動的情形來陳述互動的情形 (( 如學生的討論如學生的討論次數次數 )) ，是較可信的，但是，若要據此推論，是較可信的，但是，若要據此推論 (( 如如 :: 用前述用前述的資料推論學生批判思考的程度的資料推論學生批判思考的程度 )) ，就較難很明確。，就較難很明確。

整理自 Rourke L., & Anderson, T. (2004). Validity in Quantitative Content Analysis, Educational Technology, Research and Development, 52(1), 5-18.

QCAQCA 相關議題相關議題 Identifying Behaviours That Represent theIdentifying Behaviours That Represent the ConstructConstruct 當確定了目的之後，接著就是要確認該所要觀察行為可以當確定了目的之後，接著就是要確認該所要觀察行為可以呈現出所要探討的構想，也就是要檢驗確定是否有囊括了呈現出所要探討的構想，也就是要檢驗確定是否有囊括了不恰當的行為或遺漏了應觀察的行為。應由多面向去觀察不恰當的行為或遺漏了應觀察的行為。應由多面向去觀察，不應僅由單面向去決定所要觀察的行為。，不應僅由單面向去決定所要觀察的行為。 (( 例如例如 :: 要了要了解行動參與與其他變項的關係解行動參與與其他變項的關係 ..)..)

經常用於增加經常用於增加 QCAQCA 構想代表性的方式有文獻探討與質化構想代表性的方式有文獻探討與質化的內容分析等，或的內容分析等，或 critical incident technique(critical incident technique( 關鍵事件找關鍵事件找尋技巧尋技巧 ) (Flanagan, 1954) ) (Flanagan, 1954) 或口語分析或口語分析 (protocol (protocol analysis) ( Ericsson & Simon, 1993)analysis) ( Ericsson & Simon, 1993) 等等

整理自 Rourke L., & Anderson, T. (2004). Validity in Quantitative Content Analysis, Educational Technology, Research and Development, 52(1), 5-18.

QCAQCA 相關議題相關議題

進階效度的檢核進階效度的檢核問卷問卷訪談訪談質化文本分析質化文本分析效標關聯分析效標關聯分析多元編碼交互解釋多元編碼交互解釋運用已運用過的編碼表運用已運用過的編碼表

結構論的基本假設結構論的基本假設 Sleeper effect (Hovland et al.,1949) Sleeper effect (Hovland et al.,1949) Discounting cue Discounting cue 假設假設 :: 說服者的負面特性成說服者的負面特性成為訊息打折扣的線索，人們可能將內容與傳遞為訊息打折扣的線索，人們可能將內容與傳遞者分別儲存而遺忘此傳遞者者分別儲存而遺忘此傳遞者

態度測量要與其所預測之行為在行動、目標、態度測量要與其所預測之行為在行動、目標、脈絡、時間四項因素的聚題程度上處於同一層脈絡、時間四項因素的聚題程度上處於同一層次才具有預測性次才具有預測性 (Ajzen & Fishbein, 1977)(Ajzen & Fishbein, 1977)

每個個體每個個體 (( 無論是否為研究者無論是否為研究者 ))皆有其各自的皆有其各自的本體論觀點、知識論觀點與方法論觀點，因此本體論觀點、知識論觀點與方法論觀點，因此研究者的互動與詮釋應更多元研究者的互動與詮釋應更多元 (( 假設假設 ))

理論取徑理論取徑 ::區隔經驗與行動表層與其深層結構，區隔經驗與行動表層與其深層結構，探索其隱性結構探索其隱性結構 ((潛意識潛意識 ))

結構論的基本假設結構論的基本假設客觀詮釋之說明客觀詮釋之說明於可重建的法則基礎上，互動文本形成客觀意義結構，這於可重建的法則基礎上，互動文本形成客觀意義結構，這些意義結構再現或表達互動本身的隱性結構些意義結構再現或表達互動本身的隱性結構 (latent (latent structures)structures) ，此等客觀意義結構獨立存在，不受具體參與，此等客觀意義結構獨立存在，不受具體參與互動之個人對於該互動之意向之影響互動之個人對於該互動之意向之影響 (Oevermann et al., (Oevermann et al., 1979)1979)

如如 :ant, PSO:ant, PSO 演算法、演算法、 structural protein structural protein 語言分析、嚴格之序列分析介入，以一定程度語言分析、嚴格之序列分析介入，以一定程度表現其客觀結構表現其客觀結構

小敘事、小理論之蒐集並加以量化客觀詮釋小敘事、小理論之蒐集並加以量化客觀詮釋多面向的內容分析多面向的內容分析與與視覺化行為模式視覺化行為模式呈現呈現序列行為分析序列行為分析 ::補充質量研究之侷限補充質量研究之侷限 (( 如如 ::量量化成效分析之疑點與質化詮釋之主觀性與代表化成效分析之疑點與質化詮釋之主觀性與代表性限制性限制 ))

結構論的基本假設結構論的基本假設 ((總結總結 )) QCA, LSA, SNA, OPA, Cluster…QCA, LSA, SNA, OPA, Cluster… Decoding and ExploringDecoding and Exploring Protein view->Brain view->Cognitive view-Protein view->Brain view->Cognitive view-

>Behavioral view->Educational view>Behavioral view->Educational view E.g., GroEL or HSP90 structure and functionE.g., GroEL or HSP90 structure and function Folding correctlyFolding correctly 基因解碼、分子變化基因解碼、分子變化 =>=> 實徵觀察與行為分析實徵觀察與行為分析生命現象、醫學生命現象、醫學 =>=> 實徵觀察行為分析實徵觀察行為分析認知發展認知發展 =>=> 實徵觀察與行為分析實徵觀察與行為分析自然生命科學與社會科學的交界與探究方法觀自然生命科學與社會科學的交界與探究方法觀教育、科技運用之行為分析教育、科技運用之行為分析

序列分析是什麼序列分析是什麼 ??

Analysis a behavioral sequence Analysis a behavioral sequence (ex. ABABCBEBABAEDBD…)(ex. ABABCBEBABAEDBD…)

A method of sequential analysis (Bakeman & A method of sequential analysis (Bakeman & Gottman, 1986 ; Gottman, Markman, & Notarius, Gottman, 1986 ; Gottman, Markman, & Notarius, 1977)1977)Lag sequential analysis is the examination of Lag sequential analysis is the examination of whether particular sequences of behavior occur whether particular sequences of behavior occur more or less frequently than would be expected by more or less frequently than would be expected by chance.chance.It could examine the sequence that a particular It could examine the sequence that a particular target event would occur subsequent to a given target event would occur subsequent to a given event having already occurred.event having already occurred.

序列分析圖範例序列分析圖範例

Hou, H. T., Chang, K. E., & Sung, Y. T. (2008). Analysis of Problem-Solving Based Online Asynchronous Discussion Pattern. Educational Technology & Society. 11(1), 17-28.

序列分析步驟序列分析步驟Building coding systemBuilding coding system

Data collectionData collection Observation (Observation ( 需考慮評分者一致性需考慮評分者一致性 )(Kappa))(Kappa) Web logsWeb logs

Data analysisData analysis Frequency of BehaviorsFrequency of Behaviors Conditional Probability CalculationConditional Probability Calculation Expected frequenciesExpected frequencies Z-Score CalculationZ-Score Calculation Draw Transitional DiagramDraw Transitional Diagram

IntroductionIntroductionPattern-Analysis Methods for e-Learning Pattern-Analysis Methods for e-Learning processprocess

Lag Sequentialanalysis

Code Scheme A

Online KS discussionContent records

original protocol

descriptions

Coded Data Set A

QuantitativeContent analysis

Code Scheme B

Coded Data Set B

Behavioral/ContentPatterns

Quantitative analysis approach

Qualitative analysis approach

Triangulation

original protocol analysis

Social Network analysis

STUDY ISTUDY IResult & Discussion: Result & Discussion: Original Protocol AnalysisOriginal Protocol Analysis Table5. Extraction of an actual discussion example (Student no.: S041)

Message No. Author Content Time

#4101 S041 My topic introduces teaching system of customer service from Company A.

4/24 07:50:21

#4102 S005 I found that your analysis is exactly the same with S124~…so…did one of you two upload the wrong data?

5/10 22:26:48

#4103 S041 Well~ if there is repetition, I will charge for copyright! 5/11 12:39:52

#4104 S007 The font in your report is too small and I suggest for larger font, otherwise it’s such an eye sore to read it!~~ …

5/11 14:42:15

#4105 S019 A big applause for you first, the content is very professional and the analysis is quite detailed. I could not pick up any fault, please keep the hard working going! Go go go!! …

5/11 23:15:48

#4106 S012 For this system, is it possible that students cannot effectively grasp the essence due to the rather extensive aspect of function, which causes bad learning efficiency?

5/15 16:51:41

#4107 S041 The system design adopts table and procedures often used in companies; many setting and items of development tools are omitted, so it will be faster during operation.

5/17 08:35:08

STUDY IISTUDY IIResult & Discussion: Result & Discussion: Analysis of the Behavioral Differences between Analysis of the Behavioral Differences between High-achieving and Low-achieving StudentsHigh-achieving and Low-achieving Students

Analysis of the Behavioral Differences between High-achieving and Low-achieving Students (P code)

Behavior Code Group N Mean Mann-Whitney U Z

P1High 15 4.0667

44.5 -2.247*Low 12 2.4167

P2High 15 10.3333

86.5 -.171Low 12 10.7500

P3High 15 2.2000

82.0 -.398Low 12 2.3333

P4High 15 1.3333

88.0 -.117Low 12 1.0000

P5High 15 0.4000

86.0 -.289Low 12 0.1667


Analysis of the Behavioral Differences between High-achieving and Low-achieving Students (C code)


C1High 15 9.6667

77.0 -.638Low 12 8.4167

C2High 15 1.4000

80.5 -.490Low 12 1.1667

C3High 15 3.5333

86.5 -.175Low 12 3.9167

C4High 15 0.2000

74.0 -1.079Low 12 0.3333

C5High 15 0.0000

90.0 0Low 12 0.0000

C6High 15 0.4000

86.0 -.289Low 12 0.1667

STUDY IISTUDY IIResult & Discussion: Result & Discussion: Correlation AnalysisCorrelation Analysis

Analysis of Each Behavioral Code and the Learning Achievements

** p<.01 *p<.05

P1 P2 P3 P4 P5 C1 C2 C3 C4 C6 Score

P1 1 P2 .498** 1

P3 .434** .742** 1

P4 .431** .677** .707** 1

P5 .374* .511** .150 .306* 1

C1 .698** .883** .552** .438** .501** 1

C2 .390** .520** .513** .177 .436** .525** 1 　　　C3 .417** .865** .796** .868** .404** .597** .277 1 　　　C4 .308* .426** .708** .689** -.025 .221 .079 .629** 1 　　C6 .374* .511** .150 .306* 1.00** .501** .436** .404** -.025 1 　

Score .334* .123 .127 .225 .309* .172 .253 .105 -.013 .309* 1

Results and DiscussionResults and Discussion

We organized all We organized all recorded operations recorded operations - a total of 7048 - a total of 7048 codes during the codes during the online PBL activity.online PBL activity.

We analyzed the We analyzed the average average percentages of all percentages of all students’ early, students’ early, middle, and late-middle, and late-stage behaviors as stage behaviors as shown in Figure. 1.shown in Figure. 1.

Figure 1 Time allocation of students’ interactive behaviors during online PBL


In order to in-depth In order to in-depth explore whether these explore whether these percentage-variations percentage-variations of each codes between of each codes between the early-stage to the early-stage to middle-stage and the middle-stage and the middle-stage to late-middle-stage to late-stage reach statistical stage reach statistical significance (e.g.: significance (e.g.: Whether the Whether the percentage of a certain percentage of a certain behavior increased behavior increased from early stage to from early stage to middle-stage middle-stage significantly?)significantly?)We subsequently We subsequently conducted paired conducted paired sample tests to sample tests to compare the compare the variations, the results variations, the results was listed in Table 2.was listed in Table 2.

StagePaired Samples Test

Behavior(N=43)

Early (%)

Middle (%)

Late (%)Early – Middle (t)

Middle- Late(t)

Sharing Information(S)

M 20.14 11.21 3.26

2.33* 5.91**SD

23.13 8.24 4.75

Providing Feedback (F)

M 12.12 8.07 6.51

1.44 1.07SD

18.39 6.87 7.08

Proposing Question(Q)

M 1.51 7.81 3.84

-6.05** 3.67**SD

3.21 6.65 4.41

Answering Question(A)

M 5.30 7.30 4.98

-1.40 1.96SD

8.00 5.79 4.38

Viewing Peers’ works(V)

M 56.21 65.42 81.56

-2.01*

-10.83**SD 29.41 13.42 10.96

* p<=.05, ** p<.01

Table 2 Means and standard deviations of percentages for each type of students’ interactive behaviors in each stage during the online PBL activity

STUDY ISTUDY IResult & Discussion : Content AnalysisResult & Discussion : Content Analysis

Among the 332 Among the 332 message codes. The message codes. The total number and total number and percentage of each percentage of each code are shown in code are shown in Figure. Figure.

STUDY ISTUDY IResult & Discussion : Sequential AnalysisResult & Discussion : Sequential Analysis

Adjusted Residuals Table (Z-scores)

C1 C2 C3 C6

C1 3.98* 0.30 1.22 -6.82

C2 -0.24 1.60 -0.33 -0.59

C3 -0.81 -0.32 -0.06 1.53

C6 -2.56 -1.35 -0.62 5.13*

Behavioral transition diagram for peer assessment online discussions

STUDY ISTUDY IResult & Discussion : Sequential AnalysisResult & Discussion : Sequential Analysis


Analysis of the Behavioral Differences between High-achieving and Low-achieving Students (P code)


P1High 15 4.0667

44.5 -2.247*Low 12 2.4167

P2High 15 10.3333

86.5 -.171Low 12 10.7500

P3High 15 2.2000

82.0 -.398Low 12 2.3333

P4High 15 1.3333

88.0 -.117Low 12 1.0000

P5High 15 0.4000

86.0 -.289Low 12 0.1667


Analysis of the Behavioral Differences between High-achieving and Low-achieving Students (C code)


C1High 15 9.6667

77.0 -.638Low 12 8.4167

C2High 15 1.4000

80.5 -.490Low 12 1.1667

C3High 15 3.5333

86.5 -.175Low 12 3.9167

C4High 15 0.2000

74.0 -1.079Low 12 0.3333

C5High 15 0.0000

90.0 0Low 12 0.0000

C6High 15 0.4000

86.0 -.289Low 12 0.1667

STUDY IISTUDY IIResult & Discussion: Result & Discussion: Correlation AnalysisCorrelation Analysis

Analysis of Each Behavioral Code and the Learning Achievements

** p<.01 *p<.05

P1 P2 P3 P4 P5 C1 C2 C3 C4 C6 Score

P1 1 P2 .498** 1

P3 .434** .742** 1

P4 .431** .677** .707** 1

P5 .374* .511** .150 .306* 1

C1 .698** .883** .552** .438** .501** 1

C2 .390** .520** .513** .177 .436** .525** 1 　　　C3 .417** .865** .796** .868** .404** .597** .277 1 　　　C4 .308* .426** .708** .689** -.025 .221 .079 .629** 1 　　C6 .374* .511** .150 .306* 1.00** .501** .436** .404** -.025 1 　

Score .334* .123 .127 .225 .309* .172 .253 .105 -.013 .309* 1


We organized all We organized all recorded operations recorded operations - a total of 7048 - a total of 7048 codes during the codes during the online PBL activity.online PBL activity.

We analyzed the We analyzed the average average percentages of all percentages of all students’ early, students’ early, middle, and late-middle, and late-stage behaviors as stage behaviors as shown in Figure. 1.shown in Figure. 1.

Figure 1 Time allocation of students’ interactive behaviors during online PBL


In order to in-depth In order to in-depth explore whether these explore whether these percentage-variations percentage-variations of each codes between of each codes between the early-stage to the early-stage to middle-stage and the middle-stage and the middle-stage to late-middle-stage to late-stage reach statistical stage reach statistical significance (e.g.: significance (e.g.: Whether the Whether the percentage of a certain percentage of a certain behavior increased behavior increased from early stage to from early stage to middle-stage middle-stage significantly?)significantly?)We subsequently We subsequently conducted paired conducted paired sample tests to sample tests to compare the compare the variations, the results variations, the results was listed in Table 2.was listed in Table 2.

StagePaired Samples Test

Behavior(N=43)

Early (%)

Middle (%)

Late (%)Early – Middle (t)

Middle- Late(t)

Sharing Information(S)

M 20.14 11.21 3.26

2.33* 5.91**SD

23.13 8.24 4.75

Providing Feedback (F)

M 12.12 8.07 6.51

1.44 1.07SD

18.39 6.87 7.08

Proposing Question(Q)

M 1.51 7.81 3.84

-6.05** 3.67**SD

3.21 6.65 4.41

Answering Question(A)

M 5.30 7.30 4.98

-1.40 1.96SD

8.00 5.79 4.38

Viewing Peers’ works(V)

M 56.21 65.42 81.56

-2.01*

-10.83**SD 29.41 13.42 10.96

* p<=.05, ** p<.01

Table 2 Means and standard deviations of percentages for each type of students’ interactive behaviors in each stage during the online PBL activity

Q&AQ&A謝謝指教謝謝指教 !!!!

http://140.122.76.115/hou/web

[email protected]

Behavioral Pattern Analysis and Quantitative Content Analysis for Online Learning Process

Documents

Transcript of Behavioral Pattern Analysis and Quantitative Content Analysis for Online Learning Process