Korea Management Association Measurement, Evaluation & ROI 2006/05/08 서울대학교...

Measurement, Evaluation & ROI

2006/05/08 서울대학교 산업인력개발학 이 찬 교수

마스터 제목 스타일 편집Contents

Overview1

Evaluating Training Programs: 4 Levels2

Aligning Kirkpatrick’s 4 Levels with BSC3

4 Summary

마스터 제목 스타일 편집Overview

Measurement, Evaluation, & ROI

Performance Improvement

Career Planning & Talent Management

마스터 제목 스타일 편집Overview: M.E.R.

SU415 Implementing Training Scorecards: Prove That Training Pays Lynn Schmidt

M102 Evaluating Training Programs: The Four Levels Donald Kirkpatrick

M105 Justify, Manage, Improve a Learning Enterprise With Four Evaluation Events Diederick Stoel

M116 Making Evaluation Work Patti Phillips

마스터 제목 스타일 편집Overview : M.E.R. (cont’d)

M204 Making the Business Case Theresa Seagraves

M216 Demonstrate Training Value: Aligning Kirkpatrick's Four Levels With Balanced Scorecards James Kirkpatrick

M303 Tomorrow's Evaluation Opportunities Diederick Stoel, Jack Phillips, Donald Kirkpatrick, Jac Fitz-enz, Robert Brinkerhoff


TU104 Using Needs Assessment to Drive Business Results, Vanda Vieira

TU115 Training Impact Evaluations That Senior Managers Believe and Use, Robert Brinkerhoff

TU118 See the Organizational Big Picture: An Action-Oriented Evaluation Case Study, Deborah Stone

TU204 Evaluation System Design, Rodrigo Lara Fernandez


TU203 Meaningful Metrics to Measure the Value of Knowledge Management Bruce Aaron

TU214 How to Evaluate "Soft" and "Hard" Data Judith Hale

TU215 Measuring Human Capital Dennis Kravetz

TU303 A Practitioner’s Roadmap for Data-Driven ROI W. Byerly


TU315 Create the Annual Report for Training and Development Rachelle Verkamp

W101 Evaluating Training Programs: The Four Levels Donald Kirkpatrick

W114 Take Charge: Design Training for Results and ROI Ron Stone

W204 Global Integration of the Learning Function: ASTD/Raytheon Professional Services Study Jeff Lucas


W205 Demonstrate Training Value: Aligning Kirkpatrick's Four Levels With Balanced Scorecards James Kirkpatrick

W310 Leveraging Technology for Practical and Scaleable Learning Analytics Jeffrey Berk

마스터 제목 스타일 편집Evaluating Training Programs: 4 Levels

Results: 결과

Behavior: 행동

Learning: 학습

Reaction: 반응

4 단계 : Final Results

3 단계 : Change in B.V.

2 단계 : KSA

1 단계 : CS

마스터 제목 스타일 편집Evaluating Training Programs: cont’d

효과적인 프로그램의 조건

1. Needs

2. Objectives

3. Time

4. Place

5. People

6. Leader

7. Techniques

8. Objectives

9. Satisfied

10.Evaluated

마스터 제목 스타일 편집

프로그램의 지속 여부

프로그램의 향상 여부

HRD Pro. 존재 이유

PerformancePerformanceImprovementImprovement

Evaluating Training Programs: Why?Evaluating Training Programs: Why?

마스터 제목 스타일 편집Evaluating Training Programs: 1 단계

1 단계 평가 지침 : 반응

• 무엇을 파악하고자 하는지 결정하라

• 반응을 수량화 할 형태로 설계하라

• 의견을 적도록 권장하라

• 100% 즉각적인 응답을 얻어라

• 바람직하다면 , 지연된 반응도 얻어라

• 수용할 수 있는 표준을 정하라

• 표준에 상응하는 미래 반응을 측정하라

마스터 제목 스타일 편집Evaluating Training Programs: 1 단계 반응 평가 양식 Evaluating Training Programs: 1 단계 반응 평가 양식

마스터 제목 스타일 편집 cont’d

• SA: Strongly Agree

• A: Agree

• N: Neutral

• DA: DisAgree

• SD: Strongly Disagree

마스터 제목 스타일 편집Evaluating Training Programs:1 단계 반응 평가 양식 Evaluating Training Programs:1 단계 반응 평가 양식

(5x10+4x5+3x3+2x1+1x1) / (10+5+3+1+1)


2 단계 평가 지침 : 학습

• 교육 전후에 지식 , 기술 , 태도 등을 측정하라• 지식과 태도에 관한 지필 고사를 보라• 기술에 관한 실기 고사를 보라• 100% 응답을 얻어내라• 실질적이라면 , 통제집단을 사용하라

마스터 제목 스타일 편집Evaluating Training Programs: Scoring Form


3 단계 평가 지침 : 행동 • 교육 전후에 지식 , 기술 , 태도 등을 측정하라

• 행동변화가 일어나도록 3-6개월의 시간적 여유를 줘라

• 교육생이나 그들의 상사 , 부하직원 , 그들을 행동을 아는 사람들과 설문이나 인터뷰하라

• 100% 응답이나 샘플링하라

• 적당한 시기에 반복하라

• 통제집단을 사용하라

• 비용 대비 이익을 고려하라


1. 변하고자 하는 의지

2. 필요한 지식과 기술

3. 알맞은 직무 풍토

4. 격려와 도움

5. 변화를 위한 보상

• 행동 변화에 필요한 것들행동 변화에 필요한 것들

• Job Climate:Job Climate: PPreventing – reventing – DDiscouraging – iscouraging – NNeutral – eutral – EEncouraging - ncouraging - RRequiringequiring


행동 변화행동 변화 : :

교육 교육 3-63-6 개월 뒤 개월 뒤


4 단계 평가 지침 : 결과

• 교육 전후에 측정하라• 가능한 결과를 위해 6-12 개월의 시간적 여유를 줘라• 적당한 시기에 반복하라• 통제집단을 사용하라• 비용 대비 이익을 고려하라


4 단계 평가 지침 : 결과

마스터 제목 스타일 편집Aligning Kirkpatrick’s 4 Level’s with BSC

마스터 제목 스타일 편집 Aligning Kirkpatrick’s 4 Level’s (cont’d)

마스터 제목 스타일 편집 Aligning Kirkpatrick’s

4 Level’s (cont’d)

마스터 제목 스타일 편집Evaluating Training Programs: 4 Levels

•교육 프로그램 평가를 위한 제안

•1. 모든 프로그램의 반응을 측정•자신의 양식 개발•자료 수집•수용 가능한 성과 기준 설정•프로그램 향상

•2. 평가를 다음으로 확대하라•학습•행동•결과 (ROI 포함 )

•3. 다음을 고려하라•가능한 평가 자원•해당 교육프로그램 빈도•프로그램 비용•평가 비용•임원의 관심과 요구

[email protected]@snu.ac.kr

Performance Improvement

2006/5/10 서울대학교 이 찬 교수

마스터 제목 스타일 편집Contents

Introduction

How Top Companies Invest In H.C. (TU110)

HPI & the Balance Scorecard (TU Performance RM)

Q & A

11

22

33

44

마스터 제목 스타일 편집Click to edit title style

How do I incorporate my logo to a slide that will apply to all the other slides?

On the [View] menu, point to [Master], and then click [Slide Master] or Notes Master. Change images to the one you like, then it will apply to all the other slides.


HOW TOP COMPANIES INVEST IN HUMAN CAPITAL:

Three New Models

일류 기업이 인적 자본에 투자하는 방법 :세가지 새로운 모형

Jac Fitz-enz, Ph.D.Founder & CEO

Human Capital Source


- AGENDA -

It’s All About Human Capital Investment인적자본 투자에 관한 모든 것

Working from a HC Planning Model인적자본 모형으로부터의 작업

Linking Human Capital Investments To Operations, Sales and Service인적자본투자와 경영 , 세일즈 , 서비스로의 연계

Knowing Where to Invest to Solve HC Problems인적 자본 문제를 위해 어디에 투자해야 하는가


U.S. Companies lose: 미국 회사들이 잃는 것

½ their investors 투자자 in 1 year½ their employees 직원 in 4 years½ their customers 고객 in 5 years

BUSINESS IS PEOPLE

5% Gain in Customer Retention CanAdd Up to 25% Growth in Earnings *고객유지 5% 획득으로 수익 성장 25% 추가 가능 *

*Fred Reichheld, The Loyalty Effect


CFO’S CATCHING ON

72% Believe HC Impacts Innovation 혁신

92% Believe HC Impacts Customer Service

82% Believe HC Impacts Profitability 수익성

AHA!


FUNCTIONAL UNITS

Design & Production & Sales & Customer Development Distribution Marketing Service

RESOURCE MANAGERS

Human Finance & Information Security & Resources Accounting Technology Maintenance

CURRENT STATE

설계 & 개발 생산 & 물류 영업 & 마케팅 고객서비스

인사 재무 & 회계 정보기술 경비 & 시설

마스터 제목 스타일 편집 MOTOROLA BOOKINGS & HC - WORLD,3/12 ROC

CORPORATE STRATEGY OFFICETOTAL MOTOROLA WORLD BOOKINGS & HEADCOUNT

(3/12 ROC)

LATEST PLOT: JANUARY 2001 TOMW HEADCOUNT - BLUE, BOOKINGS - GREEN

-40

-20

0

20

40

60

80

1/94 7/94 1/95 7/95 1/96 7/96 1/97 7/97 1/98 7/98 1/99 7/99 1/00 7/00 1/01

BOOKINGS

-40

-20

0

20

40

60

80

HEADCOUNT

P1

P1 P2

P2

P3

P3P4

T1 T1

T2 T2

T3

T4?

HIRING (Blue) NOT COORDINATED WITH BOOKINGS (Green)MISALIGNMENT COST: $1.500,000,000


“If everybody concerned is absolutely clear about the goals and directives and

far purposes of the organization practicallyall other questions then become simple

technical questions of fitting means to ends”

Abraham Maslow Eupsychian Management

21st CENTURY INTELLIGENCE


3 HUMAN CAPITAL MODELS3 인적자본 모형

Mental정신

Talent재능

Structural구조


Human CapitalSkills - Attitudes

Compensation - Culture

Relational Capital Customer - CompetitorsVendors - Government

Structural Capital Facilities - Equipment Processes - Material

1. NONLINEAR UN-ALIGNING 비선형 비정렬

Workforce IntelligenceAnalytics Assessment Benchmarks


Leadership DevelopmentRecruitment Management

CONVERGENCE 집중성

WORKFORCE

PLANNING

RetentionPerformance

인력 기획

채용 관리 리더쉽 개발

성과 유지

마스터 제목 스타일 편집2. TALENT MODEL 재능 모형

C-Level has Balanced Scorecard

TMI

TMI

Finance has Accounting

Production has Quality

S - M - CS have CRM

NowHuman Resources

has TMI


Employee Customer Profit 5% 2% 1%

HR INVESTMENT RESEARCH: HR 투자 연구

Saratoga Institute: HR Metrics & BenchmarksHuselid & Becker: HR Practices Productivity/TurnoverWatson Wyatt: HCM & Consistent Shareholder ValueGallup: Managerial Behavior & Employee Performance

Sears:

마스터 제목 스타일 편집WHAT IS TMI?

StaffingBusinessIntelligenceApplied To

Human Resources’ ManagementofHuman Capital

Leadership Development

Training & Education

Workforce Planning

Performance Management

Retention


2003 Intelligence - TMI

NEXT GENERATION 차세대

Pre-1980: Data - Feelings

1980 – 1985: Information - Metrics

1985-2002: Knowledge - Benchmarks


Staffing

TMI FRAMEWORK

OperationsSales

Service

Retention

Planning

Leadership

Performance

Development

마스터 제목 스타일 편집HR MANAGEMENT&TALENT MANAGEMENT INVESTMENT

HR: Internal Efficiency내적 효율

&TM: External Effectiveness

외적 효과

HR

마스터 제목 스타일 편집Efficiency vs. Effectiveness

Efficiency : Does one approach achieve the outcomes faster, and is the cost less than the value of the HR outcomes?

Effectiveness : Does one approach result in superior work outcomes, and is the cost less than the value of the work outcomes?


Factor

Measure

Cost

Time

Quantity

Quality

Reaction

HR EFFICIENCY MEASURES

Staffing Pay & Ben Training & Retention Admin Develop.


Factor

Service

Hiring

Paying

Supporting

Developing

Retaining

TM EFFECTS ON BUSINESS

Operations Sales Service

마스터 제목 스타일 편집VALUE REPORTING CHAIN

TM Action Immediate Result Business Effects

Hire . . . . . . Fit = Stay & Perform . . < People Cost = > Margins

Pay . . . . . . Incent = Innovate . . . New Product = > Market Share

Support . . Focus = Productive . . > Output = > Rev

Develop . . Skilled = Quality . . . . Retain Cust. = < Marketing

Retain . . . Commit = Stay . . . . . . < T cost = > Income

재능관리 행동 즉각적인 결과 경영 효과


PREVIEW

3 Capitals as Viewed

Structural

FacilitiesTechnologyProcessesPatents

Relational

PatientsFamiliesRegulatorsSuppliersCompetitors

Human

ExecutivesManagersProfessionalsTechnicians

External

AnticipatedMarketChanges &AssociatedCosts

Key Position Requirements By YearAnalysis of Sources and Methods

Executives – Managers – Professionals - Technicians

FUTURECASTFuture Sources and Methods


Recruitment Management Leadership Development

PERFORMANCE MEASUREMENTMeasures of Business Effects

Internal

VisionBrandCultureStrategyPlan

1. 2.3.

4.

5.

6.

3. STRUCTURAL MODEL

마스터 제목 스타일 편집4. KEY POSITION REQUIREMENTS

Analysis of Recruitment Sources and Methods채용 자원과 방법 분석

Sourcing Media

AdvertisingAgenciesReferrals

Events

Workforce Performance

ExecutivesManagers

ProfessionalsTechnicians

마스터 제목 스타일 편집5. FUTURECAST

Future Sources and Methods 미래 원천과 방법


Recruitment Management Source Analysis &Selection Assessment - Fit - Capability - Tenure - Lost Time Vacancy Projections

Leadership Development Requirements Analysis - Job Assignments - Projects - Training - Mentoring - Education Readiness Projections

마스터 제목 스타일 편집TM LINKAGE

COMPANY RESULTS

FINANCIALS RevenueGross ProfitOperating IncomeEarnings per ShareReturn on Assets

MARKET POSITIONMarket ShareShare Growth RateCustomer Retention

OPERATIONSIT SpendR&D SpendMarketing Spend

HC INVESTMENTS

Cost to Manage Contingent Percent HR spend per Employee

LABOR COSTS Labor cost as % of revenue Labor cost per employee Benefits cost as % of revenue Benefits cost per employee

TM INVESTMENTS Staffing Leadership Development Performance Management Workforce Planning Training & Education Retention


TM LINKAGE

HC INVESTMENTS




COMPANY RESULTS

FINANCIALS Revenue Gross Profit Operating Income Earnings per Share Return on Assets

MARKET POSITION Market Share Share Growth Rate Customer Retention

OPERATIONS IT Spend R&D Spend Marketing Spend

마스터 제목 스타일 편집TMI ANALYSIS

RESULTS COMPANY

FINANCIALS RevenueGross ProfitSG&A ExpenseOperating IncomeNet Income Earnings per ShareReturn on AssetsPreTax Profit Margin

MARKET POSITIONMarket ShareGrowth RateCustomer RetentionCustomer Net Gain

OPERATIONSIT Spend as % of RevenueR&D Spend as % of RevenueMarketing Spend as % of Rev

HC INVESTMENTS




마스터 제목 스타일 편집EARLY RESULTS

PLANNING & LEADERSHIP DEVELOPMENT Big Pharma Found How to Link P & D for Accelerated Leader Development, Improved Manager Retention

STAFFING Assessment Pays Off in Retail and Telephone Sales (+25% time to standard and + 6%) in hourly sales) and Retention (+40%)

PERFORMANCE MEASUREMENT Logistics Service Uses Restructuring, Pay Redesign and Employee Survey to Support 20% Gain in Profits

마스터 제목 스타일 편집EARLY RESULTS

Training & Development Medical Center Linked Training to Competency Models for All Staff andMeasured Results Against Financials and Patient Care

Retention International Supply Chain Manager Studied Turnover in Warehouses and Found PeopleJoin, Stay and Leave for Different Reasons: Dropped Turnover 50% Reducing Operating Expense

Productivity Distribution Center Pickers Turnover 3x Norm - Cost $2.25M. Applied Training & Comp Saving $1.85M in Productivity Loss due to Turnover

마스터 제목 스타일 편집HPI & the Balance Scorecard (TU Performance RM) 1/4

마스터 제목 스타일 편집HPI & the Balance Scorecard 2/4

Line of Sight: problem, gaps, root causes, solutions, and evaluation

Problem:

(who) sales associates in the (where) central region are (what) $360,000 below Product Y sales goals in the (when) 1st quarter potentially impacting ABC Company’s (worth) net profitability.


Gaps Individual Team__________ Quantitative

Expected 3.0 sales/person/day$3K 300 sales assoc=$900KActual 1.7 1.8K =$360K

Gap 1.3 1.2K =$540K

QualitativeExpected mystery shop:9.0/10.0 Customer Sat 4.7/5.0Actual 6.5 3.2

Gap 2.5 1.5


Root Causes:1. Product push for higher net value of sale rather than

based on customer needs2. Workload and staffing: number of customer accounts

assigned to fewer sales associates3. Lack of skills and knowledge on needs based sales and

service

Recommend Solutions:1. Implement sales and service process aligned with

selection and performance management2. Workload, workflow, structure, and staffing models

analysis and revision3. Targeted needs based sales and service skills training

and job aids followed by OJT coaching


Q & A

Dr. Chan Lee

[email protected]

Tel. (02) 880 - 4838

mailto:[email protected]





Career Planning & Talent ManagementCareer Planning & Talent Management

2006/5/11 서울대학교 이 찬 교수

마스터 제목 스타일 편집Career Planning & Talent Management

마스터 제목 스타일 편집순서

2. Leader’s Role in Growing Other Leaders2. Leader’s Role in Growing Other Leaders

W200 (LEGEND) by Dr. Beverly Kaye

3. Q & A3. Q & A

1. Introduction1. Introduction

마스터 제목 스타일 편집BZ Case for CD

Attract a world-class team Retain the stars Match workforce skills to b

z strategy Communicate your brand Provide employee “voice” p

ower Acknowledge managerial le

gacy

세계 정상급 팀을 유인

스타급 유지인력 기술 & 사업 전략

매치브랜드와 호응직원에 여론 권한 부여

경영 전통에 감사

마스터 제목 스타일 편집Bldg a Dev. Culture

The manager provides the Support

The Employee provides the Spark

The Organization provides the System

관리자는 지원을 제공

직원은 스파크를 제공

조직은 시스템을 제공

마스터 제목 스타일 편집Must-Do’s for the Dev.-Minded Leader

Listen: pay attention to abilities and talents

Level: Provide feedback “Plus”

Look Ahead: Talk realistically about the future

Leverage: Discuss optionsLink: Suggest learning

opportunities

경청하기

피드백하기

앞서 보기

추진하기

연결하기

마스터 제목 스타일 편집Listen

Skills / Competence

Interests / Enjoyment

Values / Commitment

기술 / 역량

관심 / 재미

가치 / 몰입

마스터 제목 스타일 편집Level

Observation

+Judgment

+Label

= Reputation

관찰+

판단+

낙인

= 평판

마스터 제목 스타일 편집Look Ahead

Your Job

+Your Org.

Vs.

Your Profession+

Your Industry

직무+

회사

Vs.

직업+

산업

Global Economy

마스터 제목 스타일 편집Leverage

Right Person

Right Place

Right Time

적재 ( 適材 )

적소 ( 適所 )

적시 ( 適時 )

마스터 제목 스타일 편집Link

An Observation

An Assignment

A Course

A Mentor/Coach

A Peer Group

관찰

임무

과정

코치

동료

마스터 제목 스타일 편집Are Managers Prepared to provide Support?

To listen hard so they understand employee capabilities?

To level about ways employee can increase their effectiveness?

To jointly look ahead to see how the direction of the company and their converge?

To help them leverage the choices available to them?

To link them to the appropriate resources?

마스터 제목 스타일 편집Are Employees Prepared to provide the Spark?

To Say … these are my talents

To Ask… are we in agreement about my capabilities?

To Propose…here’s where I fit in this organization’s future

To Offer…these career options excite me

To Recommend…here’s what I want to learn

마스터 제목 스타일 편집Is your Organization Prepared to Provide the System?

Competency profiling?

Performance Management?

Strategic staffing?

Succession planning?

Training and development?

마스터 제목 스타일 편집The Retention Punchline:Development Me or I’m History

Will you utilize the full rage of my talents?

Will you provide on-going performance feedback?

Will you clue me into the future direction of this organization?

Will you explore multiple avenues for my career growth?

Will you assure my continued learning?

마스터 제목 스타일 편집Q & A

마스터 제목 스타일 편집ASTD 2006~ Now, Let’s Think Bigger!

Korea Management Association Measurement, Evaluation & ROI 2006/05/08 서울대학교...

Documents

Transcript of Korea Management Association Measurement, Evaluation & ROI 2006/05/08 서울대학교...