Program Performance: Evaluations and Measurement Performance... · Program Performance: Evaluations...

1

Program Performance:Evaluations and

Measurement [Session105]

Tom Schumacher, University of Minnesota

Ken Zeko, KPMG LLP Forensic

John Stoxen, 3M Company

Overview for Program Assessment

� Background

� The foundation and culture

� Commonly used metrics and tools

� Examples

� Questions and comments

2

Background: Why Do a Compliance Program Effectiveness Evaluation?

� Required under Federal Sentencing Compliance Program Guidelines

� Identifies gaps and weaknesses within and across your various programs

� Tells you the “big picture” –How are your doing as an organization?

� Creates leadership support

� Results matter. Period.

Guidelines Standard§8B2.1(b):

(b) Due diligence and the promotion of an organizational culture that encourages a commitment to compliance with the law within the meaning of subsection (a)minimally require the following steps:…

(5) The organization shall take reasonable steps—(B) to evaluate periodically the effectiveness of the organization’s compliance and ethics program

3

“Due Diligence”� Diligence: “Vigilant activity; attentiveness”

� Due Diligence: “Such a measure of prudence, activity, or assiduity, as is properly to be expected from, and ordinarily exercised by, a reasonable and prudent man under the particular circumstances.”

What does “due diligence” mean for your organization?

� What is reasonable and prudent?

� Is there an industry standard?

� Is it the same degree of care used for other management priorities within a prudent organization?

� What are the hallmarks of due care for management priorities?• Resources to do it right

• Metrics + Trending to measure progress

• Accountability for results

� (owned within operational management structure, not HR, Compliance, Counsel, Audits, etc.)

• Rewards/incentives

• Responsive action to improve points of weakness

4

Quick Quiz. How is your organization doing with compliance

effectiveness evaluation?

Completed Evaluation; In good shape

5

Completed Evaluation. Would rather not discuss results

“File 13ed” the idea; in other words

6

“If you find that a person had a

strong suspicion that things were not

what they seemed or that someone

had withheld some important facts,

yet shut his eyes for fear of what he

would learn, you may conclude that

he acted knowingly, as I have used

that word.”

Results of “File

13”

Quiz #2

� Write down the approximate number of employees in your organization

� Divide that number in half

� THAT IS THE NUMBER OF PEOPLE, ON AVERAGE, WHO HAVE WITNESSED SIGNIFICANT MISCONDUCT IN THE LAST YEAR AT YOUR ORGANIZATION

• Source: cross industry studies (two separate studies)

7

What to measure� Your program itself

• Look to the Guidelines elements� E.g. do you have training, is it working, do you have a Code, is it useful, is it followed, etc., do you have policies, are they accessible, do people follow them, etc. Just track the Guidelines.

� Outcomes: • Is the dang thing working?

� How• Compliance and Ethics Metrics

� Delivery Options• Many options, may depend upon level of reliability sought

� Surveys (as part of a broader survey, as an independent survey, from a selected subset, etc.), interviews, consultants, peers, etc.

Common Compliance and Ethics Metrics

� Observations of perceived misconduct� Willingness to report misconduct/violations and violations in fact are

reported when they are perceived to occur� Perceptions about the organization’s responsiveness to misconduct� Fear of retaliation for reporting concerns� Willingness to seek help within the organization for ethical issues� Supervisors demonstrate/pay attention to ethics� Leadership demonstrates/pays attention to ethics� Open discussion of ethics in the workplace encouraged� Ethical behavior rewarded at all levels� Unethical behavior punished at all levels (management accountability)� Perceptions of fair treatment in the workplace� Employee willingness to deliver “bad news” to management� Employee knowledge of workplace rules� Employee “commitment” to the organization� Confidence in preparedness to respond to ethical situations

� --May have many related/sub-questions organized in different compliance domains; may just ask a few questions in a broad employee survey. You have lots of options.

8

Resources (samples only)� Academic Research

• E.g. Managing Ethics and Legal Compliance: What works and what hurts, Trevino et al, 41 California Management Review, No. 2 1999

� Consultants• e.g. Executive Branch Employee Ethics Survey 2000,

http://www.usoge.gov/pages/forms_pubs_otherdocs/fpo_files/surveys_ques/srvyemp_if_00.p

df

• E.g. United Nations Organizational Integrity Survey 2004, http://www.un.org/News/ossg/sg/integritysurvey.pdf

� Reports• E.g. National Business Ethics Survey, How Employees View Ethics in their

Organizations, 1994-2005, available from www.ethics.org

• KPMG Forensic, Integrity Survey 2005-06, www.us.kpmg.com/news/index.asp?cid=2051

� Benchmarking• ERC Benchmarking Initiative (7 Questions for licensed use), more at

http://www.ethics.org/ecoa-benchmark/

Example: University of Minnesota

� Process

• Embedded within broader employee satisfaction survey

• 6 +1 question

� 6 culture

� 1 do they know about the hotline

� Observation/lesson

• Will look harder at external benchmarking metrics next time.

9

Survey QuestionsI have experienced or observed significant misconduct (violation of law, workplace rules, or significant University policy) in my unit/department within the last twelve months?

Yes No

If Yes,

If the misconduct was not known by responsible University officials, did you or someone else report it to responsible University officials or the University’s confidential reporting service?

Yes, Yes, No, Don’t

I reported itothers reported it it was not reported know

If Yes,

Do you believe responsible University officials took appropriate corrective action?

Yes No Don’t Know

54321

University leadership

demonstrates integrity and ethical behavior.

54321

I believe I would be protected from retaliation if I report a

suspected violation.

54321

I know where to report violations of

law or policy (such as the University's confidential reporting line.)

Strongly Agree

Agree to Some ExtentUncertain

Disagree to Some Extent

Strongly Disagree

10

Example:

Bi-Annual Business Conduct

Self-Assessment Process

Audit Committee

Exec VP

CEO

Div. VP/ Sub. MD

Business Conduct

Committee

Central

Compliance

Department

Compliance

Contact

Compliance

Contact

BoardOversight

ExecutiveDirection

Operational

Initiatives

Business Conduct Program Structure

3

11

3

BACKGROUND

BUSINESS CONDUCT SELF-ASSESSMENT PROCESS

Revised March 2007

3M Confidential

This document provides background information on 3M’s Business Conduct Self-Assessment (Self-Assessment)

process and on 3M’s tradition of legal and ethical behavior. It can be shared with any 3M employee who will be

involved in completing a Self-Assessment.

3M’s Tradition of Legal and Ethical Compliance

3M has a long history of decentralized leadership in its worldwide operations. Senior management of business

units, international subsidiaries and staff groups have been given the freedom—and the responsibility—to design

and implement programs needed to succeed in their particular operations. This freedom has been a key driver of

3M’s success. This responsibility has always included creation of an environment in which all employees act in

accordance with 3M’s values of uncompromising honesty and integrity.

Over the years, 3M has learned that some aspects of an effective compliance and ethics program can most

efficiently handled on a global basis. 3M first published business conduct policies in 1988, to provide direction on

the universal compliance and ethics principles that guide 3M’s global operations. The Business Conduct

Committee was created in 1991 and was given responsibility for putting together a program to carry out the global

aspects of 3M’s business conduct program, including:

• Creating business conduct policies

• Communicating policies and expectations to 3M’s global workforce

• Providing training to help employees apply business conduct policies to

their jobs

• Ensuring employees have a confidential and trusted means by which they can report suspected business

conduct violations

• Fully and fairly investigating all reports of suspected violations

3

REPORT

2007 BUSINESS CONDUCT SELF-ASSESSMENT

____ DIVISION

Revised June 2007

3M Confidential

Section 1: Division Overview

A. Division Structure

1. Name and job title of the 3M employee with ultimate accountability for legal and ethical compliance in your

division.

2. Compliance Contact (name, regular position and effective date of Compliance Contact appointment).

3. Number of U.S.-based, current full-time equivalent 3M employee positions in your division.

4. List operating subsidiaries, joint ventures, etc.

B. Business and Industry

1. List your key customers (by industry segment, if appropriate) in the market channel(s) in which you

participate.

2. If you have government customers outside the United States, briefly list which governments and types of

products sold.

3M

INSTRUCTIONS

FOR DIVISION COMPLIANCE CONTACTS


Revised June 2007

3M Confidential

These Instructions are for use by division and Big B Compliance Contacts in completing the 2007 Business

Conduct Self-Assessment (“Self-Assessment”).

Introduction

Compliance with law is only the beginning of 3M’s business conduct program. We require compliance with

the law, but we expect much more of ourselves. We have worked hard to build a culture where employees

not only follow the letter of the law, but also the spirit of the law.

The Self-Assessment process is intended to help 3M business leaders measure and improve upon the culture

of legal compliance and ethical business conduct in their part of 3M’s worldwide operations. At the

direction of the 3M Business Conduct Committee, all 3M operating divisions complete this version of the

Self-Assessment.

The Self-Assessment is not intended to assess business or financial risk. It is not an audit or an investigation.

Instead, the Self-Assessment is designed to help your division identify existing and emerging business

conduct issues in its business activities and to think about how those issues can be dealt with most

effectively, recognizing that risk can never be eliminated entirely. This type of periodic analysis is needed to

ensure your organization has an effective compliance and ethics program. For more on the importance of an

effective compliance program and why 3M uses the Self-Assessment, read the “Background on the Self-

Assessment Process” document.

A. Process for Completing the Self-Assessment

To complete the Self-Assessment, you will need the following documents, all of which are available in the

Compliance Contact TeamRoom in the Business Conduct Self-Assessment category, in the folder titled

“2007 Self-Assessment Materials for Divisions”:

• Compliance Contact Tips for 2007 Business Conduct Self-Assessment

3M

COMPLIANCE CONTACT TIPS


Revised June 2007

3M Confidential

These suggestions should help Division and Big B Compliance Contacts efficiently complete the Business

Conduct Self Assessment ("Self-Assessment"). Each organization in the company is unique, so use your judgment

as to whether these ideas will work in your business unit.

Those of you who were Compliance Contacts in 2005 should notice a number of changes throughout the Self-

Assessment that have been made to make the process more focused and streamlined. Divisions which have

already completed the 2007 Self-Assessment have noticed a substantial reduction in the time required, compared

with 2005.

If questions come up while you complete your Self-Assessment, please call or e-mail the Central Compliance

Department resources listed in the Background document.

Tips for Division Compliance Contacts

1. Read the materials. Start by reading the Background, Instructions and Report documents in the

Compliance Contact TeamRoom in the “2007 Division Self-Assessment Materials” folder. You cannot

map out an efficient strategy for completing the Self-Assessment until you visualize the whole process.

2. Seek assistance early. As you read the Instructions and Report, note who in your organization has the

information you need to complete various sections. By asking for assistance on these sections early, you

can give people weeks, rather than days, to respond.

3. Keep your Operating Committee informed. Only Division Vice Presidents/General Managers and

Compliance Contacts received the kickoff e-mail notice from the Central Compliance Department about

the upcoming Self-Assessment. It is your responsibility to inform the other Operating Committee

members about the Self-Assessment process. You may forward the original kickoff e-mail to them if you

12

Section 1: Overview

• Follow Up on Action Items from Previous Self-Assessment

• Business Environment

• Operational Change Risk Areas

• Business Conduct Culture and Ethical Leadership

• Training and Evaluation

Section 2: Risk Assessment

• Senior management discusses 26

business conduct risk areas on C&E

matrix

• Consensus rankings assigned for

– Likelihood of violation

– Severity of violation

– Adequacy of training

• C&E matrix prioritizes risks

14

Section 3: Objective Metrics

• Online Training Completion

• 3M Standard Opinion Survey Ethics

Questions

• Audit Findings Related to Ethics

• Business Conduct Violations

15

Section 4: Identification of Action Priorities

Operating Committee & Compliance

Contact:

– Review Self-Assessment Report

– Reach Consensus on Top 3+ Opportunities to Improve Compliance Culture

– Create Specific Action Plans For Each Priority Item with Person Responsible and Due Date

Business Conduct Self-Assessment Report

Widgets Business

Business Conduct Committee Presentation

March 1, 2007

16

2006 Online Training Completion Percentage

87889288939482Division 6

100100100100100100100Division 2

88.391899292.695.993All Division Average

93959592949690Business-Wide Average

90929697949396Division 8

100100100100100100100Division 7

1009897999810098Division 5

100100100100100100100Division 4

94979998959999Division 3

100100100100100100100Division 1

Advertisin

g

Preventing

Harassme

nt

(Supervisor Edition)

Preventing

Harassme

nt

(Employee Edition)

Business

Conduct

Under-

standing

the FCPA

Employ-

ment Law

for

Leaders

What You

Need to

Know

about

E-mail

Division

“I can report unethical practices without fear of

reprisal.”

89 11 0

90 10 0

82 14 4

81 15 4

90 5 0

83 16 1

71 18 11

0% 20% 40% 60% 80% 100%

Division 6

Division 5

Division 4

Division 3

Division 2

Division 1

3M Avg. 2006

% Fav % Neutral % Unfav

17

Priority 1: Confidential Information

Division 1: Employees have access to many types of confidential information and this was determined the highest risk area. Legalcounsel will conduct a 30 minute group training on the importance of confidentiality with reference to the appropriate sections in the Business Conduct Policies before July 1.

Division 2: Extensive external interactions in all functions and the division is exposed to risk if these are not handled properly. Ensure 100% compliance with corporate training and review key confidential risk areas by function. Owned by operating committee functional leaders and will be completed by end of April.

Division 3: The organization’s strategic direction and growth requires significant M&A and alliance activity with many types of confidential information being handled. Legal counsel to developa specialized briefing for Steering Committee to be presented byApril 30. HR to review and enhance module for new employee and refresher training by April 30.

Division 4: Confidential information is shared with customers at multiple levels and by multiple functions. Develop Division Confidentiality Policy and Communications Plan by Compliance Contact and a designated team with efforts initiated by May 1.

A D V I S O R Y

K P M G F O R E N S I C

Integrity Survey 2005-2006

18

© 2006 KPMG LLP, the U.S. member firm of KPMG International, a Swiss cooperative. All rights reserved. KPMG Forensic is a service mark

of KPMG LLP. Forensic advisory and expert witness services may be subject to legal and regulatory restrictions..

Objectives

Provide a behind-the-scenes look at corporate fraud and misconduct in the post-Sarbanes-Oxley era

Offer organizations insights as they consider:

Their exposures to fraud and misconduct risks

The effectiveness of programs and controls relied on to mitigate

fraud and misconduct risks

Compare findings to previous KPMG Integrity Survey



Methodology

Blind national survey of pre-screened working adults that fell into demographics spanning:

All levels of job responsibility

16 job functions

11 industry sectors

4 thresholds of organization size

Paper–based survey conducted between Nov. ‘04 and March ‘05

Benchmarked results where possible against findings in 2000 survey

4,056 respondents

Confidence level 95%; Precision level (margin of error) +/- 2%

19



Key Findings

Level of misconduct remains unchanged

74% reported that they have observed misconduct in

the prior 12 month period compared to 76% in 2000

Half reported observing serious misconduct

Conditions that facilitate Management’s ability to prevent, detect, and respond to fraud and misconduct have improved since 2000

Pressure to engage in misconduct is down

Confidence in reporting concerns to management is up



Key Findings

Employees in companies with comprehensive ethics and compliance programs reported more favorable results across the board than those employees at companies without such programs

At companies with ethics and compliance programs:

Fewer observations of misconduct

Higher levels of confidence in management’s commitment to

integrity

20



Prevalence of Misconduct During the Prior 12 Months

76

74

0 20 40 60 80 100

76

74

0 20 40 60 80 100

2000

2005



Prevalence of Misconduct That Could Cause a“Significant Loss of Public Trust if Discovered”

49

50

0 20 40 60 80 100

49

50

0 20 40 60 80 100

2000

2005

21



Root Causes of Misconduct

33

46

47

49

49

52

55

57

0 20 40 60 80 100

33

46

47

49

49

52

55

57

0 20 40 60 80 100

Feel pressure to do “whatever it takes” to meet business targets

Lack understanding of the standards that apply to their jobs

Believe that their code of conduct is not taken seriously

Lack resources to get the job done without cutting corners

Believe they will be rewarded for results, not the means used to achieve them

Believe policies or procedures are easy to bypass or override

Fear losing their job if they do not meet targets otherwise

Are seeking to bend the rules or steal for their own personal gain



Propensity to Report Misconduct

6

10

38

53

81

0 20 40 60 80 100

6

10

38

53

81

0 20 40 60 80 100

Notify supervisor oranother manager

Call the ethics orcompliance hotline

Look the other wayor do nothing

Try resolving the matter directly

Notify someone outsidethe organization

22



Channels Employees “Feel Comfortable” Using to Report Misconduct

Note: Chart does not foot to 100% due to rounding.

32

39

44

52

53

56

57

62

78

35

37

29

28

28

23

20

20

9

33

24

27

20

18

21

22

19

12

32

39

44

52

53

56

57

62

78

35

37

29

28

28

23

20

20

9

33

24

27

20

18

21

22

19

12

0 20 40 60 80 1000 20 40 60 80 100

Supervisor

Local managers

Peers or colleagues

Human resources department

Ethics or compliance hotline

Legal department

Senior executives

Internal audit department

Board of directors oraudit committee

Agree Unsure Disagree



Channels Employee “Feel Comfortable” Using for Advice and Counsel


40

48

54

61

61

64

75

78

37

27

28

21

24

19

14

8

23

25

17

18

15

17

11

13

40

48

54

61

61

64

75

78

37

27

28

21

24

19

14

8

23

25

17

18

15

17

11

13

0 20 40 60 80 1000 20 40 60 80 100

Supervisor

Local managers

Peers or colleagues

Human resources department

Ethics or compliance hotline

Legal department

Senior executives

Internal audit department


23



Perceived Outcomes of Reporting Misconduct

89

39

47

52

64

67

10

47

32

31

22

22

15

21

17

14

11

189

39

47

52

64

67

10

47

32

31

22

22

15

21

17

14

11

1

0 20 40 60 80 1000 20 40 60 80 100

Appropriate action wouldbe taken

I would be protectedfrom retaliation

My report would behandled confidentially

I would be satisfied withthe outcome

Those involved would be disciplined fairly regardless of

their position

I would be doing the right thing





Tone at the Top:Perceptions About the CEO and Other Senior Executives

65

70

67

57

55

48

65

22

19

21

24

21

25

21

14

10

12

18

23

27

15

65

70

67

57

55

48

65

22

19

21

24

21

25

21

14

10

12

18

23

27

15

0 20 40 60 80 1000 20 40 60 80 100

Are positive role models for the organization

Know what type of behavior really goes on inside the organization

Are approachable if employees have questions about ethics or need to

deliver bad news

Value ethics and integrity over short-term business goals

Set achievable targets without violating my organization’s

code of conduct

Would respond appropriately if they became aware of misconduct

Set the right “tone at the top” on the importance of ethics and integrity

Agree Unsure DisagreeNote: Chart does not foot to 100% due to rounding.

24



Local Tone:Perceptions of Local Managers and Supervisors

72

73

73

67

77

72

72

16

15

16

19

12

15

13

12

12

11

14

11

13

15

72

73

73

67

77

72

72

16

15

16

19

12

15

13

12

12

11

14

11

13

15

0 20 40 60 80 1000 20 40 60 80 100

Are positive role models for the organization

Know what type of behavior really goes on inside the organization

Are approachable if employees have questions about ethics or need to

deliver bad news

Value ethics and integrity over short-term business goals

Set achievable targets without violating my organization’s

code of conduct

Would respond appropriately if they became aware of misconduct

Set the right “tone at the top” on the importance of ethics

and integrity




Team Culture and Environment:Perceptions of Individual Teams and Work Units

55

64

57

62

75

77

64

78

24

17

21

17

14

13

18

9

20

19

22

22

11

10

18

13

55

64

57

62

75

77

64

78

24

17

21

17

14

13

18

9

20

19

22

22

11

10

18

13

0 20 40 60 80 1000 20 40 60 80 100

People feel motivated and empowered to “do the

right thing”

People apply the right values to their decisions and behaviors

People feel comfortable raising ethics concerns

The opportunity to engage in misconduct is minimal

The ability to conceal misconduct is minimal

People share a high commitment to integrity

The willingness to tolerate misconduct is minimal

Adequate checks are carried out to detect misconduct


25

Presenter’s contact details

Kenneth Zeko

KPMG LLP

(214) 840-6497

[email protected]

www.kpmg.com

The information contained herein is of a general nature and is not intended to address the circumstances of any particular individual or entity. Although we endeavor to provide accurate and timely information, there can be no guarantee that such information is accurate as of the date it is received or that it will continue to be accurate in the future. No one should act on such information without appropriate professional advice after a thorough examination of the particular situation.



Program Performance: Evaluations and Measurement Performance... · Program Performance: Evaluations...

Documents

Transcript of Program Performance: Evaluations and Measurement Performance... · Program Performance: Evaluations...