Assessment, Evaluation & Continuous Improvement The Experience of King Abdulaziz University Jubail...
-
Upload
kristina-oconnor -
Category
Documents
-
view
218 -
download
0
Transcript of Assessment, Evaluation & Continuous Improvement The Experience of King Abdulaziz University Jubail...
Assessment, Evaluation & Continuous Improvement
The Experience of King Abdulaziz University
Jubail Industrial College - Oct. 23, 2011
Ali M. Al-BahiProfessor of Aeronautical Engineering
Director of the Academic Accreditation Unit
األنبياء أشرف على والسالم والصالةوالمرسلين
وصحبه آله وعلى محمد سيدناأجمعين
JIC Presentation
Who are we? ABET TAC Accreditation Assessment and Evaluation of Program
Educational Objectives Assessment and Evaluation of Student
Outcomes Conclusions
Contents
Oct. 2011 3
Who Are We?
4Oct. 2011 JIC Presentation
King Abdulaziz University Profile KAU is Saudi Government financed institution. Founded in 1967 in Jeddah, 70 Km west of Makah holy city 18 Faculties/Colleges & 9 Deanships 110+ Departments offering a diversity of specializations. 4100+ Faculty members & 2100+ Administrative and support
personnel 95000+ Students. 30 Scientific Chairs 6 Centers of Excellence 16 Scientific Research Centers
5JIC PresentationOct. 2011
College of Engineering Profile Founded in 1975 G 2700+ Students 7000+ Alumni to date 9 Departments offering 12 programs. 240+ Faculty members 100+ Administrative and support personnel
6JIC PresentationOct. 2011
ABET Substantial Equivalency The 12 Programs started introducing themselves to the world
of accreditation and quality education in 2002 through ABET SUBSTANTIAL EQUIVALENCY.
Although this was not A FULL ACCREDITATION, it represented a huge step forward.
This was the only available international evaluation at that time and allowed some deviations from in-land accreditation:
“Program is comparable in educational outcomes but may differ in format or method of delivery”
Nevertheless the wheel of quality started to rotate
7JIC PresentationOct. 2011
ABET Full AccreditationIn 2003, once the successful ABET visit to evaluate the 12 engineering programs was completed, The College started to prepare for the next evaluation visit in 2008 which coincided with:
•the introduction of the new outcome-based Engineering Criteria EC 2000, •the phase-out of the substantial equivalency in favor of full accreditation.
EC 2000 represented a major paradigm shift influenced by pressures from industry and global competition. It addressed the effectiveness of engineering programs by focusing on assessment & evaluation process that assures the achievement of educational objectives & outcomes.
In August 2008 the 12 KAU Engineering Programs obtained the full ABET Accreditation under EC 2000
8JIC PresentationOct. 2011
Academic Accreditation Unit (AAU) Established in May 2005 4 faculty members (working
part time) Initial mission: Qualify the
education system in the Faculty of Engineering to Meet ABET EC 2000 Standards.
9Oct. 2011
AAU: Quality Assurance In KAU Engineering Education
JIC Presentation
ABET TAC Accreditation
10Oct. 2011 JIC Presentation
ABET The Accreditation Board of Engineering and Technology ,
actually known as ABET Inc. The recognized accreditation entity for college and university
programs in applied science, computing, engineering, and technology in the US.
A federation of 30 professional and technical societies in these fields.
For over 77 years, ABET has provided leadership and quality assurance in higher education.
Currently ABET accredits some 2,800 programs in more than 600 colleges and universities inside the US and worldwide.
11JIC PresentationOct. 2011
Elements of ABET Criteria1. Definitions2. General Criteria: apply to all programs accredited by
an ABET commission. Each program accredited by an ABET commission must satisfy every Criterion in the General Criteria for that commission.
3. Program Criteria: provide discipline-specific accreditation criteria. Programs must show that they satisfy all of the specific Program Criteria implied by the program title. Any overlapping requirements need be satisfied only once.
12JIC PresentationOct. 2011
ABET-TAC Definitions
Program Educational Objectives: Broad statements that describe what graduates are
expected to attain within a few years of graduation. Program educational objectives are based on the needs of the program’s constituencies.
Student Outcomes: Describe what students are expected to know and be
able to do by the time of graduation. These relate to the knowledge, skills, and behaviors that students acquire as they progress through the program.
13JIC PresentationOct. 2011
ABET-TAC Definitions (Continued) Assessment : one or more processes that identify, collect, and
prepare data to evaluate the attainment of student outcomes and program educational objectives. Effective assessment uses relevant direct, indirect, quantitative and qualitative measures as appropriate to the outcome or objective being measured. Appropriate sampling methods may be used as part of an assessment process.
Evaluation: one or more processes for interpreting the data and evidence accumulated through assessment processes. Evaluation determines the extent to which student outcomes and program educational objectives are being attained. Evaluation results in decisions and actions regarding program improvement
14JIC PresentationOct. 2011
ABET-TAC Continuous Improvement
Criterion 4
The program must regularly use appropriate, documented processes for assessing and evaluating the extent to which both the program educational objectives and the student outcomes are being attained. The results of these evaluations must be systematically utilized as input for the continuous improvement of the program. Other available information may also be used to assist in the continuous improvement of the program.
15JIC PresentationOct. 2011
Assessment & Evaluation of Program Educational Objectives (PEOs)
16Oct. 2011 JIC Presentation
Establishment of PEOs• Broad statements that describe what graduates are expected
to attain within a few years of graduation. • They should be consistent with the mission of the institution,
the needs of the program’s various constituencies, and ABET criteria.
• They should be documented (website, publications, posters, etc.)
• They should be established with the full involvement of all program constituencies.
• Program constituencies include Alumni, Employers, Students/ parents, faculty, institution management.
17Oct. 2011 JIC Presentation
18Oct. 2011
ABET Committee
defines/redefines PEOs
DraftsO.K.?O.K.?
Dept. Council
Approval
Publish PEOs
Other Similar Programs Worldwide
Program Mission
ABET Criteria
Final DraftStart
Input from all Constituencies
Assess PEOs Adequacy
NoNo
O.K.?
No Assess PEOs Achievement
O.K.?Modify Outcomes and/or Curriculum
Assess Outcomes
Define/redefine Outcomes
Yes
No
3-YearCycle
Yes
Input from all Constituencies
Departmental Council
Yes
Outcomes Achievement
Cycle
PEOs Adequacy Cycle
Outcomes Adequacy
Cycle3-YearCycle
AnnualCycle
Input from off campus Constituencies and/or
alumni data base
Establishment & Achievement of Program Educational Objectives
Establishment of PEOs -Continued
JIC Presentation
Assessment of PEOs
19
Effective assessment uses relevant direct, indirect, quantitative and qualitative measures as appropriate to the objective being measured.• Assessment of PEOs should be carried outside the institution -
since you need to evaluate the achievement of the graduates.
• You should target your graduates (alumni) and their employers (and not the students or the faculty).
• You should assess and evaluate:1. Adequacy of PEOs to the needs of your constituencies.2. Achievement of PEOs by your graduates.
Oct. 2011 JIC Presentation
Modifying PEOsPublish PEOs
Assess PEOs Adequacy
O.K.?
No Assess PEOs Achievement
O.K.?Modify Outcomes and/or Curriculum
Assess Outcomes
Define/redefine Outcomes
Yes
No
3-YearCycle
Yes
Input from all Constituencies
Outcomes Achievement
Cycle
PEOs Adequacy Cycle
Outcomes Adequacy
Cycle3-YearCycle
AnnualCycle
Input from off campus Constituencies and/or
alumni data base
Assessment & Evaluation of Program Educational Objectives
JIC PresentationOct. 2011 20
Adequacy and Achievement of PEOs
21
PEOs Adequacy cycle (3 year cycle, twice during an ABET accreditation cycle). It may result in redefinition of PEOs.
PEOs achievement cycle (normally carried in the same time as the adequacy cycle). It may result in modifying the student outcomes or enhancing of their implementations.
Oct. 2011 JIC Presentation
Direct Assessment of PEOs
22Oct. 2011
Facts Finding Questionnaire (Versus opinion-based surveys) Employer survey (every four years) Alumni survey (every three years)
Archival Data Placement records (mean time to find a job) Admissions to and success in graduate programs
(if applicable)
Inputs from the Industrial Advisory Board
JIC Presentation
Direct Assessment of PEOs - Continued
Define for each Program Education Objective a set of:
1. Indicators.2. Assessment tools.3. Performance criteria for each assessment
tool.4. Time line for assessment and evaluation.
23Oct. 2011 JIC Presentation
Assessment and Achievement of Student Outcomes (SOs)
24Oct. 2011 JIC Presentation
25
Establishing Student Outcomes
Oct. 2011
• ABET is an outcome-based approach.• Your Student Outcomes should be documented.• They should foster the attainment of your PEOs.• There should be a documented process to review
and revise your SOs.• The students should demonstrate achieving SOs
before graduation.JIC Presentation
26Oct. 2011
Assess Outcomes Through Courses & Constituencies
Define Assessment Tools
SOs Achieved?
Improve Courses/ Curriculum
No
AnnualCycle
Yes
Implement Outcomes into
Courses & Curriculum
3-YearCycle
Outcomes Adequacy
Cycle
Outcomes Achievement
Cycle
3 cycles Completed?
Yes
No
Results from Previous PEOs
3-Year Assessment Cycle
Input from Constituencies
Define/modify Outcomes
Define Outcome
Elements & Attributes
New or Modified PEOs, if Any
JIC Presentation
Outcome Elements and Attributes
Development of each outcome intoOutcome Elements: different abilities specified in a single outcome that would generally require different assessment measures.
For each outcome element define outcome attributes or indicators: measurable actions that explicitly demonstrate mastery of the abilities specified in an outcome element, at each level of learning.
27JIC PresentationOct. 2011
IntendedLearning
Outcomes ofthe Lesson
IntendedLearning
Outcomes ofthe Unit
IntendedLearning
Outcomes ofthe Course
Deliver Forward
Design Backward
Infusion of Student Outcomes into Courses
StudentOutcomes ofthe Program
28JIC PresentationOct. 2011
Assessment of Student Outcomes • Direct Measures
– Course assessment (every semester)– Internship evaluations.
• Indirect Measures– Exit interviews – Focus groups (every year)– Exit survey (every semester)
• Yearly evaluations
29JIC PresentationOct. 2011
Assessment and Grading• Both intend to identify what the students have learned. • Grades don’t identify which specific student learning outcomes
and at what levels students have learned. • Some course grades include additional student behaviors that
are not related to student learning outcomes .• For example, a grade of B in a mathematics course indicates a
student learned an acceptable level of math, but not specific math knowledge or skills or what areas need improvement.
• Grades need to be clearly linked and aligned to learning outcomes and/or rubrics to be sufficient as direct evidence for assessment purposes.
30JIC PresentationOct. 2011
Key Courses Approach
31
• Assessment of SOs is the task of the program.• The program is required to collect SOs assessment data supported
by convincing evidences to demonstrate that the students achieved all SOs.
• Each course is responsible to introduce some of these outcomes, enhance the students abilities to achieve them, and collect convincing evidences transferable to the program committee responsible for evaluation and continuous improvement actions.
• It is sufficient that each course will do this for only a limited number (2 or 3) of the SOs (Known as critical outcomes, the course is a key course for these SOs) if and only if compulsory courses together cover all outcomes with reasonable redundancy.
JIC PresentationOct. 2011
Course-level assessment of key outcomes
32JIC PresentationOct. 2011
What is a rubric A rubric is a descriptive scoring guide composed of
• Things you are looking for in an assignment• Guidelines for evaluating each of those things.
A rubric is some kind of scorecard that breaks down a written or demonstrated assignment into manageable, observable pieces.
A rubric is a descriptive scoring/grading tool that is generally used for subjective assignments.
33JIC PresentationOct. 2011
Workshop Assessment Rubric
34
LevelCriteria
4 3 2 1
Intended Outcomes
Intended outcomes were manageable, clearly evident throughout and strongly supported by learning activities.
Intended outcomes were clearly stated, evident in lecture and supported by learning activities.
Intended outcomes were stated, but lecture and learning activities were not clearly organized to achieve them.
Intended outcomes were not stated and unclear. Lecture was unfocussed, and any learning activities did not seem to directly support any particular outcome.
Expertise
Speaker’s expertise in content area was remarkable.
Speaker’s expertise in content area was evident.
Speaker’s expertise in content area was unclear or minimal.
Speaker’s expertise in content was not evident.
Presentation
Presentation was well-organized and easy to follow. It was interesting.
Presentation was clear, and easy to follow, using a standard format.
Presentation followed a main idea, but was repetitious or tangential to no purpose.
Presentation was disorganized and therefore difficult to follow.
Logistics
Media and learning materials were used effectively to enhance the presentation.
Media and learning materials supported the presentation appropriately.
Some media and learning materials did not contribute to the presentation.
Media and learning materials were missing, distracted from the presentation or poorly used.
Learning Engagement
I was meaningfully engaged in learning, making connections to my current professional practice and seeking new learning for continuous improvement.
I was actively engaged in learning and sought to find aspects to contribute to my professional practice.
I chose to be involved, and followed instructions to some degree.
My involvement in learning was passive. I chose not to be engaged in the presentation.
Oct. 2011 JIC Presentation
35
Conclusions To get accredited we were required to demonstrate
to ABET evaluators that:1. WE KNOW WHAT WE DO.2. WE KNOW WHY WE DO IT.3. WE KNOW HOW TO DO IT.4. WE DO IT WELL.5. WE CAN PROVE IT.6. WE RECEIVE INPUT AND FEEDBCAK.7. WE HAVE A PROCESS TO MAKE CONTINUOUS IMPROVEMENTS.
To get accredited you are required to have:1. WELL LAID-OUT PROCESS, and 2. INVOLVEMENT OF THE ENTIRE FACULTY IN THE PROCESS.
JIC PresentationOct. 2011
رب لله الحمد أن دعوانا رب وآخر لله الحمد أن دعوانا وآخرالعالمينالعالمين