Concerns non-complianceABET 2000 PANEL DISCUSSION
SAE 2001 CONGRESS, MARCH 7, 2001
Chairman, Chuck Allport, Cedarville Univ.
Panelists:
Tom Kenney, Ford Powertrain Research, Evaluator
Jerry Jakubowski, Dean Eng. & Sci., Loyola Marymount Univ.-LA, Evaluator and Evaluatee
Ken Kline, Chair, ME Dept., WSU, Evaluator and Evaluatee
TOM KENNEY’S COMMENTS:
Introduction:
Institution drafts mission and objectives; enables program differentiation
Emphasis on outcomes
Programs must demonstrate how criteria and educational objectives are being met
Emphasis:
Continuous improvement
Focus on process(es)
Are outcomes and assessments linked to objectives?
Is 2 loop process being followed?
Evaluators visit in Sept-Dec. of Year 1
Evaluators study Draft Statement (Self-study report)
Accreditation action – Aug. of Year 2
Evaluation of ABET 2000 Criteria:
Is a process in place?
Are the results documented?
Student portfolios. Results of national examinations
Input from alumni.
Placement data.
Evaluators meet with constituencies (e.g., Industrial Advisory Committee
Questions:
Are the objectives defined?
How were the objectives determined?
Are the objectives consistent with the mission?
Are the constituencies defined? Who, Why?
How many constituencies? (3-6 is ideal, from a workability viewpoint)
[Select 3-6 constituencies and work actively with them.]
How are the constituencies involved in the process?
Are the inputs from the constituencies considered (seriously)?
Do the students provide input?
Outcomes:
What is the expertise at graduation?
Has the expertise been demonstrated?
Processes:
What are the ABET 2000 processes in place?
What does each process do?
Is each process effective?
Who owns each process?
Assessments:
Provide data for the objectives.
Assessments must be linked to outcomes and objectives.
What is measured? How often is it measured?
How are the data used? [Modify processes, if required.]
What are the metrics? Are they correct?
Results:
Have the outcomes been achieved?
What is the quality of the program?
Have the identified improvements affected the results?
Are there plans for additional improvements?
Are the feedback loops working?
Are the constituencies pleased with the results?
Audit Form for Program:
Categories are Concerns, Weakness, and Deficiency in each area.
Sunday evening prior to visit: Evaluation Team reviews student evaluations of program and courses.
ASME Specific Problems Areas:
Both thermal and mechanical systems areas need to be strong. Some institutions emphasize one at a cost to the other.
Computer usage, Design, and Laboratory Experience need to be emphasized in both systems areas.
Credit Hour Dilemma:
Faculty, Program criteria, Constituencies want to increase credit hours.
Students, Administration, Parents, Legislators want to decrease credit hours.
Future Emphasis:
Policy and Procedures Manual
Staff support services (are they in place and effective?)
Course evaluation criteria – what are they?
Academic policies.
Does the Institution support the program objectives?
ABET will be taking a “holistic” view of program objectives.
JERRY JAKUBOWSKI’S COMMENTS:
Dean of Science and Engineering, Loyola Marymount, LA
ABET accredits programs not Institutions
ABET Committees:
EAC Engineering Accreditation Committee
TAC Technical Accreditation Committee
RAC (Other programs) Accreditation Committee
CSAC Computer Science Accreditation Committee
Criteria:
7 for all engineering
Program specific for ECE, ME, etc.
Experience:
Program evaluator, ME, EAC
Fall, 1999 visitations
Possible Accreditation Actions:
NGR Next General Review 6 years
IR Interim Report 3
IV Interim Visit 2
SC Show Cause 1
EAC Terminology:
Concerns non-compliance
Weaknesses Increasing non-compliance
Deficiencies criteria not satisfied
Actions:
One deficiency
SC
One weakness IR
or IV Increasing compliance
No weakness or deficiency
NGR
1999 Fall Visits:
85 schools, 356 programs evaluated
95% EC2000 Criteria
65% NGR
34% IR or IV
1% SC
Primary IR, IV Rationale:
No assessment data
Not closing feedback loop
Not involving constituencies
Secondary IR, IV Rationale:
Poor faculty morale; no cohesiveness
Lack of Institutional support
Poor advising
Courses out of sequence
Lack of engineering design
Final Statements:
School A Assessment process
in place NGR
Expect loop to be closed
School B Recent implementation
of outcomes NGR
Loop not closed, but plans in place.
School C Weakness: limited
constituency involvement IR
Process needed to determine achievement of objective
Need to use assessment results to improve program.
Obj Metrics
Results Improvement
Evidence for satisfactory improvement not demonstrated.
Personal Observations:
There is uncertainty among evaluators: what level of implementation is expected?
There tends to be inconsistency in level of expectations from program to program.
Evaluators now tend to be lenient with respect to new aspects of EC2000
Some evaluators are tending to stray back to old ways of evaluating programs.
LMU’s EC2000 Experience:
CE, EE, and ME evaluated
2 evaluators focused on old criteria
Little time spent in reviewing educational objectives, program outcomes, process for evaluations, and whether assessment method was in place.
No weaknesses or deficiencies were found in any programs.
Conclusions:
EC2000 is a good thing.
Expected level of implementation will go up each year.
Evaluators are putting forth a conscientious effort.
Some evaluators have a difficult time in going from the old, conventional criteria to EC2000.
Team Chairs play a key role in directing and guiding the program evaluators in keeping with the EC2000 criteria.
KEN KLINE’S (WSU) COMMENTS:
Individual department perspective.
Focus on Criterion 3 – Program Outcomes and Assessments:
Take a Literal Interpretation of the Criteria:
Need continuous quality improvement demonstrated.
Can’t have 1 or 2 people accumulating data.
Need sustainability.
How to implement continuous improvement model?
Look at best practices – websites (e.g., WSU)
Talk to all constituencies.
Obtain written feedback from all constituencies.
[Ford Motor “adopted” 6 universities (WSU, MSU, OSU, Univ. of Texas, etc.) a while ago and assisted in implementing continuous improvement processes.]
WSU Experience:
24 faculty members, Industrial Advisory Committee, Student Advisory Committee, Employers, Alumni.
4 course groupings.
Initially faculty members were unaware of content of other courses.
Objectives of each course were listed. There was focus on what was not covered (e.g., Design of Experiments)
Faculty learned objectives of all courses.
12 graduate assistants aided process.
Self-study report was posted on website; hard copies were also distributed.
Course learning objectives are now included in syllabi.
Extract data from student course surveys.
WSU trains its graduates from industry, not for graduate study.
WSU has 2 capstone courses: Thermal fluids and thermal systems.
Instructor prepares self-evaluation of learning objectives.
Instructors state how course objectives contribute to Program outcomes.
Jan/Feb – student assessments, instructor assessments are prepared; evaluations go to 4 course groupings. In ABET years, assessments are done semi-anually.
Industrial experts attend presentations of student project teams.
Graduating seniors are interviewed for impressions on school effectiveness.
A formal meeting in held once a year (» May); all constituencies are invited to participate.
DISCUSSION
EAC is not looking for monumental changes in programs.
A bad student assessment may imply a poor teacher, who should be counseled and perhaps replaced. Insufficient pre-requisites may be a significant contributing factor.
WSU has 136 credits required for graduation. Standardized 4 Credit Hour course modules have been implemented to overcome credit hour vs. work hour inconsistencies, with the later being significantly larger than the former in some cases (1-2 Cred. Hr. really means 3 Cred. Hr.). Assessment data are collected and discussed by faculty. Faculty and students should be completely honest.
Adjunct faculty need to be aware of program objectives and outcomes. Adjuncts should have a long time relationship with the school. There should not be over reliance on adjuncts, especially in the teaching of required courses. Adjuncts should participate in ABET 2000 faculty retreats. Adjuncts should be well known to full time faculty.
Evaluators examine excellent, good, and poor examples of student work, sometimes focusing on the good and poor examples.