Chapter 10 – Evaluation in Instructional Design
The authors talk about 2 forms of evaluation for instructional design:
- Formative – when evaluation is done to support a process of improvement. Example – focus groups and prototyping to gage user response and understanding of the design implemented.
- Summative – evaluation done to understand conclusion. Example – A final evaluation done on the results from the previous mentioned focus group and prototyping evaluation results.
The authors also mention the importance of evaluation for the assessment of mastering knowledge. We see this done in many classroom environment through testing. I personally believe that essay writing gives a deeper understanding of the learner’s knowledge than multiple choice. I’m thinking back in particular to being an undergrad student and memorizing information for tests that I don’t think I could recall at all today. Yet, I have a deeper recollection of writing out essays on topics ranging from english literature to earth sciences.
For the logic of evaluation we want to select
- criteria for method of evaluation
- performance standards
- data performance and level of performance
- make a final judgement (results)
Some evaluations methods proposed
Stufflebeam mentions a 4 step evaluation process:
- Context – what environment it will be used in (a needs evaluation), example: a Kiosk in an airport as opposed to a Kiosk in a loud subway station.
- Input – what resources will be used for evaluation
- Process evaluation – development, implementation and effectiveness of the evaluation
- Product evaluation – success in product outcome
To best understand the full scope of a project, evaluation should go through a process of iteration.
Rossi – 5 Domain Evaluation Model (tailored to fit local needs, resources and types of programs):
- Needs Assessment
- Theory for the program
- Implementation Assessment
- Scientific evaluation – this is the impact assessment
- Economic evaluation – this is the efficiency assessment
Kirkpatrick’s Training Evaluation Model – created for evaluating training programs: 4 Different Levels of Evaluation
- Reaction – how do people react to the learning experience
- Learning – what was learned in the training
- Behavior (transfer of knowledge) – was their On the Job behavior changed
- Results – did the training lead to final results desired
I remember training in at work to switch from Quark Xpress to Adobe InDesign for Desktop Publishing. Many of my coworkers were hesitant to change, but were very surprised at how similar both platforms were. Our boss performed an evaluation of our new knowledge during training. This information was collected and documented for future implementation of similar training programs in our company.
Brinkerhoff’s Success Case Method:
The focus here is SCM (Success Case Method):
- plan success case study
- construct visual impact model
- conduct survey that identifies best and worst case scenarios
- conduct in-depth interviews
- communicate evaluation findings
How an evaluation is reported back to individuals being evaluated is crucial in helping people understand not only their concern but similar attitudes and/or changes that need to occur in group settings.
Patton’s Utilization-Focused Evaluation (U-FE):
- evaluation used for specific intended use
- participatory evaluation approach
- meta evaluation – the evaluation of the evaluation
Meta evaluation can be tricky. If the information provided is not true then an evaluation of the evaluation is gonna be flawed.
Chapter 11 – Intro to ROI (Return of Investment)
The focus of this chapter is the importance of money and value and how it is perceived by stakeholders (those who are employing, paying your salaries, own companies, making big investments and placing their trust in a profit return)
Levels of Data:
0 – Input – the scope of the effort
1 – Reaction – the reaction to the project
2 – Learning and Confidence – learning to use the project
3 – Application and Implementation – effective implementation
4 – Impact and Consequences – making the connection between project and business
5 – ROI – the monetary impact compared to project costs
We collect and document data to show the impact of the project. It is far easier to follow trued and tried methods when an ROI is expected than to spend too much time and money researching.
When doing data analysis we have to convert data to monetary value. This kind of visual helps stakeholders understand the benefits of their investments.
When a project manager is writing a proposal he or she has to make a solid case of validating that a solution is needed.
Consistency of data and data analysis helps give projects direction and the credibility for stakeholders.
Chapter 12 – Managing On-Site and Virtual Design Teams
A successful project manager makes objectives and expectation clear for both their team and their clients.
Managing is the process of working with individual to execute the plan at hand.
Leadership in a successful managers is seen when goals and objectives are achieved. Motivation is a key factor.
A successful manager possesses 3 types of abilities:
- cognitive – diagnosing and assessing group environment
- behavioral – adapting group behavior to solve goal or objective
- process skills – communication with groups
Subsequently successful management recognizes the importance of tailoring their management skills for the specific situation. Not all groups can be managed in the same manner, just like not all projects can be approached in the same manner.
Example: Managing different generations in the workplace. This article discusses the 3 major different generations that exist in today’s workforce.
- Gen X
- Gen Y/Millennials
A successful manager should recognize the strengths and weakness of all 3 generations in order to achieve a favorable group outcome.
Effective communicators always provide feedback. Most people who don’t receive feedback at work feel a sense of displacement.
SME (Subject Matter Experts) – in order to build good relationships with them, it is important to include them in the project development from early on. People who participate in projects from inception to completion provide better feedback when change is needed.
How do you build a productive Team?
Be personable. Provide clear feedback. Be open to questions.
Create morale. Build team traditions. Get together outside of work.
The Future of Instructional Project Management
What we are seeing is a trend at remote managing. Many large scale projects involve several project managers, sometimes in different parts of the world and in different time zones.
A successful project manager leads by example while providing the direction and motivation needed for their team.
Chapter 13 – Managing Scarce Resources in Training Organizations.
The 3 main resources a project manager has:
Scoping a project helps define and sort out the resources needed.
3 kinds of resource models:
- scarcity – when demand exceed supply
- equilibrium – when supply equals demand
- inefficiency – over abundance or lack of abundance (extreme opposites of the spectrum)
Many projects are guided by the Economic Cycle (the state of the business environment)
Stages of Economic Cycle:
- Trough (lowest point in business)
Project managers have to consider the magnitude and duration of economic cycles. Observing that short cycles tend to volatile while long cycles tend to be easier to manage.
Anticipation to change in the economic cycle can help save a project from economic disaster. Thus it is highly recommended for project managers to move parallel to the current economic cycle.
Resources monitored through the economic cycle are not fixed but dynamic.