One of the first presentations I sat in on at ASTD-ICE 2014 was on Maximizing Informal Learning’s Impact: A New Approach to Instruction Design given by Michael Leimbach. I attended the presentation because informal learning is a huge interest of mine. Here is a PDF of his presentation. Read the rest of this entry
I personally was looking forward to this session on evaluating informal learning by Saul Carliner, Director of the Education Doctoral Program and an Associate Professor at Concordia University. The reason I was excited about this presentation is because it very closely related to my recent dissertation, A study of informal learning among University of Wyoming Extension educators, soon to be posted on ProQuest.
Kirkpatrick Levels of Evaluation
Carliner began with a quick review of Kirkpatrick Levels of Evaluation:
- Level 1 – Satisfaction
- Level 2 – Learning
- Level 3 – Behavior
- Level 4 – Impact
He introduced a number of informal learning scenarios and explained the challenges of measuring with Kirkpatrick Levels of Evaluation.
Scenario – A manager wants you to collect Level 1 evaluations for on-the-job training.
- Hard to do a satisfaction survey because only onesy and twosy, hard to be candid.
Scenario – A training administrator wants enrollment information for the quick tour of the new Learning Management System, which the department uses to “sell” the system to users.
- No importance to collect the data.
Scenario – A product developmenager asks you to report on the extent of learning from the online help provided with the software her team develops.
- How to identify who is using the system.
Scenario – A board member of your nonprofit wants data on the impact of your confidential health information site that your agency publishes.
- HIPAA issues
Carliner finds that Kirkpatrick Levels of Evaluation does not work for informal learning. He thinks Kirkpatrick Levels of Evaluation is a useful tool, but not in this case.
Carliner began exploring a definition of informal learning. Basically, he explained that it is how much the learner has control over process, location, purpose, content, and consciousness. In some cases, worker controls and in other cases employer controls.
Carliner explained that employees use informal learning to fill gaps between the classroom and the workplace. It is the practical application of concepts. These are tricks that help novices become a journeyman. According to some studies, informal learning is the source of 70 percent of all work-related knowledge. Evaluating informal learning “fosters a culture of reflection and analysis to improve performance” (Laiken et al.)
Measuring informal learning
Carliner encouraged us to learn from and adopt methods used by other industries to measure informal learning. These industries included museums, marketing, and Web communications.
Museums and informal learning
According to Carliner, museums coined the term informal learning… now free-choice learning. Museums discovered that sometimes we learn because we want to, and sometimes we learn because others dragged us along. Museums use a number of techniques for measuring learning. They read the museum comment book. They measure how long people stay. Museums control space so you can read labels and information board. They use layout to guide the participant along. They also use self-selected survey methods.
Marketing agencies are very skilled at measuring direct marketing, and we should learn from their methods. Marketing agencies also rely on recognition studies. Are learners tied to your brand?
Carliner also offered a number of techniques used by Web communications such as counting usage stats, running analytic reports, placing satisfaction surveys on every screen, conducting period surveys, and conducting usability tests.
Questions to be answered
- What is the satisfaction with various resources used for informal learning?
- What did workers learn?
- In what ways does the organization benefit from informal learning by workers?
- What is the extent of participation in various informal learning activities?
- Which informal learning efforts that the organization formally supports are providing tangible benefits to the organization?
- How can workers receive recognition for their learning?
Basically, learning and development is interested in what resources are being used by employees to learn. We also need to know how people are learning so they can be recognized and advanced in their employment for skills developed informally. Self-assessments are less painful than exams, e.g., PowerPoint self-assessment. There are other ways to identify how workers learned informally such as portfolios and coaching interviews.
Process portfolio – individual reflects on each item to identify strengths and weaknesses. This can be a challenge to do. Here are instructions for creating a portfolio.
Coaching interview – Simply, ask employees what they know, and take an inventory.
Recognizing prior learning
How can you formally recognize the competencies workers have developed through their informal learning efforts?
- recognized acquired competencies
- employee education records
- skills assessments
Certifications are demonstrations of competency to a third party. Typically, they include a written exam and portfolio or work product. A certificate is for attendance and certification is for demonstrated performance. With certification, an evaluation at the individual level, it is a way to assess individuals for new opportunities.
On an organizational or group level, we can measure the success of informal learning by determining the use of individual resources. This is done through analytics and compiling data from individual learning efforts.
Blogs are great learning tools.
Carliner noted that it is possible to assess satisfaction with resources. It can be done with surveys that focus on individual resources as well as focus on informal learning processes. Focus groups can also assess resources.
The impact of individual resources can be assessed with rater systems, specialized reports, and long-term studies. In terms of long-term studies, new employees followed and be evaluated how they are learning over time.
Other ways to evaluate informal learning include embedding questions in an employee survey and establishing “panels”.
Carliner asked an important question Level 1 evaluations. Why do them? If you are a steady state, make the survey optional to only gather warnings about a problem.
Training is about the long term not next quarter.
Right now, I am working on supporting content for the third Webinar of a six part series. This installment is on creating content to support informal learning.
The upcoming Webinar will focus on four tools that I believe support informal learning in different ways. These tools are blogs, wikis, Flickr, and YouTube.
Presently, I have finished working on three of them, and I am starting to work on the fourth.
Here is are the pages completed:
If you have time, please take a look and let me know what you would like to see. The idea of these pages is to have additional support content for viewers of the Webinar.
Throughout the summer, I will be giving Webinars to our Extension educators on strategies for improving their informal learning. This month, I am presenting on information collection strategies like RSS feeds and Twitter feeds. One of the ideas presented is on how to create a search query in the university’s digital journals, and save it as a RSS feed in Google Reader.
Rather than repeatedly search the university’s digital journals using the same search queries, you can set a search query and wait for updates to appear in your RSS aggregator. Here is a short video showing that process:
The second afternoon session I attended The Yin and Yang of Formal and Informal Learning by Allison Rossett and Frank Nguygen. They started their session by explaining that formal training was the primary training method used by the enterprise; however, personal preference is informal learning. Rossetti cited Bozarth when reporting that 83% of companies saw value in informal learning, but only 36% employed it.
In the enterprise, courses are not dead and very much alive ,but there is an interest in employing informal learning strategies. Individual informal learning makes great sense, but for the enterprise there are more concerns. One of the challenges is that informal learning lacks certification, thus does not meet the promise of the enterprise. Enterprise or companies have a promise of service they must meet.
Here are some important questions to ask.
Is success defined?
Who chooses ends and means?
Must we prove that we can do what we say we do?
Are we in a position to go towards more choice and freedom? Is the culture forgiving?
Rossetti and Nguygen created a study to examine the disparity between what is wanted (informal learning) and what actually happens (formal learning). Their study resulted in a tool of 15 questions and two table of strategies. This tool is called the YinYang Tool, and it can be found at Http://yinyang.frankn.net. This tool helps to determine if a program or company is a good fit for informal learning strategies.
The Coast Guard used the YinYang tool to see if informal learning was good for them and new boat program. They stuck with a formal approach because of dangers of getting it wrong.
When you are talking about informal learning, you are talking about choice. The tool helps you determine the choices you have for delivering training. It will help you identify opportunities how to present training. Use the tool as a conversation starter to discuss training options and possibilities. Remember, you can evaluate a whole program or just sections of a program with the tool.
The goal is not to be informal or formal. The goal is to be better at what we do.