Learning Outcomes Assessment – A Step By Step

Doug Miller

A post by Guest Blogger, Doug Miller, faculty member and New Media Manager at DePaul University.

Douglas Lee Miller – Chicago, IL | about.me

On Twitter: @douglasLmiller

Learning Outcomes Assessment: Step by StepPresenters: Gail Rooney and Julia Panke Makela, University of Illinois at Urbana-Champaign

Nothing says real world application like a “step by step” and this one was both useful and inspiring. What follows are brief notes on a much more complete presentation about the deployment of these learning outcomes assessments in a particular context. The hope is that these notes and their brief outline of the constituent parts my inspire you to learn more.

From the presentation:

First, as a framework, it is important to define assessment as a process that is continuous. It tells a story and shares the dual function of providing continuous service and celebrating achievement.

What can we assess? We can assess needs, participation, satisfaction, and learning. There are many different desired outcomes, but learning is at the heart of what we are about in career services.

To be a part of the core function, career services needs to think of themselves as learning partners. -Rooney

Learning outcomes focus on client experiences.

We measure what clients can know, do, demonstrate or feel. How have they changed?

Rooney describes the ALOI cycle

Rooney describes the ALOI cycle

Focus in career services is too often on process and not on the change we want to see happen in our client experience. The purpose of the learning outcomes assessment is to be able to measure the changes we see in the client experience and behavior as a result of interactions with particular learning elements.

The steps below shed light on the path to deploying some learning outcomes assessments and on some of the more important questions an institution my ask while deploying them. They also illustrate the cyclical nature of the process while illustrating the need to assess and revise.

Step 1 – Defining Context

Guiding Questions: Who is the audience? What are the circumstances?

Step 2 – Brainstorming Outcomes

Guiding Questions: What do you want your audience to be able to do, demonstrate, value, or feel?

Step 3a – Writing Outcome Statements

Formula = “Intended learners who take will be able to .”

Step 3b – Writing Outcomes Statements

Useful construct = Blooms Taxonomy

Step 4 – Connect Theory to Practice

Helpful Suggestion: Look for a rubric from the field and place it in your context.

Step 5 – Prioritize Learning Outcomes

Guiding Question: If you had to pick only 3 which would be most helpful? Most important to stakeholders?

Step 6 – Evaluate the Outcomes

Helpful Suggestion: The purpose is to make a judgement. Too often we just use surveys – examine some other tools that may add dimension to the common survey. Look for demonstrable behavior changes.

Step 7 – Reflect on Results and Process

Guiding Questions: What happened that you expected? What took you by surprise?

Step 8 – Use the Assessments

Guiding Questions: What are you going to keep, modify, discontinue, explore?.

Bloggers note: The above steps were presented in a specific context – resume – that made it fairly easy to translate client behavior changes into learning outcomes. This author can imagine more than a few contexts whereby charting the behavior changes would be difficult if not problematic, especially in the areas of so called “soft skill” development. Despite this, there were some healthy and creative suggestions offered up by attendees that included everything from video taping to web analytics.

Finally, the most important advice from Rooney in the session is to start soon, start small, and to start with what you know you do well and build out from there. Making learning outcome assessments work is not easy and requires constant fine tuning – but the end results are by their very nature measurable.

A last reminder of Rooney's mantra as you set to the task of creating your own learning outcomes assessment tools: what did we help our clients to do, demonstrate, value, or feel that was different than from before we interacted with them?

In this case Rooney prompted at least this author's learning to demonstrate the outcomes assessment as a process – specifically to go home and revise my next syllabus before next quarter begins.

For more information see the story below:

The National Career Development Association | Learning Outcomes Assessment Step-By-Step: The Story Behind NCDA’s New Monograph


LEGO & The Career Architecture Framework

Doug Miller

A post by Guest Blogger, Doug Miller, faculty member and New Media

Douglas Lee Miller – Chicago, IL | about.me On Twitter:@douglasLmiller



Presenters: Anne Scholl-Fiedler and Jim Salvucci, Stevenson University

What is the career architecture model?

It mentors students through a process of learning

1)who they are at their best (personal direction)

2)what they are learning (discipline expertise) and

3)how they will apply their skills (professional know-how.)

Tag line: “dream about your future-design your career.”

The first year seminar is housed in academic units and led by faculty but is heavily facilitated by career services. It culminates in a massive competition amongst teams of freshman students who undertake a project to represent – in LEGO bricks – what career architecture means to them on their own terms.

Functionally, the program begins with the students being given Holland assessments of interest and skills. This guides the faculty in creating teams whose individuals will perform functions related to the idea generation, design, and ultimately the build process of their LEGO creations. Working in three dimensions with elements familiar from childhood, the students soon find themselves involved in a meta-discussion about Career Architecture as a framework.

783 students participated in one year's Lego challenge. The whole process is engineered to reflect the form and structure of the Career Architecture Framework itself, full of learning objectives. The LEGO project itself is judged based on the successfulness of its representation; how well does the plastic physical model represent the three more ephemeral ideological elements of the career architecture framework? That's what they are judged on. Each major's representation is highly unique and specific to their field despite starting from the same point. There is a high degree of symbolism to the lego models much like with something like parade floats.

After the event, a full array of assessments are deployed in person and personalized career architecture plans are developed.

Salvucci's working theory is about the transactional nature of or common interactions with students in higher education as juxtaposed against a more transformational learning model. Both aspects are at play in the interactions and both are necessary but there may be value according to Salvucci in focusing on the transformational nature of a student's exposure to higher education characterizing the transactional as “the descent” vs. “the ascent” – purchase of the transcript vs the birth of an enlightened being.

How, you might ask, is this done from a curriculum development and approval point of view? Salvucci as the Dean works very closely in tandem with Scholl-Fiedler in Career Services.

Most English programs are content driven; fthe challenge is to create measurably attainable graduated skills – like problem solving – but they also tie them to career. They work collaboratively within the divisions to tailor the capstones toward career.

English, psychology, theatre and film-video are all modeled here. All require internships. Many have career track courses.

Psychology has three 1 credit courses on career development.

“Quad stories” was deployed to gather career stories to keep people from thinking careers are linear. Their objective with this tool? “Creating a culture of personal narrative.” Using personal narrative to determine skills and interests.

They have Industry liaisons in their office to help meet those goals

In the end, they see themselves as brokers of knowledge not just spinners of information.

Bloggers thoughts: As an avid fan of LEGO, learning outcomes, ideological frameworks, and personal narrative, I took a host of ideas and inspiration from this fascinating session. The only caveat offered about working so closely with faculty for this amazing strategic construct and series of events echoes common refrains I hear in my own position. “It's great that you have someone at your institution like you – but what if that key component is missing at our school?” Anne Scholl-Fiedler and Jim Salvucci both represent Stevenson University well and it is clear this kind of innovation and freedom could not exist were either of them the type to not be open to collaboration. Salvucci sees the need for collaboration as obvious. My caveat to other schools? Some results may vary.


Notes on Motivating Students & Grads to Get on the Ball

Doug MillerA post by Guest Blogger, Doug Miller, faculty member and New Media Manager, DePaul University

Douglas Lee Miller – Chicago, IL | about.me On Twitter: @douglasLmiller

Motivating GenY is different from GenX – it is less about them and more about you.

GenY unemployment at 9% but what often gets overlooked is an even worse situation about UNDERemployment. Over half according to the Atlantic.

Many are delaying major life decisions as a result.

Retiring workers need to be replaced so there are some opportunities. There is a “grey ceiling” that exists for GenX since many boomers are delaying retirement due to recent economic downturns. When they do leave, GenX is not large enough to fill those spots so GenY will benefit – eventually. Luckily it seems employers are hungry for the fresh perspectives and creativity which is good for GenY but might be bad for GenX since there is baggage that comes along with bringing them on.

GenY students can benefit from this but often drop the ball when it comes to online presence and offline soft skills in interviewing and communication of transferable skills.

As a result the Career Advisory Board has been developed to meet the challenges facing career advisors.

47% of directors view lack of motivation as a major barrier for recent grads.

35% rank that lack of motivation as the number one barrier yet only 24% felt students lacked the skills needed to get careers. Why is this dissonance here?

Part of that is in the perceived mindset of GenY and in their girth. 4 million more than boomers (largest in US history.) It is believed that most GenYers believe they are better than the competition – obviously this can't be the case but their enculturation was such that they were all highly valued.

More insight can be found in an exploration of motivation (intrinsic vs. extrinsic.) A great resource about this is Dan Pink's book Drive.

Appeal to their personal passion and interest. Give over control to increase intrinsic motivation. Share the big picture.

Amazon.com: Drive: The Surprising Truth About What Motivates Us (9781594484803): Daniel H. Pink: Books

Sometimes extrinsic is appropriate sometimes intrinsic. It depends on the person and the task. Luckily seeing people as individuals increases affinity among GenY in general so doing so increases intrinsic motivation.

Make the most of the first interaction – it will be key. Judgements will be made quickly.

They will remember more about how you made them feel than what you said. – Alexandra Levit

Start early, start often. Share expectations early – communicate what GenY is expected to do. Utilize peer influence – hold support group style events that pair older students with younger students (mentoring circles) to connect and share issues. Facilitate where you can. Motivate both students and peer mentors.

Model successful GenYers as success story marketing via in-person or virtual billboards for praise.

Bloggers thoughts: My concern with testimonials is that there may be unmeasurable negative consequences – that many students may have decreased motivation with the lifting up of outlier examples and great successes, due to lowered self esteem and a feeling of being left out or unfairly excluded. If GenY all feel like they are the best, should we be prepared to showcase all students to avoid making some feel excluded? Most in the room don't seem to share my concern.

Get top executives to call out those good examples.

They played a voicemail of an emotionally charged success thank you as a way to remind career service folks to feel empowered.

Present individual challenges to boost intrinsic motivation. Making challenges intriguing and game like can be highly effective but requires the creativity and involvement of the entire community. Motivation decreases quickly with failed expectations so make sure to manage expectations. Help them understand the mechanics of the process to keep them from being demotivated.

Faculty and parents can be key allies, just as can peers.

Final Blogger Thoughts: There was some discussion of the need to integrate digital tools. While I agree, I feel the need to warn that there is far more to leveraging social media than just Facebook – and in fact it may soon be the last place we want to be. Likewise, I worry that using digital tools could create negative impressions because we simply cannot match the user experience offered by modern digital tools.

Career Advisory Board | Devry University

Mobile Engagement – The Chocolate Peanut Butter Cup of #NACE13 Sessions

Doug MillerA post by Guest Blogger, Doug Miller, faculty member and New Media Manager, DePaul University

Douglas Lee Miller – Chicago, IL | about.me On Twitter: @douglasLmiller

Once upon a time, hip youngsters strutting down the street and oblivious to anything but their mobile device looked a little bit different…

Yet, as much as the music, devices, and marketing seem to have changed there are many things that stay the same. The taste of chocolate and peanut butter is still a favorite (and why not?) and youth still seems synonymous with the early adoption of technological trends.

Despite these consistencies, we who work in Higher Education consider it a given that there is a need to stay on top of the world of our audience (and why not?) so we can better communicate with and prepare them. In recent years this has been especially so when it comes to the myriad ways technology has taken center stage in nearly every aspect of our lives.

First it was the Internet – a digital revolution that had us all predicting the death of the printed page. Suddenly every conference presentation and professional learning module is peppered with talk of moving services online, building websites, sending emails. Then we perseverated over all things Social Media – if there were no comments, it wasn’t Web 2.0 enough and everybody was learning about the predicted death of one-way communication and the end of emails. Today, the object of our obsession is Mobile – if there isn’t an app for it, you’re doing it wrong, and you might as well Snapchat that resume tutorial and set it to expire in seven seconds.

The truth of the matter is that the printed page is still alive and well, most of us would like nothing more than to see the death of email (which is nowhere in sight) and doing “mobile” right may not necessarily require building an app at all.

But we still must perform our due diligence to keep ourselves educated about the world of our audience. What then must we cover? How then must we learn? These are exactly the types of things we aim to cover in our learning session about Mobile at NACE13.

There is so much data covering the rise in adoption of mobile devices. We will cover some of that in specific terms from a variety of sources which are by no means exhaustive but hopefully a great start. Then we will talk about some options on the table for how to approach thinking about folding mobile contexts into your strategies in general. Then we will discuss some specific tactical mobile deployments and cover the entry points of how-to and where to find more information.

One of the common themes of the session will be to discuss the ways mobile, social, and Internet in general all stem from that same great combination of flavors that has fueled the fusion of technology, community, and communication. In some ways and in some situations, the best answer for mobile may look a lot like the best answer for social.

And, who knows – maybe we’ll bring some Chocolate Peanut Butter Cups along just to drive the point home and leave everybody happy. Won’t you join us?

Is there something about mobile you are dying to know that you think we may not cover? Let us know in advance via this Google doc form:

Mobile Issues You’d Like to See Covered via our Session at #NACE13 #NACE13mobile http://bit.ly/15KJZnf

Also – there are a number of sessions that will likely talk about Mobile this year, yes? Let’s aggregate everything under the tag #NACE13mobile, shall we?

See you in Orlando!

Mobile Career Services: The Next Frontier in Student Engagement

Track: Branding & Marketing

Level: Intermediate

Program Format: Peer-to-Peer

Audience: College

Want to reach students? The mobile web is where you need to be: Nearly 60 percent of college students use smartphones, and an increasing percentage are using them to access web resources from mobile devices. Before you launch your own mobile initiative, learn from this panel of experts about trends, adoption, and user behavior, and find out how key technologies—geolocation, APIs, and social media—impact the mobile experience.

Presenters: Janet Sun, ConnectEDU; Doug Miller, DePaul University; and Harold Bell, Spelman College

NACE – Workshops: NACE 2013 Conference & Expo