Episode Transcript

Ø  Dr. Streveler:  Welcome to the Research Briefs Podcast.  I’m your host, Ruth Streveler, Professor of Engineering Education in the College of Engineering at Purdue University. 

In Research Briefs, we’ll speak with engineering education researchers about what their lives are like, what they are finding out, and how their research is being used. 

My guest today is Dr. Kerrie Douglas, Assistant Professor of Engineering Education at Purdue University and head of the SEED lab [Science and Ethics of Educational Data]. Her research focuses on supporting high-quality assessment practices in engineering education. 

Today I’ve asked her to speak about her current development of a holistic framework for online course evaluation, something that’s very timely.  And that particular work has led her to focus more broadly on how students are being supported during the pandemic and beyond.  

So, Kerrie, welcome to Research Briefs.

v Dr. Kerrie Douglas:  Thank you for having me.  This is exciting. 

Ø  I’ve admired your work and so I’m glad to share it with other people.

So, I’ve talked a little bit about this holistic framework for online course evaluation.  Can you tell our listeners a bit about that?  

v Sure.  So, we developed a framework called the Contextualized Evaluation Framework.  And it really began with the need to take a holistic view of online courses rather than relying on one or two sources of data to really get a more holistic, or a whole-er picture of the online courses and how to evaluate them.

So, the work really started in 2015 when I was asked to lead an evaluation of some Massive Open Online Courses, [MOOCs].  And when I was looking into how to go about evaluating them, I found that there really wasn’t a whole lot of consistency in what was being evaluated and the type of metrics, the numbers, or what kind of outcomes were being reported across different MOOC projects or entities.  And so, I ended up leading up a proposal that was supported by NSF to study how MOOCs should be evaluated and that’s what led to the Contextualized Evaluation Framework.  

And from my perspective and others, evaluations really when we use multiple sources of meaningful evidence to arrive at a judgment about the goodness of the thing that we’re evaluating.  So, at the end of the day, evaluations should be about gathering information of what we think would make the product, or tool, or thing being evaluated actually good; what we value about offering it.

So, when I was looking at that and doing that work, I found that approaches that were being used were focused either primarily on the design of the course, or they were focused on just what kind of reactions the students had to the course such as the end of course surveys.  But there just seemed to be more missing pieces.  

So, it was really developed from the need to have that fuller picture that was going to provide the most meaningful information for the larger group of stakeholders.  So, that really began with doing a lot of interviews with different stakeholders, people that the platforms, or administrators from institutions that were financing or putting money behind offering the MOOCs, the instructors themselves that were teaching the courses, the learners who take them.  So, a lot of research from those different stakeholders to determine what kind of metrics really were worth capturing, what would make a MOOC good.

Ø  So, you had mentioned when you spoke with me that you found something initially that you said was the previous gold standard of online education evaluation.  Could you say a little bit about that? 

v    Sure.  So, I think my understanding the most popular evaluation tool for online courses is the Quality Matters Framework.  And so, courses will get designed and then they go through this review process where then they are either given feedback on how to meet the standards to get certified or get the stamp of it’s been approved, or they go ahead and get the approval.  So, feedback on how to make it.  

But the evaluational criteria, or what they’re evaluating are based off of a lot of literature on best practices in online education.  And how they are rolled out though is primarily focused on the design aspects of the course.  So, you know, the things like having solid alignment between the learning objectives, the courses activities, the assessments.  They also are looking for things like accessibility.   Very much again focused on the design of those materials for the course.

So then, another model that is often used in education technology is the Kirkpatrick Model.  And that’s the model I was kind of referring to before that it really evaluates the outputs of the education.  So, the learners’ reactions, what kind of behavioral changes they made, has the organization made changes as a result?  That model doesn’t include the pedagogy or the design of the materials; it’s more just the straight outputs.  

I really wanted to have a framework that was holistic, that we were looking at it from a variety of perspectives of what leads to a good learning opportunity.

Ø  So, I know quite well that you described this framework as still obviously being under development, which really all frameworks always are.  But that you have published an article, a conference proceeding, from the 2020 American Society of Engineering Education.  Do you want to say a little bit about what the framework is?  I know people will be curious.  And then also what your next kinds of developments are.  

v Sure.  So, currently the framework has five levels of evaluation.  And, as I mentioned and as you said, this is a work in progress because we made it for MOOCs and then we adapted it for professional learners.  So, then that was another sort of change where we were getting a little more specific with who the framework was applying to.  And now that we’re working with more online courses that are offered for university credit, that sort of changed the framework somewhat as well.  

So, the current framework is:
 

1.    Learner satisfaction – so in terms of what did they like, what didn’t they like?
 

2.    Course design – is it aligned in terms of the content, the assessment, the pedagogies used?  Are they following good practices?
 

3.    Actual delivery of the course – so, this is something I think is really important.  To what extent is there a genuine community of inquiry?  Do the learners get feedback?  Is there someone who from the teaching team that is genuinely interacting with learners or are they just left on their own?  Is there someone who knows something interacting with them?
 

4.    Learner engagement – this is in a few different ways.  One is, what materials are they actually engaging with?  And then, what groups of learners are engaging with what materials?  Looking at those we’re able to identify areas for continuous improvement and if we can see there’s a quiz that very few people watch this module before they take the quiz or these sorts of things where we can point out what’s actually happening behaviorally.
 

5.    Learner outcomes – so, to what extent do the learners achieve the learning goals of the course?  So, the course sets out these things like what percentage of learners actually achieve that?
 

So, that’s the current state of where the framework is at. 

Ø  And I know you said you are working now with Julie Martin, who was also a former podcast guest at the Ohio State University to look at some things during the pandemic.  Isn’t that correct?  

v Yeah, thanks for asking about that.

So, because of the work that we’ve been doing on the evaluation framework, I’ve been looking at a lot of different online courses, and the different ways of offering them.  And the thing I was noticing is how difficult having that genuine community of inquiry really is.  And I know you’ve talked about that a little bit on your previous podcasts.

Ø  Right.  Yeah.  Ruth Wertz was speaking about community of inquiry and also the last episode of Research Briefs discussed a new podcast called, “Reflective Teaching in a Digital Age,” and they had Randy Garrison who’s kind of the creator of the Community of Inquiry model as a guest.  

So, if some of the listeners are interested in the Community of Inquiry there’s definitely some other places to peak around about that.  

v Yeah, that’s awesome.  That’s really cool that they had him on, I definitely want to listen to that.  

So, from the research we’ve been doing, just seeing how difficult it really is to establish that.  And it’s one thing when we’re talking about Masters Level students, you know, people further along and when we were notified that we would not be returning back to campus in the spring [2020], that courses were going to be remote, my biggest concern became this aspect of online education.  You know, the instructors were moving to what we would call an emergency remote teaching where this is not just online learning.  I mean this is like they did not take months to prepare to teach these courses online.

Ø  We had a week I believe, right?  We had Spring Break.  

v Yes.  We had Spring Break to get it transitioned.  And so, you know, what was going to happen in terms of supporting students with each other?  How readily available were they going to be able to have contact with their instructors?  And really that aspect was really sort of bothering me.  Because I’m thinking, you know, instructors are going to be held accountable for teaching this content.  There will be end of course surveys taken.  But who is capturing the kind of support students have during this time and what is this going to mean for them moving forward?  

And so, as I was thinking about this during that Spring Break [2020] I have just I’d say a lot of nervous energy and was feeling in so many ways just sort of helpless, like I think a lot of us.  There is so much uncertainty even in my own life, and so I was thinking in that time I wanted to do something that felt meaningful in response to what was happening because of the pandemic.  And so, I sort of just really focused a lot of my energy into generating resources to help faculty support students.  And thinking about how to do research to capture what was happening.

So, Julie Martin, as you mentioned, is a good friend of mine and we had not collaborated before on a research project that we were just discussing this because her research has been a lot in engineering students’ social capital.  And so, as I was talking to her about being concerned about how students are going to be supported, what kind of…I mean most faculty are not trained before they teach anything.  I mean, they’re just hired to do a job that we don’t get trained for.  But fewer have been trained to teach anything online.  And then even if we did go through one of those professional development, or had an instructional designer working with you, the need for students to be interacting with each other or having that instructor…I mean it just seemed like, so few people had been trained and skilled in this, through no fault of their own.  It’s just the reality.  

So, as we were discussing this and sort of thinking about it then I decided that I wanted to lead a proposal called “RAPID” which I can’t recall the exact acronym.

Ø  I think I have looked it up.  Let’s see, did I put it in my notes?  It’s something like, “Rapid Response to Research,” or something like that.  But yeah, it’s a grant that you can get to capture something.  It’s a relatively small amount, but the turnaround is really quick so that you could have those opportunities.  “Rapid Response Research,” I did capture that.

v Right.  So, I think it ends up about $200,000 for one year.  So, it’s not too bad.  But $200,000 I know in terms of the bigger grants and stuff is relatively small.  

So, we wrote up the idea to do case studies because we couldn’t control anything going on.  And case studies are perfect for studying what happened.  You know, you’re sort of trying to capture and go deep so relationships happen in situations that the researcher has little or no control.  So, we identified some cases where the courses were being moved online but they were also team based because there’s very little research on doing team based collaborative courses in engineering online.  

And so, we were really curious what were faculty going to do, and how were they going to facilitate this teamwork, how were they going to facilitate students interacting with each other?  And what kind of social capital would students have in their courses and then their engineering time.  

I think what we weren’t expecting, which now I think everybody understands, was how much they perceived different support during that pre-pandemic, and during the pandemic.  

Ø  Could you say a little bit more about that?  

v Yeah.  So, most of us have like the solid people in our lives that are with us forever, or like longstanding relationships.  But they aren’t necessarily the ones that are helping us professionally always.  I mean there are people who we have mentors, but then we also have, you know, like my parents are supportive, or my spouse, or my best friend from high school, and those sorts of things.  

But during the pandemic those relationships were really strengthening.  But the access to the, what from the theoretical perspective we would call the “weaker ties” were essentially being cut.  And those are really important because those are those interactions on campus where maybe it’s the person in the grad office that you could just pop in on.  Or the advisor in first-year engineering that if you were on campus you would just see if they were available, or make a quick appointment, and how to do that online.  Involvement in professional organizations, interactions with TAs, I think that really reduced.  So, there were all these small ways that the students perceived all these intangible things that we were able to capture as being significant or less.  

And then what we’re seeing now is the students that really started during the pandemic and how little connections they’re forming to campus despite efforts.  Like supposedly there’s in-person instruction happening.  And, you know, all of this.  And to see that they’re still not forming natural relationships.  But with the social distancing and these other precautions that it really is challenging to develop informal interactions.  And those are so important. 

Ø  Particularly with those weaker ties which you said could be the person that might be able to answer a homework question or give you some specific kind of thing that the people in your immediate sphere just don’t have the expertise to do.  Yeah?

v Exactly, exactly.  You know, one of the things we talked about was in class, when we’re all physically there, I can look at the person sitting beside me, I don’t even have to know their name, but I can ask them, “Did you catch what Streveler said?”  Or “When did she say that quiz was?”  You can ask that without having their contact information.  

And the first-year students, in the case study that we looked at, talked about walking to class with students.  So, they would walk there and walk back, pre-pandemic.  But pandemic hit and they didn’t even know their last names.  So, they no longer had…it was like the rug got ripped out from under them.

Ø  No way to find them.  

v No way to find them.  They’re in this course of 300 students or whatever, and I don’t know their first name is Jenny.  That was really different than the case study with the senior students because they had pretty well-developed networks with roommates that were in the same major, their teams were actually friends.  So, I think our attention was really looking at the earlier students, the beginning students, because they hadn’t had that time on campus to develop all the connections that we know are so important.  

Ø  So, you were really able to see through your framework the importance of measuring how the community might be formed and supported.  And seeing then that that’s always important but perhaps maybe more so in an online environment where you don’t have those informal, natural ways to create that support system.

v Yes, absolutely.  I think in the last year, I definitely have developed the opinion that one of my most important jobs as an instructor is to facilitate the relationships my students have with each other.  And it’s not just about the relationship they have with me, and I think I need to be available for sure, but that they are resources to each other.  And that will go beyond one course with me.  

Especially, in this time, we’re all having the shared experience in some way.  You know, we’re all living in this time of pandemic, living in this time where advocating that Black Lives Matter seems to be a controversy, you know, we’re going through these things together as a society.  And I think sharing these experiences we can be resources for each other.  We can support each other during this time.  It’s not just in one particular course, or one thing.  But I tell my students, you could be developing your best friend for life right now because you’re sharing so much.  You have so much in common right now.  You’re going through your first year at the university and things are messed up but at least you have each other.  

So, I think not expecting them to form those relationships naturally because with all of the safety precautions in place it’s very difficult.  It’s just not happening naturally I don’t think.  And then, in an online course it’s super-challenging especially with sort of the most common version going is asynchronous, giving students in different time zones flexibility.  But then that can create a really lonely experience without really being purposeful about how you establish that.  

And that students feel that if they don’t show up someone’s going to notice, someone cares, someone’s going to realize that they’re not taking part.  And being able to hold each other accountable.  And, again, going through it together I think is super-important, especially for 18-year-olds, 19-year-olds.  It’s very different than a 22-year-old doing a master’s degree, I think.

Ø  Although, we all like to have somebody notice if we’re there or not.  I still do want to be noticed I guess; want to feel like my presence means something.  

v Right, right.  

Ø  So, I usually end the podcast by asking what advice or takeaways you have for the listeners.  And I have a request with this, and again you could also have other takeaways, but I always have a pre-podcast meeting with the guests, and in our meeting, you told me about an example that you were able to study in Electrical and Computer Engineering (ECE) that was just really uplifting.  So, I wondered if you would indulge me in having that at least be one of your takeaways of that experience.

v Sure.  Awesome.  We’re actually currently writing up the results of this case study; we don’t have all of them yet.  But I’ve been very excited about it also because we’re studying an introductory level course on circuits which I think this course, regardless of the institution you’re at, sort of has a reputation as being just like horrible.  Like Intro to Circuits has some challenging concepts.

Ø  And often everybody has to take it too even if you’re not going to be an electrical engineer, and so I’ve had like chemical engineering friends said, “Oh, my God, I had to take Circuits, and I’m just not a circuits person,” and so that’s in the mix too.  

v Right, right.  Yeah, there’s a lot.  They don’t see themselves using the information necessarily.  I mean, they don’t get it.  

So, the course is typically a couple hundred students in a section.  They’ll have recitation.  There’s also, for the ECE students, there’s also a lab course that they take sort of in companion.  But most students, you know, in all the different disciplines have to take it.  

So, traditionally in-person, they would go to the lecture and they would go to the recitation.  Well over the course of some time, we’ve been doing some collaboration with an instructor who’s very much interested in introducing more active learning into the classroom and wanting to develop videos in support of that.  And so was thinking that if he could flip the classroom that the time they had in-person could be more learner-centered.  Which is a big undertaking in something like that because it is a very content-heavy course.  And a lot of students do not do well in the course.  I mean it’s a tough one.  And you have a lot of instructors that teach it, so getting coordination across them can be really challenging.  

So, even though he had made some changes in person, it wasn’t like one semester he just flipped it all.  I think he was taking it very wisely which was what could get done this year?  What’s my like small bite stuff this year?

So, he had the videos going in person.  During the recitations they started doing some group work; not a ton but they were starting to just have conversations in it.  Well then, he decided during the summer of 2020 that that was the time to have the whole course online, and that it was going to be asynchronous, so he had the videos going.  So, instead of having recitation, he created learning groups where they were in the same team throughout the course of the semester.  And they had homework assignments where there were the traditional calculations and the equations that they had to learn and do, but he also built in discussion questions where they were having to make more meaning out of it and sort of think a little bit more conceptually about what was going on, take it broader about the bigger context of the calculations.

So, we interviewed students, we did the surveys.  And the thing that’s been fascinating is how supported the students felt in that course, and just how positive that the … we’ve been in this course for maybe three years now [looking at] the changes along the way.  And this was by far, like the student-reaction piece was way more positive than the previous semesters.  And it may be that they were just so happy…because they knew it was a pandemic and their expectations were different, or they just went off for the spring and that was like crazy, I don’t know.  But they expressed so much appreciation for the instructor because of how he designed that course and was very purposeful about creating small groups of students working together.  

And I don’t even know if they realized that’s what made the difference.   Because they talked a lot of about just how positive they were about the course, and the enjoyable experience.  Not that it wasn’t hard; it was challenging.  Some of the students we talked with didn’t get As, but they thought it was a really positive learning experience by and large.

Ø  So, what I find so uplifting about that is that I know I worry about the online experience and everybody have a screen in front of them, and all of my students now are just these tiny little pictures on the gallery view. And how much harder it is to create a community.  But you have an example of, again, being very purposeful, and having them do meaningful things, and really thinking about it being quite intentional about it.  

But the end result was that the students felt more supported and more connected than they did in the face-to-face setting.  And so, I think that’s a real inspiration that if one really is serious about the community-building aspect that you can be quite successful in it. 

v I think that is so true, Ruth, I do.  You know, people will often, for years now, people will say, “Oh, we should do a comparison of this course online and this course in-person.”  And like somehow this experiment is going to just answer the question is one better than the other?  And to me it’s just such a nonsense question because it’s how its done.  It’s really how we do it and how we design and implement the course whether it’s online or in-person.  And I know I’m like you, my preference, by far, is to be in-person.  And I much prefer walking around and looking over students’ shoulders.  But I can create breakout rooms and pop in and out of them and this generation of students, they’re a lot more comfortable with technology, distance, and they do so much more electronically.  It’s how they’ve grown up.  

Ø  The digital natives, right

v There you go.  But it’s all about how it’s done.  

I know we’re wrapping up here, but that’s another take home to me.  So, I have this holistic evaluation framework for online courses, and I really think that all courses should be evaluated holistically.  

Ø  Right, right.  Now the best resource right now for people is to find your ASEE article and we could put a link to that on the Purdue website.  I think it is called, “A framework for evaluation of large online graduate level courses for professional learners” by yourself and your student, Hillary Merzdorf from the 2020 American Society of Engineering Education Conference.

But I know you’re also developing resources for instructors as well.  So, do you want people to contact you if they’re interested?

v I’d say “stay tuned” ‘cause I don’t have anything to handoff right now.  But we are, as you said, creating…so we have a project funded by the Koch Foundation that is essentially applying some machine learning to give us information in the framework.  And then we’re trying to visualize that information for instructors to evaluate their courses.  So, I’ll be posting things as that comes about.

Ø  And the SEED lab, I think you have a good student who’s your social network person, right?  Social media person?

v Yes.

Ø  Well, is there anything else you’d like to pass on to the listeners before we end?

v I think we got a lot of it.  If anyone has questions, I’m always happy to interact.  But I really appreciate you having me on.  This was fun.  I always enjoy talking with you. 

Ø  Well, I enjoy talking to you too.  And I think with that I will say, Arrivederci!  Adios. 

v Ciao.  

Ø  Thanks again, Kerrie. 

v Take care, I’ll see you later. 

Research Briefs is produced by the School of Engineering Education at Purdue.  

Thank you to Patrick Vogt for composing our theme music.  The transcript of this podcast can be found by Googling “Purdue Engineering Education Podcast.”  And please check out my blog, RuthStreveler.Wordpress.com.