Students don't know what they don't know: EeCheng Ong on concept checks, scavenger hunts, and engaging students

On 30 October EeCheng Ong led a Stone Centre and CTaLE workshop on “illusions of knowing”: the gap between what students think they understand and what they actually do.

‍

We caught up with EeCheng, associate professor on the educator track in economics at the National University of Singapore, to talk about concept checks and experiential learning.

‍

Could you start by telling us a little about yourself and your approach to teaching?

‍

A lot of what I do is grounded in constructivism — Piaget’s theory that we learn by “constructing” knowledge. I’m also interested in experiential learning — learning that isn’t just pen and paper, which students tend to perceive as boring — and, more recently, the pedagogy of play.

‍

I’ll talk about two projects, both of which are collaborations with colleagues at NUS. The first is what I presented on at the workshop, which is work with my colleague Yang Zhang. We both teach microeconomics: I teach intro and intermediate micro at the undergraduate level, and she teaches intermediate micro and advanced micro, a master’s course.

‍

Your workshop was about concept checks. What are concept checks?

‍

As educators, we’re told it’s a good idea to pause every 15 minutes or so and ask students if they have any questions. The problem is that students often don’t know what they don’t know, so they don’t know what questions to ask. Or they may think they understand, but that understanding could be flawed.

‍

We thought that instead of asking students whether they have any questions, we should think of questions to ask students to gauge their understanding. This serves two purposes. It tells the instructor what students know and don’t know, and it tells students what they know and don’t know — that’s developing students’ metacognition.

‍

We call these questions concept checks, because they focus on conceptual understanding. Economics is abstract, and because it involves a lot of maths and graphs, students tend to think that if they can solve a problem mathematically, they know what’s going on. But they could be just plugging and chugging: solving problems mechanically without thinking about what’s happening.

‍

What does a concept check look like in practice?

‍

They’re in-class exercises that take about 10 minutes. Our classes are blended, so before class students should have watched videos and done some exercises, which we go through in class.

‍

The concept check then focuses on a concept that we know from experience students struggle with, and it’s framed differently from how they’ve seen the material, so that it’s not just memorisation and regurgitation. It’s trying to probe their understanding.

‍

Students have a few minutes to think about the concept check on their own, and then they submit their answers. In a large lecture they do that via Poll Everywhere. We can then display the responses. If it’s a multiple-choice question, we can see straight away that, say, 20% of students chose A, 20% chose B, 50% chose C, and 10% chose D. We show that distribution to students, so they have a sense of where they and their classmates stand. And the last step, of course, is that the instructor explains the solution.

‍

It works because it’s anonymous. Each student knows whether they got the right answer, but no one around them does. And it’s based on participation, so it’s not punitive in any way. You participate, you get it wrong, and that’s fine. Nobody knows you got it wrong, but now you know: “I thought I understood this, but I don’t, so I’m going to pay attention.”

‍

There are different ways of doing it. One is to make it fun with a Kahoot!, which is gamified: there’s a timer, a competition, and music. Students really enjoy that.

‍

How have students responded?

‍

We’ve surveyed students about their perceptions, and the response has been overwhelmingly positive. Around 90% strongly agree or agree that the concept checks exposed gaps in their understanding. So at least in terms of developing students’ metacognition, we’re achieving our goal.

‍

Do concept checks also work in smaller lectures?

‍

They also work really well in small classes. There, students can split into small groups to discuss their answers and put them up on the whiteboard, if your classroom has whiteboards on the walls. I’ve definitely seen that when students are working in groups and their answers are visible to everyone in the class, there’s very little embarrassment when they get it wrong.

‍

We can think of concept checks simply as a way of teaching, which some of us are probably doing. Maybe we haven’t described the method as a concept check, but seasoned educators will likely have thought about how to evaluate students’ understanding in class rather than waiting until the final exam.

‍

Has AI made it harder to evaluate students’ understanding?

‍

I don’t see a huge difference pre- and post-AI. One thing I’ve stopped doing is getting students to submit problem sets, because that was where I saw that, for the more difficult questions, students were copying and pasting AI answers without even trying. They were copying selected pieces of the answer, so it wasn’t a complete answer, but they didn’t seem to realise that.

‍

Beyond that, because in-person exams are a significant portion of the grade, students still have the incentive to put in effort. They know that at the end of the day, it’s just them and their pen and paper, and no AI.

‍

So the gap between what students think they understand and what they actually understand isn’t a modern phenomenon, and probably isn’t specific to economics either.

‍

Yes, and it’s something I’ve realised over the years: I’ve been consistently overestimating how much students understand.

‍

Let’s talk about your second project: an economics scavenger hunt. How did that come about?

‍

We’ve done physical field trips with our students, but those are usually a maximum of 15 people, you have to schedule a time, and very often you have to pay the tour guide. In our intro course, showing students how relevant economics is in the world, and not just in the textbook, would have value. But our intro courses are large! We have over 600 students in our first-semester intro economics course.

‍

How do we scale this? We saw that there are now companies — there are probably some in London — that run gamified experiences where you walk around the streets following clues and solving puzzles. And we thought: let’s do that, but with economics involved.

‍

‍

How does it work?

‍

This is a collaboration with my colleagues, Kok Hoe Chan and Timothy Wong. We use Telegram, but WhatsApp works as well. Students form groups of three or four, and each group is in a chat with a chatbot. It’s basically a rules-based decision tree built on messages that we’ve written. It starts by saying something like, “Hi, my name is Odder Otter, and I’ll be your guide for the scavenger hunt … these are the rules of the game”. Then it gives them the first puzzle location, but in a cryptic way — not just “turn left on this street, and when you see the traffic lights, turn right”.

‍

For example, one of our clues is: “Follow the Celestial Guardian of the East from head to tail.” That’s a dragon. There’s a huge dragon mural on a wall: students start at the head of the dragon, and they walk along the wall until they get to the tail. The cryptic navigation makes it more fun.

‍

Then, when they get to the puzzle location, the bot posts the puzzle. The puzzles are based on physical features: an old building that used to be a clinic; a Hindu temple; a mural depicting a Chinese opera; an old, abandoned well; or a signboard in a park explaining that it used to be a clove and nutmeg plantation. Students try to solve the puzzle using what they see around them, as well as the economics they’ve learned in class.

‍

‍

Could you give us an example of a puzzle?

‍

One of the puzzle locations is a Hindu temple. Outside, there’s a sign that says everyone is welcome, but please take off your shoes before you enter. There’s a shoe rack outside, and depending on the time of day there might be a few pairs of shoes, or many, many pairs. The puzzle asks: if there are many pairs of shoes outside the temple, what sort of good is the temple? And if there are only a few pairs, what sort of good is it?

‍

This gets at concepts we’ve taught in intro micro. We can think of goods along two dimensions. One is whether they’re excludable or non-excludable — whether people can be stopped from using them. Things in the grocery store are excludable, because you have to pay for them. A temple with a sign saying everyone is welcome is non-excludable.

‍

The other dimension is whether a good is rivalrous or non-rivalrous in consumption — whether someone else’s use of the good affects your use of it. If there are many pairs of shoes outside the temple, there are many people inside and it’s very crowded. So the temple is now non-excludable and rivalrous in consumption, which is what we call a common resource. You may have heard of the tragedy of the commons, which describes a problem with a lot of these sorts of goods.

‍

After students have solved a puzzle, we often provide more contextual application. For example, for one about religious institutions, we pulled an excerpt from The Economist about how we can think about religion using economics — salvation as a good, with buyers and sellers, and so on.

‍

How did students respond?

‍

We had over 600 students do this last autumn. There were upsides and downsides. The main downside was that it’s very hot in Singapore, and some students didn’t like that — and I don’t blame them, it gets really hot.

‍

Overall, the response was really good. Students seemed to enjoy it. A lot of them talked about how they now see economics around them, how it felt like being a tourist in their own country, and about bonding and teamwork.

‍

We also surveyed their perceptions of learning before and after the scavenger hunt. We found statistically significant shifts towards students perceiving learning to be more playful, more experimental, and more collaborative.

‍

In a nutshell: how do we make experiential learning possible in a very, very large class? Technology can be useful here.

‍

In Stone Centre workshops with employers, we’ve heard that “human” skills — teamwork and collaboration — are highly valued, and that they’re getting worse, especially in light of AI. These exercises seem to address that directly.

‍

Exactly. And to motivate students, we should be a lot more explicit: “What skills do you think employers are interested in?” I’m pretty sure at least some of them will come up with collaboration. Then we can remind them throughout the semester that while they’re students, the stakes are close to zero, so why not work on the skills they know will be valued?

‍

I think students are also in a catch-22: they know the easiest choices right now are probably not the best for them in the long run, but if everyone else is making them, what do you do? Extrinsic motivation, like employment, helps. But we need to think about how to design our assessments and our classes to create more opportunities for collaboration. The irony is that with AI, we are turning back to in-class exams because we cannot verify the integrity of unproctored assignments.