Life in academia

Evaluations are in (again): approaching excellence

We've got more course evaluations to look at. I delve in the details of one of my courses to feel good about it.

Six months ago, I wrote about teaching evaluations for the marketing research course I had been giving last semester. Here’s how that post ended:

To teach or not to teach, that is the question. At face value, our improvements were rather small. There was a bit of up and a bit of down. A simple repetition of our original course maybe would not have done much different. On the other hand, we introduced some novelties that we had no experience with before. (Peer-grading, notably.) They maybe didn’t help much with the evaluations, but I do see how they helped students to improve themselves. We’ve learned to use another teaching tool: yes, that was certainly worth the effort.

That was a slim take-away, but better than nothing. Six months into the future, I’ve taught even more marketing research. Again, I put quite some effort into the teaching. Now I have many more numbers to look at. So, how did that go?

It’s necessary to add a little bit of background about what I taught this semester. For this post, I’m going to highlight one of the courses. It’s the one in which I ditched the textbook and in which I made students care about their readings.

Marketing research in cultural settings

This is an elective in Bocconi’s bachelor on economics in arts, culture, and communications. Alternatively, they have some very cool courses. There used to be one on tourism management in small places in the mountains. Most of their alternatives allow them to leave the class room. I find it difficult to teach marketing research skills on a hike in the mountains. So what happens is that almost no Bocconi student picks my class.

But exchange students do, as it’s a topic they easily can convert into credits at home. Naturally, they are in Europe not only for studying marketing research, but also for studying, well, Europe. It’s a fine balance between excellence that allowed them to go abroad and lack of interest that comes as a byproduct of being abroad. These are boundary conditions I need into account when I plan my teaching.

Previously

I’ve taught this course also in 2016 and in 2017. It has gone through a series of adjustments since. In 2016, this was a very standard marketing research course. I’d teach them how to define a research problem, how to approach a marketing research project, which survey channel to use and which sampling strategy to employ. I would spend a big amount of time on qualitative research. At the very end, I’d do some statistics-focused explanation of regression and ANOVA. As part of the course, they would run a small research rpoject in a group of four students. They could chose the topic themselves. The exam would then be based on the textbook.

For 2017, I changed the mode of the group project. I now had all students work on the very same project, so that there would be some coordination and a feeling of coherence. All projects would add to one bigger research question. This bigger picture allowed me to give them better methodological directions and it cut out the challenge to identify an interesting research project. I also gained some insurance that the variables they would measure in a survey would be adept for simple quantitative analysis. I switched to a shorter version of the textbook. And I superficially covered some contemporary methods like big data, predictive modeling, and natural language processing.

Evaluations didn’t change much and they didn’t really impress. They were good average.

New in 2018

For my third season, I eventually abandoned the textbook and replaced it with a series of self-curated resources. Basically, I made students read the materials that I had used to prepare my classes. And I actually made them read it. (And watch it.) There’s a full post on this on my blog already and I intend to write a follow-up on it now that the semester has finished.

The second big innovation brought an end to the group project. Instead, I tried to follow more of a portfolio approach to teaching marketing research. Students go through a series of weekly assignments that will cover the typical elements of a group project. Clear task description. Clear deadline. For any element of the portfolio, there’s a new and suitable dataset. No team risks to mess up their later submissions with a mistake in the early weeks. In order to control workload, they had to submit only four (of eight) assignments. Had they submitted more than four (about half of the class did that), I would have selected their best four.

On top of that, contemporary methods (conjoint, prediction, machine learning, natural language processing) now cover a substantial part of the course. To make space for that, my coverage of qualitative research is down to a week and I spend less time on discussing the absolute basics. Ditching the textbook gave me a lot of freedom here, because I wouldn’t run into an issue where half the book wasn’t covered in class and half the class wouldn’t be supported by readings.

cleacc_timetable

Approaching excellence

Bocconi’s teaching evaluation contains 16 questions. Students can give answers from 1 to 10, where 1-2 express strong disagreement and 9-10 express strong agreement. Personally, I care about the low end of the scale. A course is not a success when it scores more 10s then 9s. It’s a success when it scores no negative evaluations and everyone is on board.

To that end, the histogram of course evaluations looks more satisfying than last year: there’s not a single student anymore who gave any rating below 3 for any of the 16 questions. All ratings less than 8 appeared less frequent than last year.

evaluationshistogram.PNG

Three questions earned an average of less than 8.  Of these, two just marginally missed it. One question earned an average of 7.3.

I am interested in the course topics.

This is an interesting example of the means-vs-median game. The median value is 7 now and was 8 in 2017. The average was 6.82 in 2017. Anyways, interest is an uphill battle for methodological courses. I also think that this is the item we have least influence on. So, our biggest concern is now down to the interest of students that they might never have had. Honestly, that’s pretty satisfying.

For what it’s worth to look at the high end: each of the 16 questions earned full points from a third of the students. The sample is also larger than in 2017: twice as many students cared to fill out the survey. That is important, for silence in student evaluations expresses dissatisfaction-turned-indifference rather than satisfaction.

Overall, the course scored 8.79 on the composite score (7.99 in 2017) and I earned 9.19 on the teacher composite score (2017: 8.02). A full point more. Smooth. I’ve made it into a select group of teachers. At this point, it would be nice to have a look at where I’m coming from. Unfortunately, Bocconi has changed twice in the past 12 months how it calculates the overall performance of courses and teachers. And I can’t manually re-calculate it based on the data I have access to. I have asked for comparative data to one of our administrative angels, but apparently this might take a few weeks to get.

Comparing with myself

In 2018, this was my standout course. The other course in which I am the course director did not score much worse (9.06 on the teacher composite score and 8.35 on the course’s composite score). That one is also a marketing research course, but I’m teaching it in the bachelor of international governance and public policy. I put a stronger focus on nudging, and I have only half the sessions. On top of that, that one is a pass/fail course. Students complained heavily about the workload. They were right. For a pass/fail course, they had assignments and an exam. I thought we have to have an exam in every course at Bocconi. At least that’s what they had told me when I came here. Turns out that’s not true. They told me now. So… sorry to any student who might read this. In the next season, I’d drop the exam.

On top of that, I was co-teaching Marketing Decisions in the international management master. That’s Bocconi’s flagship program. My colleague was the co-director and I was new in this course. We went for a total reboot and developed the course around the infamous Markstrat business simulation. We had high ambitions, but apparently international management master students are a tough audience. This is a known fact at Bocconi. They give the most critical evaluations across all IM courses. According to the program director, our course actually performed really well in a program-specific benchmark. Our average across both classes returned a course composite score of 7.72. Still almost half a point higher than the course had had in 2017 (7.35).

And I think the course has a lot of potential. It’s one of those cases were experience really matters to get a course up and running.

It’s still not not all about numbers

My opinion is that it may take up to three years to develop a really good course. The first year I develop a feeling for what the course is like. Students are similar across courses, but each course challenges students uniquely. Group projects might work here, but not there. Peer evaluations can add value, and they can remain irrelevant. Even the content selection and the setup and design of slides can make or break a course.

Unfortunately, there’s no prototyping and there are no test markets in academic teaching. So, in the first year I come with lots of new ideas and give it an honest best try. Then, I will eliminate the most crucial mistakes. Hopefully. In the third year, I can focus my attention on the details.

After three years, there are still some details left to improve in this course. In my marketing research course, the qualitative feedback indicates that some readings are too long and too complex. To that end, the feedback surveys that I had them answer every week are now a wonderful resource to single out the worst readings. (In a way, I self-created my own system of instant evaluations that I advocated for last fall.) The other elephant in the room are the assignments. Overall, students like them, but they asked for more guidance and more explanations. That should be possible. Nothing of this is anymore a structural issue.

So, with the words of one of my students:

Thanks for being a great and inspiring professor. Your class was truthfully the most interesting one I took at Bocconi, and, while I’m excited to finish with exams next week, I’ll miss having such an interactive and fun research class.

And now I can lean back.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s