Impact studies often rely on traditional research methods like interviews and focus groups. While valuable, these methods have limitations. For one, there is often a power dynamic within interview settings. Secondly, because researchers come to an interview with a prepared set of questions, interviewees may feel constrained in terms of what they can share. Related to this, there's the problem of the socially desirable response: interviewees saying things not because they reflect their actual experiences or perspectives, but because they align with others' expectations of them.

To get around these difficulties, over the years, a variety of "researcher-absent" strategies have been proposed. Many of these revolve around self-produced data — that is, things created by study participants in their own way and in their own time. Examples of these include diaries (including video journals, which we've used in some of our evaluations), drawings and photographs, and self-selected objects.

Studies show that these methods can be empowering, as they alleviate anxieties associated with being in researcher-controlled spaces and give people a chance to "exercise their agency in the research process." By minimizing researcher input, these methods can also yield a "more naturalistic understanding" of people's lives. As a way of "studying the elusive," they make it possible to gather information about those everyday situations, contexts, and behaviors that researchers otherwise have no access to. Gathering data in settings that approximate people's everyday lives is useful, because it means that study findings are more reflective of and applicable to their daily realities.

Audio recordings can also be a form of self-produced data. In a recent study, we asked pairs of Radiolab listeners to record themselves while listening and reacting to an episode of this award-winning show. The conversational data they produced offered useful insights into the different metacognitive processes that emerge during the experience of podcast listening. In this blog post, we share what we learned from this experience — and what others might learn from it as well.

Study Overview

Our study was part of a broader project aimed at understanding Radiolab's impacts on learning and critical thinking. Twenty-four participants were placed in co-listening pairs, and asked to record themselves listening to and discussing a specific episode of the show. In keeping with established researcher-absent methodologies, we did not give people many explicit cues as to the type or frequency of comments to make while listening. The goal of the co-listening sessions was to simply capture in-the-moment engagement and learning.

Participants arranged a time and place to co-listen to their assigned episode, setting aside roughly 60-90 minutes for listening and conversation. While listening, participants paid close attention to the contents of their episode, observing not just key points and arguments, but also emotional cues or any other elements of the show that stood out to them. Participants also shared in-the-moment reactions to what they were hearing, blurting out their immediate thoughts and feelings and offering stream-of-consciousness commentary. If something they heard sparked a lengthy verbal exchange, participants often paused the episode until finishing their discussion. Many kept the recording on after the episode's conclusion, so they could wrap up their conversations.

What Did We Learn?

Before getting into our findings, it's important to note one crucial caveat: none of the things people did during co-listening reflect what the actual experience of listening to Radiolab is like. With the exception of road trips and other occasional events, most participants said they only listen to Radiolab on their own. More broadly, we know that for many people in the US, podcast listening is often a solitary experience.

The co-listening environment we created was thus an artificial one. Despite that, people's interactions gave them a chance to verbalize and share some of the real mental processes that everyone experiences during podcast listening. Their exchanges were examples of metacognition, or, "thinking about one's thinking." The recordings gave us a chance to identify different metacognitive processes in action — or in other words, to peer inside the "black box" of podcast listening. Aware of the fact metacognition promotes learning in a variety of settings, we thought it would be valuable to categorize and count all of these processes.

In all, we detected just over 2,500 metacognitive "utterances" in these recordings. Adapting and expanding on existing frameworks for classifying metacognitive these utterances, we developed a scheme that includes 14 different speech acts:

• Questions: Asking a question that is either about or related to episode content.
• Evaluation: Expressing an opinion about the validity of something heard during an episode.
• Interpretation: Taking something learned during an episode to make a new observation, reach a new conclusion, ask a new question, or propose a new explanation.
• Connections: Referencing prior knowledge or experiences and building bridges between these things and new information listeners encountered.
• Comprehension: Attempting to determine how correctly listeners grasped what they were hearing, and the extent to which they were following another's train of thought.
• Predictions: Making a guess at how the discussion of a topic might proceed.
• Synthesis: Verbalizing the process of trying to assemble the "big idea" of an episode and integrating that idea into a listener's prior beliefs.
• Recall / Recognition: Describing a memory triggered by something a listener heard.
• Revision: Integrating new information in ways that expanded or altered a listener's understanding of a topic.
• Perceptual Experience: Describing an imagined perceptual experience triggered by something mentioned during an episode.
• Epistemic Vigilance: Expressing skepticism about a person making a claim and/or the system of knowledge underlying that claim.
• Rehearsal: Recounting the content of something observed or remembered while listening.
• Deixis: Words or phrases (e.g., "Wow!" or "No way!") that invite another person to pay attention to something said by a host or guest.
• Grounding: Expressions that signal cognitive coordination with one's conversational partner (e.g., "yeah" or "mmhmm"), and that indicate whether another's attention is being shared.

The figure below shows the distribution of these utterances. For each co-listening pair, we calculated the proportion of utterances that fell into each category. We then calculated averages for each category across all of the 12 pairs. Those averages are represented by the x's in each line of the figure (Importantly, adding these averages together will not give a result of 1.0, because the x's are summary data across all listening pairs).

Because we were interested in understanding how common these processes might be among the general population (who listen alone), we also calculated the 95 percent confidence interval for each category. The results of these calculations are indicated by the whiskers on either end of the x's, and provide a sense of the estimated range each process might be expected to appear among Radiolab listeners.

Let's Put it to Work

As the figure above suggests, the co-listening recordings were an incredibly rich dataset — one that speaks to the wide range of metacognitive processes that occur during podcast listening. Despite the small number of participants in our study, it is possible to make hypotheses about the experiences of listeners more generally. Our findings also point to ways content creators can build metacognitive processes into educational media to strengthen learning outcomes.

More broadly, our study provides further support for the value of researcher-absent methodologies. The fact that participants spontaneously verbalized so many different metacognitive processes without any prompting from researchers suggests that self-produced data is an ideal way to study metacognition. In support of that goal, we are currently writing up a metacognitive utterances codebook for more general use. Stay tuned for that!

About this Article

This blog post is based on work funded by the John Templeton Foundation. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the John Templeton Foundation.

Photo courtesy of National Library of Wales @ Unsplash