Program evaluations often take the form of surveys, interviews, and focus groups. Typically, these activities are conducted post-implementation — the idea being that the best time to gather data on a program is after its conclusion. But that's not always the case. And while the aforementioned methods might be beneficial to evaluators, they're not always useful for program facilitators or the communities they serve.

In recognition of these facts, we strive to conduct evaluations that support in-the-moment program improvements, and that directly benefit practitioners and participants. One way to do this is through embedded evaluation. As an alternative to post-hoc assessment tools, embedded evaluation makes use of artifacts (such as work samples created by participants) generated during program implementation. Studies indicate that embedded evaluation offers important insights into outcomes without the time burdens associated with surveys and interviews.

As evaluators of a new initiative launched by the Museum of Discovery and Science (MODS), we've had an opportunity to make use of a specific embedded data collection practice: video journaling. In what follows, we detail our experiences with this innovative methodology — and what we're learning about the benefits of the embedded approach to data collection.

Program Overview

In October, 2025, MODS hired us to evaluate an internship program called EcoExplorers. Part of a larger initiative called "Fostering Character Through Children's Museums," the program seeks to help interns (all of whom are high-school students) develop and exemplify positive eco-character traits. These include:

• Caring (e.g., developing empathy for all living things, and taking responsibility for protecting the environment)
• Cooperation (e.g., working with others to make a difference in their communities)
• Critical Thinking (e.g., understanding how humans depend on the the complex webs in our world, and learning how to analyze complex problems from different perspectives)
• Confidence (e.g., learning how to adapt to challenges, and becoming confident public speakers)

Interns are developing these traits through participation in a Youth Climate Summit, field excursions in the Everglades, engaging with mentors on MODS' Hub for Resilience Committee, and by serving as museum guides and interpreters — especially for the museums' forthcoming Building a Better World Together exhibit. The acquisition of these character traits is a key part of how EcoExplorers is helping interns become environmental stewards in ways that fuel South Florida's green jobs pipeline.

Evaluating EcoExplorers

To determine EcoExplores' impacts on interns' eco-character development, we're collecting lots of different data from lots of different people. In addition to having interns, caregivers, and MODS staff fill out surveys and/or participate in focus groups, we're also asking interns to do video journals. Recorded during their normal working hours, these journals offer interns a chance to share in-the-moment reflections on their own experiences and learning journeys. Each month, we provide a new prompt, and interns respond to it with a 3-minute video.

Making video journals part of our evaluation plan made sense for lots of reasons. For one, we wanted to reduce the data collection burden. Interns are working full-day schedules at MODS, and asking them to devote time to the program beyond working hours felt unfair. We also wanted to ensure that evaluation activities facilitated progress toward program goals. Research shows that reflective journaling can be a powerful learning tool, increasing critical thinking, the application of new knowledge, and self-awareness. Having an evaluation activity that served an educational purpose was a way to help interns see the value of taking the time to do their journals. And lastly, we wanted to gather in-the-moment data: not just interns' studied reflections on the program, but their immediate thoughts and feelings as they're going about their work. Getting that type of spontaneous feedback is a way to provide recommendations on program improvement that reflect participants' innermost thoughts and concerns.

The Value of Video Journaling

So far, our experiences with video journaling have been altogether positive. It's a low touch evaluation activity that is neither burdensome nor contrived. It's also very low pressure: interns know that their responses aren't being scored, and that the journals have no impact on their job evaluation. To make that clear, the journals are entirely optional: while encouraged to complete them, there's no obligation whatsoever.

Interestingly, even though they're voluntary, the response rate for these journals has been really high. In any given month, upwards of 90% of interns step into a "phonebooth" area to record their answers. And even more interestingly, despite knowing there's an outside audience for these journals, interns often treat them just as they would a private journal: as a space for sharing some of their innermost thoughts about how they're growing and changing. Compared to other program data we've analyzed (most notably, survey responses and applications), the things participants share in these videos are informal and organic. Instead of trying to "sell themselves" or giving socially desirable answers, interns appear unworried about what they're sharing. There's moaning and groaning. There's casual humor. There's authentic self-expression.

Because they're recorded in the midst of a person's shift, these journals are a great way for capturing what people are feeling in the moment — as opposed to the more reflective, curated, at-a-distance feedback that tends to come through in survey responses or post-hoc focus groups. As is true of journaling in a more general sense, the evaluation activity is designed to promote honest, off-the-cuff responses. Even though participants know that the recordings are going to be shared with us, they are very much like private journals in terms of their contents — they help capture people's feelings in "unguarded moments," and promote more unscripted reflections. Beyond that, the journals are giving us information about concrete actions and behaviors interns have undertaken — something that can be difficult to recall in interviews conducted after a program has ended.

Let's Put it to Work

As all of the above suggests, these video journals are giving us all kinds of information we couldn't otherwise gather. Having this type of data will allow us to provide a more comprehensive, holistic assessment of EcoExplorers. Moving forward, we're planning to continue having interns record video journals. We're excited to see what these videos reveal about how the program is strengthening their eco-character traits — and to learn about how these traits will factor into their long-term plans. After the evaluation, we're planning to redesign our journal prompt questions into recommendations on how MODS can better measure program impacts.

In the meantime, we're wondering how this unique approach to evaluation might be used in other contexts. If you've evaluated programs using this or other embedded data collection techniques, what have your experiences been like, and what have you learned? If you're in an organization that is looking for more new and different ways of doing evaluation, how might video journaling (among other types of embedded evaluation) be useful? We'd love to chat with you about ways this innovative methodology could be applied to your work, so if you're interested to learn more, please do get in touch!

About this Article

Both the EcoExplorers program and MODS' Building a Better World Together exhibit are currently being funded through a grant from the Lilly Endowment, as part of the broader "Fostering Character in Children's Museums" initiative.

Photo courtesy of Detail.co @ Unsplash