Introduction

The popularity of maker education and makerspaces is growing, both in formal and informal learning environments, valued for the deep learning they afford through engagement in authentic construction of physical and digital artifacts. Public libraries in particular have embraced this hands-on approach to learning, integrating maker education in their program offerings, and developing makerspaces that enable patrons to tinker and create products with technology from screen printing sets to robotics kits.

Scholars have endeavored to advance empirical research demonstrating the profound learning opportunities library makerspaces offer the public (Halverson et al., 2017; Kim et al., 2022; Li, 2021; Petrich et al., 2013; Willett, 2018), but significantly less attention has been paid to supporting librarians and maker educators in assessing the impact of their own spaces. Extant scholarship about assessment in maker education writ large conflates empirical studies with on-the-ground assessment practices that are meant to support learners and improve program offerings (Lin et al., 2020; Soomro et al., 2023), giving the impression that educators are well-equipped to implement or design assessments. In reality, assessing the impact of makerspaces, particularly those situated in public libraries and other less structured learning environments, is an ongoing challenge (Cun & Abramovich, 2018; Gahagan & Calvert, 2020; Hsu et al., 2017; Kim et al., 2022; Nixon et al., 2021; Teasdale, 2020; Wardrip et al., 2019).

Additionally, the majority of assessment tools and assessment design processes that are present in the literature are not applicable to public libraries, because they were developed for K-12 schools and other formal environments (Blikstein et al., 2017; Hughes & Thompson, 2022; Rosenheck et al., 2021; Welch & Wyatt-Baxter, 2018; Yorio, 2018), or structured, informal environments like academic libraries (Welch & Wyatt-Baxter, 2018) and summer camps (Worsley, 2021), which have different assessment needs and pedagogical affordances. This gap in scholarship pushes maker educators to rely on less relevant assessment tools that are already established and available, or to forgo meaningful assessment altogether (Gahagan & Calvert, 2020). By not having access to appropriate, meaningful assessments for their learning context, library maker educators are unable to make informed decisions or claims about their programs using robust, evidence-based processes.

In an effort to expand assessment scholarship and practices related to public library makerspaces, we offer two contributions in this article. First, we share findings from a qualitative research study in which we analyzed how 17 library staff and maker educators define success and identify evidence of success in their maker programs. The findings from that study, in conjunction with our collective experience as research partners working with public library makerspaces, laid the foundation for a series of analysis tools we developed to help stakeholders identify the assessment needs of such learning environments. The Properties of Success Analysis Tools (PSA Tools) represent our second contribution. The PSA Tools invite library staff and maker educators to reflect on and unpack their definitions of success for maker education in order to identify what relevant features an assessment tool should have.

Overall, in this article, we argue that developing relevant assessments for library-based makerspaces requires one to thoroughly understand the conception of success one is striving to achieve, and we offer a series of analysis tools to help stakeholders do that. We briefly share findings from our study, introduce the PSA Tools, and describe how they can be used. Although the findings and analysis tools emerged from and were designed with public library makerspaces in mind, we discuss other informal learning environments that might benefit from this work.

Problem statement

Creating appropriate assessments is difficult in any circumstance (Hunt & Pellegrino, 2002; Pellegrino, 2005), but it is especially challenging for emerging learning environments like makerspaces (Cun et al., 2019; Shute & Wang, 2016). Makerspaces boast features for which it is difficult to design universally applicable assessments, features like interdisciplinarity, which poses a problem for content-area assessments (Halverson & Sheridan, 2014), and emphasis on a scrappy, do-it-yourself culture and heterogenous learning pathways that resist standardized outcomes (Sheridan et al., 2014; Teasdale, 2020; Willett, 2016) that resist standardized outcomes.

Assessing maker-based learning in less structured settings, like public library makerspaces, is particularly difficult. In more structured, formal settings, like K-12 schools and pre-registered camps or workshops, the learning environment is more predictable, allowing educators to prepare a more thorough assessment plan. Educators can pre-select standardized constructs to assess across students and embed opportunities for students to showcase their progress throughout the lesson plans for assessment purposes. In the case of K-12 schools, there is also a culture of assessment in which students expect teachers to document information about their products and processes over time.

In contrast, maker educators in less structured, informal settings tend to see learners for unpredictable amounts of time, and there is a general expectation that these environments should not operate like schools, which requires creative ways of engaging learners in any assessment practice. Moreover, library staff and maker educators rarely have information about the learners that will attend their programs; they are tasked with on-the-fly assessment for learners who may have dissimilar goals and prior knowledge, who may be mixed in age, and without knowing how many learners will attend any given activity. In essence, informal, less structured makerspaces have unique environmental and pedagogical constraints that make it unrealistic to implement or adapt assessments designed for structured learning settings.

In response to the dearth of appropriate assessment tools for public library makerspaces, scholars have begun advancing literature meant to support assessment design, but it is a limited body of work. Kim et al. (2022) describe the seven publications they found in their systematic review of library makerspace research, including three studies that defined outcomes and examples of success specific to library makerspaces, two that advanced assessment development frameworks, and two that conducted assessments.

Without concrete assessment tools and sufficient empirical research to guide assessment design for library makerspaces, library staff and maker educators are left to make due with less appropriate assessments designed for different learning environments, like content knowledge tests (Lin et al., 2020) and learning portfolios (Hughes & Thompson, 2022), or resource-heavy tactics drawn from research methods like surveys, observations, and interviews (Lin et al., 2020). If these options are simply not feasible, library staff and maker educators may rely exclusively on the common library practice of collecting attendance metrics for activities, which say nothing about what a learner actually experiences or gains from an activity (Gahagan & Calvert, 2020).

We address the problems left by this gap in literature by (1) advancing scholarship about library staff and maker educators’ perceptions about success and evidence of success, and (2) by introducing a series of analysis tools that help staff and educators understand what features their ideal assessment should have based on their definitions of success. First, we share findings from our qualitative research study with 17 library staff and maker educators across two public library systems. The research questions that guided this study were:

  • How do library staff and maker educators define success in their maker-based learning programs?

  • How do they recognize and describe evidence of success they identify?

Second, we introduce the PSA Tools we developed to help scholars and educators in the assessment design process. These tools were created based on the findings from our study and our collective experience as research partners working with and studying public library makerspaces. We introduce the tools, describe how they can be used, and discuss the implications they have for other learning environments with similar features as public library makerspaces.

Background

In need of something new

Despite the vast scholarship available on educational assessment and program evaluation and the relative abundance of literature about maker education in general, there is a noticeable lack of research and tools that support assessment in maker education. In their systematic review of empirical research on assessment in technology-rich maker environments, Lin and colleagues note that out of hundreds of articles related to making and makerspaces, only 60 were related to assessment (2020). The body of literature available about assessing library makerspaces specifically is significantly smaller. According to Kim et al. (2022), in a review of research specific to library makerspaces (43 articles total), only seven were related to assessment and evaluation. This is particularly surprising given that public libraries are one of the more common settings for makerspaces.

Of the literature that does exist on assessment in maker education, we recognize two main issues. First, individual studies on assessment in maker education and reviews about these studies conflate empirical research with on-the-ground assessment practices, obscuring the fact that access to research-practice partnerships is a necessary condition for implementing some of the more involved assessments (Blikstein et al., 2017; Kim et al., 2021; Yin et al., 2020). There is also a general sense that by designing assessments and making them available to the public, scholars are closing the gap in assessment tools for makerspaces, but there is little evidence to suggest that maker educators feel confident implementing these assessments in their spaces or, barring that, developing their own assessments following assessment design processes outlined in scholarship. In reality, assessing the impact of makerspaces continues to be an ongoing challenge for maker educators in public libraries (Cun & Abramovich, 2018; Gahagan & Calvert, 2020; Hsu et al., 2017; Kim et al., 2022; Nixon et al., 2021; Teasdale, 2020; Wardrip et al., 2019).

Second, the majority of available assessments are designed for formal, structured maker learning environments, making them largely inappropriate for informal, less structured settings like public library makerspaces. As an example, K-12 makerspaces often have expectations like compulsory attendance, age-based groupings of students, regular assessments of learning or tests, grading student work, and specific demonstrations of engagement (e.g., following instructions). In contrast, for makerspaces situated within informal, less structured learning environments, like public library makerspaces, there is general recognition that learners will behave less predictably; they may enter and leave an activity as they please (and are not shy about abandoning an activity they find boring or too much like school), and sometimes with their preschool aged sibling or adult guardian in tow. Additionally, their attendance over multiple programs is not guaranteed. These environmental conditions make it unrealistic to transfer or adapt assessments built for K-12 makerspaces.

Lin et al. (2020) detailed five main assessment approaches that appeared in makerspace literature overall: assessment of learners’ final artifacts; “traditional” knowledge tests; individual surveys largely concerned with learners’ dispositions and self-reported levels of confidence; interviews aimed at understanding learners’ learning process and perceptions; and observations and field notes taken in situ. However, it is unclear how effective these assessments are in the context of library makerspaces.

Without concrete assessment tools and sufficient empirical research to guide assessment design for library makerspaces, library staff and maker educators are left with few options. They can use assessments made for formal learning environments, narrowing the extent of their impact they are able to document, assess, and report. They can return to collecting attendance data, a quick but negligible approach to measuring impact in libraries (Gahagan & Calvert, 2020). Or they can forgo systematic assessment altogether and instead make decisions about learners and programs based solely on their experiences and intuition (Chen & Bergner, 2021; Saplan et al., 2022a). However, to meaningfully understand the impact that public library makerspaces have on learners, library staff and maker educators need assessment tools and strategies that are responsive to the particular needs and affordances of their learning environments. With this paper, we advance new analysis tools that can support the development of assessment tools.

Understanding success to assess

In its most basic form, assessment requires three elements (1) an outcome, construct, or competency that will be assessed along with a theory or set of beliefs about how learners will develop and demonstrate it; (2) a set of beliefs about the kinds of tasks or situations that will prompt learners to demonstrate that outcome, construct, or competency; and (3) an interpretation process for making sense of learners’ demonstrations (Pellegrino et al., 2001). These elements represent the outcome that will be assessed, a process of observation for gathering evidence of that outcome, and a process of interpretation for making sense of the evidence, what Pellegrino et al. refer to as the assessment triangle. They argue that assessment will be most effective if the assessment designer starts with an explicit and clearly conceptualized understanding of the outcome, construct, or competency they are trying to assess (p. 45). Several other frameworks support this assertion, from Universal Design for Learning (Rose, 2000) and Understanding by Design (Wiggins & McTighe, 2005) used in teacher preparation and curriculum development for formal environments, to large-scale assessment development processes like Evidence-centered Assessment Design (ECD) (Mislevy et al., 2003).

However, this body of literature is only useful to the extent that educators know what outcomes they want to assess and what evidence to look for. In library makerspaces, staff and educators do not receive uniform training in curriculum and assessment development, and it is likely that the majority of staff and educators were trained as librarians rather than informal educators. Indeed, previous research suggests that this crucial, initial step is not always obvious to library staff and maker educators, especially when they must consider the desired outcomes of multiple stakeholders (Wardrip et al., 2019). Moreover, it can be challenging to pinpoint a desired outcome for assessment due to the multidisciplinarity of maker education and the sheer diversity of potential outcomes in makerspaces.

Numerous learning outcomes are cited in maker education scholarship, from engaging in authentic STEM practices and with authentic disciplinary tools (Vossoughi & Bevan, 2014) to connecting and sustaining cultural practices through making (Barajas-López & Bang, 2018), from developing identity in a community of makers with similar interests (Willett, 2018) to encouraging and preparing learners for entrepreneurship (Teasdale, 2020). The potential learning outcomes afforded by makerspaces are so vast that several scholars have attempted to organize and categorize outcomes by type. Lin et al. (2020) identified three categories of outcomes including cognitive outcomes (e.g., STEM-related content knowledge), affective outcomes (e.g., confidence, attitude), and other outcomes (e.g., engagement, collaboration). Similarly, Mersand (2021) cites four domains of learning outcomes drawn from Anderson and Krathwohl (2001) and Steinberg and MacDonald (2019) in his review of makerspaces research: affective outcomes, cognitive outcomes, psychomotor outcomes, and behavioral outcomes (p. 181).

Gutwill et al. (2015) steered away from the cognitive and affective divide, and instead advanced four learning dimensions of making and tinkering. Their framework includes engagement, initiative and intentionality, social scaffolding, and development of understanding, which emphasize the practices or indicators learners exhibit when meeting those dimensions (e.g., displaying motivation, taking intellectual risks, expressing a realization). Wardrip and his colleague (2015) also developed a framework of learning practices in makerspaces that can be considered key competencies of maker based learning. In their framework, learners Inquire, Tinker, Seek and Share Resources, Hack and Repurpose, Express Intention, Develop Fluency, and Simplify to Complexify.

In addition to the possible outcomes and competencies that library makerspaces might design assessments around, recent literature has been published urging scholars to center stakeholders definitions of success in assessment design for library makerspaces (Teasdale, 2020, 2021). For Teasdale, understanding success instead of outcomes or competencies is important, because patrons may not align their purposes as makers with specific, discrete outcomes, objectives, or competencies. Teasdale (2020, p. 1) recognized the importance of defining success as the first step in an assessment process, arguing that doing so informs the criteria by which library programs are assessed (in contrast with schools, which tend to have built-in criteria for assessment).

And because the library makerspace is founded on the intention of supporting patrons’ goals and needs, success offers a more holistic frame through which to think about what can be assessed in these spaces. For us, success also relocates the power of what “counts” as meaningful outcomes with whomever is asked to share their definition of success. In a sense, success is in the eye of the beholder, whereas outcomes, constructs, and competencies tend to be conceptually imposed on learners and learning environments.

Practically speaking, staff and educators in library makerspaces are not exclusively focused on measuring learning outcomes or competencies. They also strive for more nebulous goals, like “reducing that educational gap especially for kids of color,” “[seeing] a segment of the community come in [to the makerspace] that isn’t a normal user,” “being able to represent artists from other cultures [in our programs],” or seeing “is there joy happening?”—as staff and educators mentioned in our data. Outcomes-based evaluation and competency-based assessments are not able to document evidence of those goals. To address the disconnect between what staff and educators consider evidence of that success and assessments, our research study and analysis tools began with staff and educators’ definitions of success and the evidence they prioritize.

The initial goal of our library partnerships for this study was to analyze the library makerspace environments in depth and eventually develop assessments that were grounded in their specific conceptions of success and assessment needs. By first studying the learning setting in which an assessment will take place, we would ensure that there was alignment between what educators wanted to assess and what they actually assessed. Unfortunately, the onset of COVID-19 forced us to adapt our work with these libraries, but not before we were able to at least begin our project. But that initial goal dictated the overarching process through which we engaged in analysis that enabled us to develop the PSA Tools.

In order to support the development of assessments for library makerspaces, we conducted a qualitative study to understand how staff and educators in two public libraries define success and identify evidence of success. We used an approach that is informed by ECD (Mislevy & Riconscente, 2005; Mislevy et al., 2003)—a process in which an analysis of a domain of learning is the first step of assessment design, in order to situate any assessment in authentic practice. Mislevy et al. (2003) state that as a result of the domain analysis process, the central outcomes or visions of success for assessment must make sense to the educators, “domain experts,” and researchers, and they be organized in such a way that leads researchers to the next stage of ECD (p. 8). We selected ECD to ensure that our assessment development process was grounded in authentic practices within library makerspaces.

Research study

Methodology

To conduct our qualitative research study, we followed the ECD process. ECD is a multi-step process that starts with identifying the “complex of knowledge, skills, or other attributes [that will] be assessed” in a learning environment (Messick, 1994, p. 16, as cited in Mislevy & Riconscente, 2005). The five steps include domain analysis, domain modeling, conceptual assessment framework, assessment implementation, and assessment delivery. In Table 1, we briefly describe each of these five steps.

Table 1 The five phases of evidence-centered assessment design
Full size table

Following ECD, we started with domain analysis of makerspaces in two library systems. Domain analysis is an opportunity for substantive gathering of information about a learning environment and the domain of learning within it, consisting of “the content, concepts, terminology, tools, and representational forms that people working in the domain use. It may include the situations in which people use declarative, procedural, strategic, and social knowledge, as they interact with the environment and other people” (Mislevy & Riconscente, 2005, p. 7). The research questions that grounded this study were:

  • How do library staff and maker educators define success in their maker-based learning programs?

  • How do they recognize and describe evidence of success they identify?

We relied on semi-structured interviews (Kvale & Brinkmann, 2009) with library staff and maker educators as our main data source for domain analysis, and we engaged in several rounds of thematic analysis of the transcribed interviews by using emic coding strategies like descriptive coding, concept coding, and subcoding (Miles et al., 2020).

Methods

Study sites

Two makerspaces were included in this study: the Learn to Make Lab (LML), and Making an Impact (MI).Footnote 1 The LML is located in one branch of a 37-branch system that serves suburban, somewhat rural, and inner-city regions alike. It has two studio spaces, “high-tech” materials common in makerspaces (e.g., 3D printers, laser cutters, robots, and interactive technologies), as well as more analog tools like a button maker. MI is housed in one library within the nine-branch system. Like MI, LML provides materials for patrons to drop in and use. The majority of materials at MI are “low-tech” (e.g., craft materials, a button maker, recycled goods), but they also boast a media lab equipped with sound and video technology. Additionally, MI partners with local artists to host programs in the space, and sends maker-based kits across all Madison Public Library branches, ensuring that maker programs can take place across the city, functioning like a “system-wide” makerspace.

These libraries were intentionally selected for two reasons. First, because they had existing relationships with the researchers, and due to these relationships, the researchers had been in regular discussion about problems of practice with assessment in their makerspaces. Second, these libraries were already active in designing and facilitating programs to support maker-based learning experiences, and they were interested in developing meaningful assessment in their spaces.

Sample

Although the overarching domain analysis of the case libraries included a variety of stakeholders from educators to participants, this paper focuses on responses by library staff and maker educators exclusively. Library staff and educators were selected to participate in the interviews, whether their responsibilities were directly or indirectly related to the maker education programs in their library. We did not conduct interviews with a large sample size, because our aim was not to determine the average response to our research questions across stakeholders. Instead, we purposely invited staff and educators we thought would provide maximum variation sampling for each library site and illustrate a range of responses (Merriam & Tisdell, 2016). This approach to selection supports inductive and exploratory research, and aligns with the constructivist paradigm that grounds our study.

Every staff member and educator who participated in the interviews was connected to a maker program to some degree, though their connection varied in terms of their exact role within the library. The years of experience staff and educators had working in their libraries varied greatly, from “almost a year now” to over 10 years. We spoke with maker educators and teaching artists, educator supervisors and other operations managers, as well as community and youth libraries who also facilitate maker programs.

Data collection

We conducted semi-structured interviews with seventeen library staff and maker educators (Rubin & Rubin, 2005), which ranged from 25 to 50 min, and were completed in person by two members of the research team. Interviews were digitally recorded. Staff and educators were asked: What is happening when things are going really well in the makerspace? How do you know? How could you know? Sometimes, staff and educators struggled to answer these questions, so the researchers would follow up with secondary prompts like, What does it look/sound/feel/smell like when things are going well? Even if it hasn't happened yet, what would it look like? As the interview process evolved, researchers would also ask, How do you know when your program is working? These questions ask staff and educators to implicitly describe what they consider success in their program and what evidence they draw on to determine whether their definition of success has been met.

Data analysis

Interviews were transcribed with Scribie software, then the thematic analysis process began when the third researcher on the team, who did not conduct the interviews, read through and cleaned up the transcriptions to gain an initial understanding of their contents. Each interview was then transferred to a spreadsheet, or analytic matrix (Maxwell, 2013; Miles et al., 2020), where the exact text was organized by the following headings: Interview Question, Follow Up Question(s)/Question Reiteration, and Verbatim Response. Other columns were added to the spreadsheets to capture the Researcher’s Initial Interpretation of the Data and their reflection on the Implications of their interpretation. Columns which identified the Initial Code and Sub-codes for each section of data were also included, along with a column for Post-coding Researcher Questions/ Notes that emerged during analysis. This analytic matrix allowed the researcher to simultaneously code the data and write memos about their sensemaking (see Image 1 for a sample of the spreadsheet). Throughout the coding process, all three researchers on the team met bi-monthly to discuss the data and engage in constant comparison of themes or “incidents” as they emerged (Corbin & Strauss, 2008).

The data was first coded with this matrix using three coding strategies: descriptive coding, which identified the topic of the response; concept coding, which described the content of the response; and sub-coding, which assigned a secondary, more detailed sub-code to each “parent code” (Saldaña, 2016).

Image 1
figure 1

Analytic matrix

Full size image

The coding matrix yielded a list of codes that were then reorganized into a code map, or a list of codes and subcodes organized by loose themes (Saldaña, 2016). The code map was “cleaned up” by condensing redundant codes and breaking down longer codes. During the cleaning process, the researcher responsible for analyzing the bulk of the data wrote analytic memos to trace their thought process and reflect on the preliminary themes they recognized.

Following that, they created another blank matrix using the same headings and coded the data again, this time drawing from the code map if it was relevant and developing new codes when necessary. This iteration enabled the researcher to compare analyses and check the cogency of their previous codes. After the second round of coding was completed, the researcher revised the code list where necessary, and applied a pattern coding strategy to the code list to organize and reorganize the codes into categories (Saldaña, 2016).

This second list of codes were shared and discussed with the main collaborator from each library site, an analytic tactic known as member checking (Merriam & Tisdell, 2016). Member checking provided collaborators the opportunity to challenge or confirm the accuracy of the codes while they were still underdevelopment. The code list was revised again with their input; some codes were renamed, subcodes were reorganized, and their feedback generally strengthened respondent validation of the codes.

The list of codes at this point in analysis reflected over 130 individual codes and subcodes related to success and evidence of success. In an effort to make sense of the common themes among these codes, the researcher conducted the third and final iteration of coding for this study, which included several rounds of code mapping and pattern coding simultaneously to further condense the code list (Saldaña, 2016). During this phase, the codes were categorized under five types of success, which we used to develop our previous framework, known as the Typology of Success (Saplan et al., 2022a). The typology identified five success types and described assessment features from the literature that could be relevant to each type. The purpose of the typology was to help educators and scholars make more informed decisions about their assessments. We presented it to scholars and educators at two conferences (International Conference of the Learning Sciences and Play, Make, Learn Conference) and received feedback that led us to recognize gaps in the framework, the largest gap being that simply identifying success types did not guide library staff and maker educators in drawing connections between their definitions of success and their space’s assessment needs.

We returned to the list of 130 individual codes and began to reorganize and refine our categories, all while reflecting and reorganizing the initial framework we published. By considering both representations of the codes simultaneously, we sought to understand how these codes could be organized in a way that operationalized staff and educators’ definitions of success and evidence of success to make decisions about assessment.

During this stage of analysis, we realized that the five common themes we identified were not an exhaustive list of overarching categories of success, but rather underlying features within a definition of success—properties that make up one’s definition of success. Conceptualizing these themes as properties, we reviewed the raw transcripts again and noticed that staff and educators implicitly described these properties more consistently than they did success types. As we continued to identify examples of these properties of success in the 17 interviews, we also recalled and discussed other studies we had conducted and publications we had read that included conceptions of success that mapped onto these properties. Ultimately, the five common themes were adapted into Properties of Success for the PSA Tools. Below, we describe these themes and examples of these themes directly from the data, and we introduce the Properties of Success into which these themes were adapted.

Findings

Our analysis revealed that library staff and educators spoke about four key themes when describing success and evidence of success in their makerspaces. They discussed who is expected to demonstrate success, how success is demonstrated, what makes success visible, and with whom instances of success must be shared.

Who is expected to demonstrate success

In their interviews, library staff and maker educators unsurprisingly asserted that learners and library patrons who participate in a makerspace are often the person they expect to demonstrate success. One maker educator offered that they know their program is successful.

When I can see that people-- and this is a general answer-- that people are engaged…when someone is in the zone on the 3D printer like getting a model into the software and getting it ready to print, but not distracted by the kid that’s ramming a little robot in their chair…[someone] that’s very interested in what they’re doing as well. When you have each person in their own domain of what they’re trying to focus on, I consider that being a successful environment. Or when people are asking for help often, because then you know that they’re trying to do something that’s beyond what they had before coming to this space, because they’re trying to make something or they’re trying to improve themselves by learning something outside their wheelhouse.

However, staff and educators also suggested that the learner or patron interacting directly in a program is not always the person they expect to demonstrate success. A library staff member described success as something that a maker educator can demonstrate, when they said that success occurs “when you start to see the facilitator or the artist adapt and change based on the different folks in the room, seeing them be able to pivot and build on what they’ve done before.” Another educator noted that while her programs are directed at young people, she looks for evidence of success in their parents:

In a drop-in program, when I get a bunch of different kids showing up every week and I have no idea who it's going to be from week to week...I get a couple of kids, a handful of kids, that are there every week, and we can track their development. But what's more interesting to me is tracking changes that parents make to their own attitudes towards play, and attitudes towards their children at home, and not just within the scope of the program. What is always the most rewarding to me is when a parent will come to me and say, “You know what? I came to your program last week and then this week when I was at home with my kid they decided they were gonna try something. And normally, I would have been like, 'No, don't do that. That's a great big mess. Or that's gonna be dangerous.' And I just stepped back and I watched them and they did it, and I was amazed. I didn't know they could do that. It totally changed how I think about my kid.” And I'm just like, “That's the thing I want.”

Staff and educators identifying different people who are the target of their definition of success inspired the Property of Success known as the Subject of Success. Based on our awareness of maker education literature on community and family learning (Roque, 2016; Tzou et al., 2019), we recognized that the subject of success might also refer to individuals or groups.

How success is demonstrated

When describing how they know when their definition of success has been achieved, staff and educators offered examples such as:

  1. (A)

    So, kids learn particular skills; kids change their mindsets about whether they're learners or not; parents get feedback on their kids' learning or parents are reporting six months later that their kids are more engaged in school or taking on more kind of active learning at home and reading at a better grade level.

  2. (B)

    If I can see a kid having an "aha moment" or figuring something out.

  3. (C)

    If kids say things like, “Look what I made or look what I did,” and show pride in their work and pride in what they’re doing-- that I consider successful. Or if they show off to other grown-ups who pass through the room, like, “Look what we’re doing.” Any of that kind of pride or talk about wanting to take something home and show parents or show their siblings.

From responses such as these, we recognized that staff and educators were referring to examples of how success was performed; either by acquiring skills and mindsets that are applicable across learning environments (example A), or by demonstrating behaviors that indicate one’s successful participation in the makerspace (examples B and C).

Drawing from our expertise as scholars, we noticed that these two ways of demonstrating success aligned Sfard’s two metaphors for learning, the acquisition metaphor and the participation metaphor (Sfard, 1998). In the acquisition metaphor, learning is described as “knowledge acquisition” and “concept development,” and it portrays the “human mind as a container to be filled with certain materials and about the learner as becoming an owner of these materials” (Sfard, 1998, p. 5). Sfard likened learning in the acquisition metaphor to the accumulation of “material goods.” In contrast, the participation metaphor resists the notion that knowledge is a commodity that can be possessed, absent, or even lost (McKinney de Royston & Vossoughi, 2021). Learning, according to the participation metaphor, is an action someone partakes in, a “constant flux of doing,” a “process of becoming a member of a certain community” (Sfard, 1998, p. 6). Learning in the participation metaphor includes “the ability to communicate in the language of the community in question, and to act according to its particular norms [which] themselves are to be negotiated in the process of consolidating the community” (p. 6).

In addition to describing how success is performed in makerspaces, staff and educators implied that success is observable at different timescales. Example A alludes to a longer timescale at which success becomes apparent, either in the undetermined time it takes for a learner to gain a skill, or in the suggested 6 months after a program that a parent shares feedback about their child’s change in behavior. Examples B and C suggest that success can occur in a shorter timescale, either immediately and in the (aha) moment, or after an activity has concluded and learners are leaving the library.

From this theme about how success is demonstrated, we developed two Properties of Success: Performance of Success and Timescale of Success.

What makes success visible

Library staff and maker educators identified two types of evidence or data they look for in order to determine whether their definition of success has been met. They rely on quantitative metrics, like “How many times [a kit] gets used” and “We had 100 story times,” but they also rely on qualitative evidence, “[Patrons are] jumping around, but that passion, that excitement from whatever inspired them is still there…In the midst of them going from here to here and then…there is still curiosity.” The Property of Success that is associated with this theme is Data for Success.

With whom instances of success must be shared

Perhaps unsurprisingly, staff and educators are concerned with who needs to hear about their instances of success, whether it's the library board, their Executive Director, other partner organizations, or families. And how instances of success can be shared with stakeholders impacts their perspective on what an assessment should do, as one library staff discussed in detail:

I drafted up a report that had the number of people who attended each event, how it went, and then broadly program feedback from the participants, and I sent that off to, you know, a couple-- like my direct manager, and then the manager who kind of communicates most directly with the foundation (funding body)…So yeah, so the foundation, that's really a big accountability piece when it's external funders, the funding things. We've talked a lot about things like project outcome, so using, like especially with the adult services group that I mentioned. It's like, how are we consistently going to be evaluating all of these workshops and programs and stuff like that that we do as we develop a more robust resource sharing thing?

As this example suggests, evidence of success that is captured through assessment can be used for a number of purposes including marketing and promotion, program improvement and development, or as justification for city funding. The purpose of assessment is inextricably linked to its intended audience and those who will be impacted by the assessment. Stakeholders who need to receive information about makerspace outcomes and instances of success are either internal or external to a learning opportunity (Black & Wiliam, 2018). Learners and educators are both internal stakeholders because they engage directly in the learning opportunity or maker activity. Caregivers of patrons, board members, and funders would be considered examples of an external audience, because they are removed from the activity itself. If a learner is considered the main stakeholder, the results of the assessment will need to be meaningful to them; perhaps the results will be used to support the patron’s ongoing learning trajectory. If the library Board of Directors is the main audience, they might need completely different information from a learner, or at least assessment results that are translated in a way that informs budgeting decisions. The Property of Success that aligns with this theme is Audience for Success.

Properties of Success Analysis Tools

The PSA Tools consist of three tools: the Properties of Success Analysis Tool, the Success and Assessment Table, and the Reflection Prompts.

The Properties of Success Analysis Tool

The Properties of Success Analysis Tool is essentially a graphic organizer that educators can use to explicitly articulate the definition of success they hold for their program or learning environment. The tool helps educators break down their definition into five essential properties they need to understand in order to select or design an appropriate assessment that measures that success (Image 2).

Image 2
figure 2

The Properties of Success Analysis Tool

Full size image

This tool asks educators to define success in their program or learning environment, to describe examples of that definition of success being met, and to answer analysis questions about their definition of success. The analysis questions include:

  • Who do you anticipate will demonstrate this example of success? Who is the Subject of your definition of success?

  • What is the Metaphor that guides your definition of success? Is this example of success something that one acquires (knowledge, skill) or performs (persistence), or something one participates in (collaboration)?

  • When do you anticipate this example of success will be observable? At what Timescale should you assess this example of success?

  • What type of Data do you believe best indicates success has been met? Qualitative, quantitative, or a combination of both?

  • Who is the Audience for this definition of success? Who do you need to share this success with?

These questions ask educators to think about The five Properties of Success, which include: Subject of Success, Guiding Metaphor of Success, Timescale for Success, Data for Success, and Audience for Success. To answer these questions, educators are encouraged to write in the boxes provided as well as in the petal diagram. The petal diagram at the center of the Properties of Success Analysis Tool illustrates how the five Properties of Success are connected to each other and to their overarching definition of success.

Properties of Success

Subject of Success

The Subject of Success property focuses on who is central to a definition of success. With this property, we advance main two assertions:

  • Many different stakeholders can be the subject of success in a makerspace, not just the library patron directly participating in a maker activity

  • Success can be assessed with one or multiple subjects at the center of assessment

Performance of Success

The Performance of Success property highlights how success is demonstrated by the subject. This Property is based on the two metaphors for learning advanced by Sfard (1998): the acquisition metaphor and the participation metaphor.

Timescale for Success

The Timescale of Success property refers to the time in which one’s definition of success is expected to be observable in the subject. Library staff and maker educators spoke about success as a phenomenon that can appear immediately or surface over different timescales. We broke this property down into three time-based categories:

  • Immediately visible: immediate, in-the-moment instances of success (e.g., a learner makes a realization that helps them move forward in an activity or project)

  • Visible over some time: instances of success that occur at the end of a maker program or at the end of a short series of programs (e.g., the majority of learners in a program stayed long enough to complete a project)

  • Visible over more time: instances of success that only emerge over multiple programs or multiple years (e.g., two siblings seek out maker programs across multiple library branches; one learner develops enough skills in the music studio to eventually record a Grammy-nominated song).

Data for Success

The Data of Success property asserts that explicitly discussing what kind of data to collect with assessment may bridge the gap for library maker educators between identifying their own definitions of success and deciding what assessments will elicit that success. This property also encourages staff and educators to recognize both as valid types of data in assessment.

Audience for Success

The Audience for Success property refers to the audience or stakeholders with whom instances of success must be shared. The design of an assessment might change depending on who the audience is.

The Success and Assessment Table

Educators are meant to use the Properties of Success Analysis Tool in conjunction with the Success and Assessment Table. The Table defines each of the Properties of Success, includes examples of each Property, describes how each Property impacts assessment development, and finally offers examples of relevant assessments foci depending on the Property (Image 3).

Image 3
figure 3

The Success and Assessment Table

Full size image

Educators should use the blue portion of the Table to help them fill out the petal diagram. The white portion of the Table conceptually links the Properties of Success to decision making about assessment.

The Reflection Prompts

After filling out the petal diagram, educators turn to the Reflection Prompts, to explicitly describe the type of assessment that is most appropriate for their definition of success, and to reflect on the affordances and constraints of their current definitions of success (Image 4).

Image 4
figure 4

The Reflection Prompts

Full size image

Used together, these tools are meant to encourage educators to engage in a cycle of reflection and practice as they consider the relevance and comprehensiveness of their definitions of success, and clarify what assessments best suit their definitions of success. These tools are not meant to be used once and only prior to assessment development. They can offer educators an opportunity for continuous reflection and adjustment. They can also be used to break down current assessment practices as well by reversing the order in which they used the tools. Educators can start by describing their assessment practices on the Reflection Prompts, use the Table to connect their assessment practices with the Properties of Success, and then fill in the pedal diagram to write out their definition of success.

Using the PSA Tools, educators will be able to more clearly see how their definition of success impacts the type of assessment they must use. When used prior to developing an assessment, these tools allow educators to articulate their understanding of success, which might not be something they have experience with. When a team of educators or colleagues with varying levels of assessment literacy use these tools, they can establish a common language for discussing success and assessment. These tools also invite educators to consider different ways of assessing learning. For example, research on assessment in informal arts education indicates that organization leaders and educators believe assessment is only valid and valuable when it draws on quantitative data (Saplan et al., 2022b). The PSA Tools challenge that belief by communicating various assessment options and reinforcing the idea that there is no singular, correct way to assess learning; rather, there are more or less appropriate assessment options depending on one’s definition of success.

Discussion

The motivation behind developing the PSA Tools was to support library staff and maker educators in selecting or designing assessment tools for their makerspaces. These tools, and the study on which these tools were built, are meant to drive theory in literature on library maker education and to be used practically. We believe they enable educators to make direct connections between their definitions of success, the evidence that constitutes success, and the assessment tools they need to document that success. The PSA Tools accomplish that by encouraging stakeholders to explicitly state the visions of success that will be the center of their assessment, and by breaking down the multiple components or properties that make up their definition of success to better understand their vision and their assessment needs. Moreover, because these tools break down otherwise complicated conceptual processes, they can be used by library staff and maker educators who have a wide range of experience with assessment. They can be included in staff meetings to develop common language and understanding of assessment, and generally support the development of a robust culture of assessment within a learning environment (Farkas, 2013).

In literature like Universal Design for Learning (Rose, 2000), Understanding by Design (Wiggins & McTighe, 2005), and ECD (Mislevy et al., 2003), formal teachers and assessment scientists are directed to connect program goals with assessment tasks and tools in an effort to design cohesive assessments. However, it is not explicit how to make those connections. The PSA Tools expand this work by introducing a specific process for linking success with relevant assessment features.

This study also builds on work that explores how the constraints and affordances of library makerspaces impact assessment use and development (Gahagan & Calvert, 2020; Teasdale, 2020, 2021), including our own previous research (Cun et al., 2019; Saplan et al., 2022a; Wardrip et al., 2019; White et al., 2022), by offering tools that uniformly identify the breadth of success library staff and maker educators consider in the design of maker-based programs and align potential assessment features with those conceptions of success. Whatsmore, this article adds to the limited body of literature focused on understanding success for assessment design in library makerspaces (Teasdale, 2020, 2021); we not only continue to advance the value of linking success with assessment, but we also provide hands-on tools that help others connect success with assessment in their own settings.

Finally, while these tools are intended to connect practitioners’ expertise related to success in their makerspaces with meaningful assessment, these tools also offer evidence-based artifacts that address pervasive misconceptions about assessment. For instance, the common assumption that assessment must draw on unvaried, comparable, and quantitative data from each learner to be considered rigorous is problematized by the inherent recognition in the PSA Tools that qualitative data is a valid option for assessment depending on one’s conception of success. Additionally, the PSA Tools encourage library makerspace stakeholders to take a more expansive view on what assessment tools can do and look like; rather than pointing them to variations on a survey, the PSA Tools illustrate the power of immediate observations and long-term data collected over time, reinforcing the value of qualitative-based, naturalistic assessment practices like making observations, asking questions to patrons, and listening to interactions between and among patrons (Michalchik & Gallagher, 2010). We urge external stakeholders of library makerspaces (e.g., Executive Directors, Grant Funders, and Library Board Members) to explore these tools as well and reflect on their own expectations about how experiences in makerspaces must be measured, and we encourage all users to think outside the box (or bubble) of the standard exit survey.

Although the primary goal of this article is to support library makerspaces and similar informal environments, we believe this analysis tool could also be valuable in other learning settings. Because we chose to analyze the underlying components of definitions of success and their connection to assessment, we recognize the potential benefit it has for educators across a range of makerspaces, educators in other informal learning environments like youth arts organizations (Saplan et al., 2022a), and prospective classroom teachers who are asked to center their desired learning outcomes and evidence of those outcomes in their lesson plans. Additionally, we see an opportunity for researchers to use this tool with partner organizations in assessment development projects, or simply in co-designing educational curricula and environments; the PSA Tools can be useful for collaborators who want to build a shared understanding of success in their program or organization and connect it to potential assessment approaches.

Limitations and future research

Our central argument in this paper is that developing relevant assessments for library-based makerspaces requires one to thoroughly understand the conception of success one is striving to achieve, and we offer a series of analysis tools to help stakeholders do that. Although the study that was the basis for these analysis tools drew from a small sample size, our collective expertise partnering with libraries and our familiarity with literature on maker education enabled us to develop a series of analysis tools that have implications for a variety of makerspaces as well as other informal and formal learning environments.

A major limit of these tools is that they have not undergone extensive testing, which will be the next step in our process. As we continue to co-design assessments with various organizations and stakeholders, we intend to use these analysis tools to build a common understanding of success and assessment, and to direct our design processes. We intend to update these tools as we learn how educators use them and expand on them as necessary. We encourage educators and scholars to test these tools to help us refine and develop them as well.

Additionally, we recognize that while the PSA Tools can support educators in connecting their definitions of success to potential assessment tools, these tools are not designed to help educators analyze the information they collect with assessments nor indicate how to use assessment data to inform their practice. This is another critical piece in terms of developing educators’ assessment literacy that we intend to address in future research.

Conclusion

We initially approached this project with a singular focus to develop a new assessment tool for library staff and maker educators in library makerspaces. The onset of COVID-19 forced us to adapt our plan, but not before we began domain analysis of Learn to Make Lab and Making an Impact. In our study, we explored how 17 library staff and maker educators’ define success and identify evidence of success in their maker programs. Our analysis led us to refocus this work on developing the Properties of Success Analysis Tools that can help various stakeholders in diverse learning environments break down their definitions of success to identify what an ideal assessment tool would do to capture that success. The PSA Tools include five properties of success: Subject of Success, Performance of Success, Timescale for Success, Data for Success, and Audience for Success. These five properties are the components that make up a definition of success. We argue that by using the PSA Tools to clarify each of these properties and link them to meaningful assessment foci, then one is better equipped to select or design a meaningful, and relevant assessment.