A Knowledge Exchange PLAYBOOK to Build Resilience Edited by Margaret M. Hinrichs and Patricia Solís A Knowledge Exchange PLAYBOOK to Build Resilience Edited by Margaret M. Hinrichs and Patricia Solís How to cite this playbook: Hinrichs, Margaret M. and Patricia Solís (Editors). (2021). A Knowledge Exchange Playbook to Build Resilience. Tempe: Knowledge Exchange for Resilience, Arizona State University. Washington, D.C.: Global Council for Science and the Environment. Available online at https://resilience.asu.edu/playbook Table of Contents Contributors 6 Introduction 7 How to Use This Playbook 10 Plays 01 Choosing the Right Geographic Scale 14 02 Identifying Community Partners 18 03 Building the Stakeholder Network 22 04 Determining Readiness to Collaborate 26 05 Creating Data Analytics and Visualizations for Shared Understanding 28 06 Managing the Inclusion Process to Optimize Stakeholder Commitment 32 and Satisfaction 07 Leading to a Productive Conclusion 36 Tools 01 Humility Audit: Conducting a Critical Self-Assessment Before Doing 44 Community-Engaged Work 02 PQRS Approach to Science Communication 46 03 Finding Connections with Collaboration Information Systems 50 04 Project Planning Playboard 54 05 Project Readiness Checklist 60 The Game in Motion 65 Glossary 69 Additional Resources 80 5 Contributors Joffa Applegate, PhD, Assistant Research Professor, School of Complex Adaptive Systems, ASU Shauna Burnsilver, PhD, Associate Professor, School of Human Evolution and Social Change; Convener, Governance and Decision-Making Theme, Julie Ann Wrigley Global Futures Laboratory, ASU Erica Goldman, PhD, Deputy Director, Global Council for Science and the Environment Margaret M. Hinrichs, PhD, Assistant Research Professor, School of Complex Adaptive Systems, ASU Erik W. Johnston, PhD, Professor, School for the Future of Innovation in Society and School of Complex Adaptive Systems; Director of Policy Informatics at the Decision Theater, ASU Thaddeus R. Miller, PhD, Associate Professor, School of Public Policy, UMass Amherst David Morrison, Graduate Research Assistant, School for the Future of Innovation in Society, ASU Marcia Nation, PhD, Principal and Owner of Nation Evaluation Consulting, LLC Christina Ngo, Director of Social Embeddedness, Office of the University Provost, ASU Shade T. Shutters, PhD, Research Scientist, School of Complex Adaptive Systems, ASU Patricia Solís, PhD, Executive Director of the Knowledge Exchange for Resilience; Associate Research Professor, School of Geographical Sciences and Urban Planning, ASU Elizabeth Wentz, PhD, Vice Provost and Dean of the Graduate College; Director of the Knowledge Exchange for Resilience, ASU Michelle Wyman, Executive Director, Global Council for Science and the Environment 6 Introduction By Elizabeth A. Wentz, Michelle Wyman, Patricia Solís, and Erica Goldman In the face of profound shock and change, individuals, organizations, and communities are seeking new ways to prepare for an uncertain future, their only certainty being that the present trajectory of change will intensify. Pandemics, wildfires, heat waves, hurricanes, flooding, social unrest, economic strife, and a rapidly changing climate system comprise a resounding wake up call: we must reinvent our institutions to think about and act with a resilient mindset. The purpose of the playbook is to support these efforts and build stronger, adaptive, and resilient communities. We define community resilience as the capacity for communities, institutions, and individuals to respond and adapt to shocks and long-term stresses. The approach taken here is to use the assets of the university to build partnerships across sectors, to use data and analytics to identify gaps and vulnerabilities, and to catalyze policy and process changes. The university acts as the anchor institution to catalyze these efforts using resources such as our research capacity and the opportunities created for students. University - community multi-sector partnerships serve as an engine towards the necessary transformations, leveraging the knowledge and know-how to identify vulnerabilities, assets and current response mechanisms proactively. Together, we can collect, analyze, and visualize data, creating, communicating, and mobilizing knowledge through a multi-sector network of collaborators for greater resilience. Such alliances support strategic investments and responses to the shocks and stresses we face, and promote or directly allocate human and financial resources for systemic/systematic impact and transformation. This can only be accomplished using multiple forms of knowledge that we co-create. And it must happen through a concerted exchange of human and institutional resources. We offer this playbook as a compendium of some of the ideas and techniques we have used as a means to scaling up the impact of this work. This compilation features contributions about what we do, how we do it, and why it matters. It is envisioned as a starting point for others to add supporting material and additional resources in a digital living repository. Why a Playbook? The term originates in sports, referring to a book with all the “plays” or game strategies that a team learns and practices. Each play has a name, so the coach or captain can easily shout the name of a play during a game and players will immediately know what to do. Furthermore, existing plays can be adapted and new plays can be added to the playbook. For our purposes, the idea of presenting our experiences in the form of a playbook allows us to articulate and share a set of discrete strategies and activities that can be easily adapted to different organizations and place-based conditions. We consider all those who work toward positive community resilience to be on the same team, working together toward a more resilient and inclusive future in the face of uncertainty and change. 1 The plays we offer here do not explicitly require a university to lead but rather any anchor institution with the capacity to convene multi-sector organizations, foster these partnerships, and compile and curate data and information. 7 Introduction Who is the Audience for this Playbook? This playbook is intended for university faculty, researchers, staff, and students, as well as local governments, nonprofits, private sector entities and all stakeholders who seek a “knowledge exchange” as a way to build community resilience. The playbook seeks to give new inspiration on how to share, discover, and meet resilience challenges in collaborative settings. How Can You Engage? Users can choose from among the “plays” that are described herein, and put together their own game plan that works for each specific context. Rather than recommending the replication of our exact experience as a model to be emulated, we aim to transfer insights on how a communitywide knowledge exchange could be built. Plays can be adapted, combined, improved upon, and implemented case-by-case through convening, interaction, seeding ideas, data sharing, convergence — in each unique game context. We aim for you to think of this as your guide for finding and contributing plays that build community resilience. When we purposely design for resilience, we aim to take into account factors that help us not only solve the problem in front of us, but also address it in a way that is mutually beneficial for everyone involved. This requires a novel design logic, which forms the basis of this playbook. We chose to highlight plays that can best apply to many other institutions in diverse collaborative contexts. With more than 1,700 universities in the United States and an estimated 28,000 worldwide, we perceive an underutilized opportunity to engage the university-based scientific enterprise to inform, support and fortify community resilience. At this moment in time, accentuated by the devastating impact of the COVID-19 pandemic, stronger collaboration between the university knowledge enterprise and decision-makers can lead to mutual benefits, creating a durable feedback loop that results in making cutting-edge science available for evidence-informed decisions that advance the resilience of communities in a sustained manner. The outcomes that we believe these plays could achieve, and what our own efforts aim to produce include: Institutional Transformation: Transform universities’ capacity to more deeply engage their communities through data-driven resilience knowledge and collaborative research of public value. Intellectual Leadership: Establish the participants of the knowledge exchange as intellectual leaders for community resilience within the various spheres of influence locally, nationally, and even internationally. Community Impact: Catalyze community level systems change by enhancing exchanges of data, knowledge, and actors around fundamental community resilience themes across public, private, non-profit and academic sectors. 8 Introduction Who Are We? This playbook is brought to you by the Knowledge Exchange for Resilience (KER) at Arizona State University (ASU) in collaboration with the Global Council for Science and the Environment (GCSE). The mission of ASU’s KER is to support the growth and adaptive capacity of organizations — including ASU, nonprofits, government agencies, and private companies — in Maricopa County, Arizona, to be resilient to long-term stresses and short-term shocks. The GCSE is a nongovernmental organization established in 1990 to advance and improve the use of science to inform environmental decision-making. In 2018, ASU designed and implemented KER, with a gift from a local philanthropy, The Virginia G. Piper Charitable Trust. We were and continue to be motivated by the ASU Charter to produce research of public value, and to define our success not by whom we exclude but by whom we include. Initially focused on community resilience in Maricopa County, we expanded our vision to learn from and share experiences around the country and around the world. ASU KER employs two strategies to build community resilience: 1) convening influential leaders and core professionals within the community who will invest time, energy, and resources to understand current response mechanisms and hidden vulnerabilities; and 2) collecting and sharing data, analytics, models, and visualizations that structure and organize conversations around responses and solutions. Together these strategies trigger research projects, promote new solutions, mobilize action, lead to new funding, and support organizations’ responses to new experiences (like shocks). For Maricopa County, the long-term concerns we address are the increasing summertime temperatures coupled with population growth which is accelerating water scarcity in an arid region, among other impacts; the shortterm shocks are acute heat waves, economic shocks, and health crises. For example, in response to the COVID-19 pandemic, KER provided human and information resources to organizations so they could pivot their services in response to the pandemic. KER teamed up with GCSE by joining the inaugural cohort of universities in the Learning Collaborative for Resilience and embracing the mission of the GCSE to improve the scientific basis of environmental decision-making. Formerly the National Council for Science and the Environment (NCSE), GCSE engages scientists, educators, policymakers, business leaders, and officials at all levels of government. GCSE leads a targeted initiative, the GCSE Learning Collaborative for Resilience, focused on engaging the university-based scientific enterprise to enhance community resilience, aligned with its mission to span the boundaries between science and decision-making. GCSE is poised to amplify and build upon the foundational work of ASU KER by bringing together diverse universities from across the network of GCSE Members, with local governments and other stakeholders.. As collaborators on the playbook, GCSE intends to bring additional members and case studies to this compilation, and welcomes your contribution. As you begin . . . We hope that you will find this playbook useful to build your own “plays” informed by knowledge exchange for resilience, one that can be a part of a design transformation across your community. We also hope that this allows communities to better access the institutional public resources of academia. It is our aim to inspire further conversations about practical and winning ways to put in place what we must do to be better prepared for shocks and stresses – to better withstand and confront future pandemics, climate change impacts, and the socio-economic infrastructure that builds community resilience. Game on! 9 How to Use This Playbook This playbook has grown out of lessons learned during the first phase of the Knowledge Exchange for Resilience and provides procedural knowledge on a number of plays and tools designed to support collaborative, community-engaged projects. For our purposes, a “play” is a set of processes which comprise a strategic activity designed to initiate and/or facilitate knowledge exchange between diverse actors. A “tool” is a specific instrument which supports one or more processes in a play. The plays and tools included here have been contributed by affiliated faculty, Cross-Cutting Scholars and partners of the Knowledge Exchange for Resilience and are by no means meant to be exhaustive or static. As the Knowledge Exchange continues to grow its collaboration with communities and refine its own best practices for constructive knowledge convergence, we anticipate that the plays and tools included herein will evolve and mature as well. In the figure below you will see that we have organized the plays and tools according to four general stages of project management - initiation, planning, execution and closure. Initiation Phase During this phase, an initiating team of faculty and/or community members jointly identifies a compelling need through a shared narrative and project scope. The shared narrative consists of a focal community resilience problem, vision, theory of change, and need which forms an essential step in the theory of change, while the project scope consists of project objectives that fulfill the need as well as project activities circumscribed by the expectations, resources, individual goals of the members of the core collaborative team.The initiating team gradually builds its capacity and begins to recruit potential collaborators, ultimately forming a core collaborative team that commits its time, energy and resources to planning and executing a community resilience project. Planning Phase During this phase, the core collaborative team assesses the readiness of the collaboration to address the compelling need, gathers the resources necessary to execute the project, enrolls and manages the inclusion of critical community partners, and creates stable collaborative processes and expectations for the execution phase. Execution Phase During this phase, the core collaborative team and critical community partners execute the planned project activities and create deliverables, thereby fulfilling the compelling need and building community resilience. Their collaborative processes and decision-making become more efficient and effective, owing to experience working together as well as clearly described rules and expectations governing their work together. 10 How to Use this Playbook Closure Phase In this phase, the core collaborative team measures and analyzes project outcomes as identified in earlier phases and reports them to the broader team and others involved in or interested in the project. The collaboration either ends gracefully, or future collaborative projects and next steps are planned, which may or may not include all of the original team members. INITIATION PLANNING EXECUTION CLOSURE 01 02 03 04 05 06 07 01 02 03 04 05 The plays and tools included here are not mutually exclusive. As the figure suggests, the majority of the plays and tools included in this playbook support the Initiation and Planning stages of a project, with some reaching into Execution. This is because we are creating and adapting the playbook from our own lessons learned within the Knowledge Exchange for Resilience, and as the first phase is coming to a close many of our projects are still in one of the first three stages listed above. We discuss Transforming Collaboration in the chapter on “Ending the Game.” This playbook is meant to support knowledge exchange processes. In the same way a coach using a playbook doesn’t necessarily win every game, some of the plays and tools included here may or may not be successful in different contexts. Each play and tool includes a brief summary table to orient readers as well as a definition, intended timing in the collaborative process, applicable conditions, procedure, evaluative criteria, related plays and tools elsewhere in the playbook, anecdotal or published examples, and additional resources. We acknowledge that there are many interpretations of concepts like resilience and equity. This playbook also includes a Glossary of terms to clarify our use of these concepts and their relation to the plays and tools included here. 11 PLAYS 01 Choosing the Right Geographic Scale What It Is and Why It Matters A shared understanding of the project objectives and scope can make or break a community resilience collaboration, and misunderstandings are commonplace. Defining an appropriate geographic scale (e.g., neighborhood, city, or state) helps focus your project and get collaborators on the same page. This, in turn, has many other benefits for the project, including: more efficient use of resources; easier stakeholder identification; better management of expectations and risk; streamlining decision making; and, ultimately, a greater likelihood of achieving the desired impact. Keep Reading If... If you are doing place-based collaborative work that is intended to have real-world impact, you must jointly specify the geographic scale the project is tailored toward. Definition and Purpose For projects addressing complex resilience challenges, an essential step toward successful collaboration is agreement on a project’s objectives and scope. To achieve this, choosing an appropriate geographic scale (“scaling” your project) — whether it is at the level of a neighborhood, city, state, or other geographic boundary — makes projects more tangible and, in turn, results in a focused project scope that is shared and understood by project collaborators. Furthermore, it is a myth that projects can run a pilot which is then scaled up — this rarely occurs. We believe many pilot projects fail because they are designed for a small-scale approach, but discover things work differently at larger scales. Collaborators should design for the geographic scale at which they want to have an impact. There are many choices, too. Larger scale is not somehow better than smaller scale or vice versa, where sometimes an intermediate scale might be a better fit. For example, urban climate research often experiences both spatial and temporal issues of incongruence. Spatially, area-based attributes — like using the size, shape, and average temperature within a 7m grid — may be insufficient for point-based (human-or touch-scale) temperature exposure. The concern arises when using the research to link science with policy and health. The next step for moving from research to practice is to provide the new evidence base to park or urban designers, as well as city officials and urban planners, with suggested actions to take and benefits of the actions. This includes raising awareness of the spatial incongruence in their decision process. You can transfer findings from data at different geographic scales, but problems arise when the attempt is made to scale up (or down) a single project to a new scale. Project design must match the scale of the original challenge. Making your project tangible and focused through defining the geographic scale results in numerous benefits for the quality of the collaborative process. 14 Choosing the Right Geographic Scale Benefits of clear geographic scale include: 1. Share the goals, expectations and constraints of team members. • Less conflict among collaborators • Enhanced coordination • Efficient use of energy and resources • Goals, such as publications or community impact; • Greater ease identifying appropriate stakeholders and actors; • Expectations (e.g. of other partners in the collaboration); • Clearer definitions of resilience • Assumptions (e.g., whose role is what), and • Streamlined, perhaps even more impactful, decision making. • Constraints, such as time, resources, and availability for the project. On an individual level, collaborators must share their: As a consequence of better upstream processes, more alignment, and a better project design, the project is more Transparency is crucial, especially about constraints. likely to fulfill project objectives downstream. 2. Discuss the impact you want the project to have. Time and Timing Is the team trying to influence policy? The state scale may be appropriate. Is the team trying to reduce heat burden on households? A smaller scale may be more appropriate. What is going to happen with the results after the collaboration? If you want to scale up, you will need to consider what that exactly means, what is the end-scale, and the implications for designing the project at hand. Geographic scaling of your project must take place early during the initiation phase of a potential collaboration, before detailed project planning. The time it takes depends on the project. Some projects may already have a predetermined scale (e.g., in the call for proposals), in which case it is a short discussion. Other projects may require deliberation among potential collaborators to agree upon a scale. 3. Conduct a team-level SWOT analysis. Applicable Conditions If you are doing collaborative, place-based work that is intended to have real-world impact, this procedure is indispensable for creating a sharply-defined, collective understanding of the project scope. Any collaborative project that fails to create a shared understanding of the project scope runs the risk of inefficiency, dysfunction, and conflict as members pulling different directions without understanding why. The strengths, weaknesses, opportunities, and threats at the team-level also have bearing on the project scale and scope. Your team may desire to do work at one scale, but have better resources, relationships, and opportunities to make an impact at another scale. Procedure Where is the biggest impact going to be? How do different scales meet the goals and expectations of collaborators? Is the proposed scale attainable within the constraints, strengths, and weaknesses of the project team? Is there a willingness among those involved to take commit to the proposed project? Scaling your project entails deliberation, decision-making, and conflict management; all collaborators should be involved. Deliberation. Deliberation on a project scale is a nonlinear and iterative process. However, the following topics may be discussed in roughly the following order. 4. List and discuss the pros and cons of different project scales. Sometimes “scale” can be congruent, or it can be a set of smaller locations, aggregated to summarize findings across a broader scale. All of these require careful attention to how to generalize the research and an ethic toward the community who are associated with that scale. 15 Choosing the Right Geographic Scale Making the final decision. The way that final decisions are made depends on the power dynamics in the team of collaborators or potential collaborators. If top-down authority is held by a principal investigator over a team that is compelled to participate in the project, the final decision on the geographic scale can be made by the PI, who would need to mitigate conflict thereafter. In cases where an initiating team is convening a group of potential collaborators whose participation is voluntary but necessary for success, consensus may be a better option, as this may be likelier to maximize the value of the project for all involved. See the Resources section for more options for group decision-making processes. Conflict management. Conflict management is also needed to develop consensus around the scale of the project. This involves understanding where people are coming from and a commitment to working through these issues. Ultimately there should be enough consensus to move forward. More details are in the Resources section. Evaluation You have succeeded in scaling your project when: • Everyone has consensus around the geographic boundaries of the effort • Everyone has shared sense of time frame • Project collaborators can name who you are focusing on, where you are focusing, and what the area of action is • You can “place” the decision makers you want to inform • You can title your project You will know if you have failed to scale your project if there is fracturing in the project team — people go off and splinter, taking on their own projects. When this happens, it is best to get everyone involved to agree that it is necessary to fork off. Blaming the scale can be helpful in this conversation. In other words, failure of the collaboration wasn’t a result of not wanting to work together, it was also a disagreement on spatial or time scales. The project team can then splinter, work on different aspects of the project, share results, and inform one another. Related Tools and Plays Community Resilience Project Planning Playboard Examples Solís, Patricia, Jenni Vanos and Robert Forbis. 2017. The Decision-making / Accountability Spatial Incongruence Challenge for Research linking Science and Policy. The Geographical Review 107(4): 680-704. doi: 10.1111/ gere.12240. 16 Choosing the Right Geographic Scale Resources • Benefits of clear geographic scale include: • Less conflict among collaborators Action Priority Matrix (www.mindtools.com). These simple • Enhanced coordination diagrams help you choose the activities you should prioritize • Efficient use of energy and resources and the ones you should avoid, if you want to make the most of • Greater ease identifying appropriate your time and opportunities. A Manager’s Guide to Resolving Conflict in Collaborative Networks (maxwell.syr.edu). Written by two scholars of public administration and service, Rosemary O’Leary and stakeholders and actors; • Clearer definitions of resilience • Streamlined, perhaps even more impactful, decision making. Lisa Blomgren Bingham, this is a practical, thorough, and overall excellent guide to managing and resolving conflict in collaborative networks. Ethical Decisionmaking Protocol (worldofwork.io). The foursquare protocol is a four stage process. It helps individuals and leaders make ethical decisions in the workplace. The stages are: gather facts, understand how previous ethical decisions were reached, look for similarities to previous ethical situations and assess self-interest or bias. Find your Elected Decisionmaker Map (myreps. datamade.us). This is a tool for looking up who your elected representatives are across the United States. Based on your address, we can find all the federal, state, county and local officials who represent you in government. Group Decision-Making Practices (uwaterloo.ca/centrefor-teaching-excellence). There are a variety of ways to make decisions as a group; this seven-step decision-making model offers an effective structure for choosing an appropriate course of action for a particular task or project. It can also be an effective method for dealing with a challenge or interpersonal conflict that arises within the group. SWOT Analysis (www.mindtools.com). SWOT stands for Strengths, Weaknesses, Opportunities, and Threats. A SWOT Analysis is a technique for assessing these four aspects of your project. 17 02 Identifying Community Partners What It Is and Why It Matters Though there are many community stakeholders that could be partners on a project, a key part of the process of creating a project with a community is to determine which additional partners are critical for the success of the project. This play describes strategies for identifying critical community partners for collaborative projects, yielding a more efficient and effective use of the core collaborative team’s resources. Keep Reading If... If you are working on a collaborative, community-engaged resilience project where more than one community partner is necessary to address a challenge or question under consideration, this play is essential for efficient, effective stakeholder management. Definition and Purpose In general, there are a lot of options for selecting community stakeholders to partner with in a collaborative project. Part of the process of creating a project with a community is to determine out of all of those possibilities which partners are critical for success. The determination of what is “critical” may vary slightly depending on the compelling need or challenge determined by the primary stakeholder or community partner. In our experience, critical partners: • • • Have relevant data and are willing to share it. Have available time, energy, and resources they are willing to commit to the collaboration. Have access to legislative bodies whose influence is necessary for project success. • • Can amplify your impact through media or other communicative channels Enable you to overcome an obstacle to the workflow of the project that cannot be overcome with any other entity The criticality of a community partner often extends beyond the existence of a relationship and into a commitment of more tangible assets or influence on behalf of the project. Identifying and including critical community partners can greatly minimize the risk of project failure. Doing so ensures that you are investing in a project with partners that have the assets — data, time, connections — needed for the project to succeed. Projects which successfully identify critical community partners are more likely to meet the determined requirements for the project to succeed, as well as form better matches between what the project needs and what partners can provide. 18 Identifying Community Partners Time and Timing • Identifying critical community partners occurs after the core collaborative team — often researchers and the primary stakeholder — have identified a compelling need or question in a community. Once the question or need is identified, the collaborative team will map out a research plan which includes data, resource, and implementation plan needs. Critical community partners are then identified in regard to meeting these needs. Answering these questions should help your team identify which partners are critical on the path to success for your project or study. The length of the identification process depends on the level of existing experience and embeddedness of the core collaborative team among critical partners of interest. If the team is new to a particular community, style of research, or legislative issue it might take several weeks to arrange introductions, conduct background research, and meet with potential partners to determine criticality and fit. If the team has already invested in the relationships necessary to execute a project with the identified critical community partners, it may take no more than an intensive strategic planning session to map out data and access needs required from a community partner, as well as a recruitment strategy for bringing them to the project. Applicable Conditions Because of the nature of community-engaged collaborative research projects, the identification of critical community partners is almost always necessary. At the outset of a project, the core collaborative team almost never has the resources, data, influence, or other assets necessary to realize the project plan and goals — the core collaborative team must use its own scarce resources to identify and manage critical community partners who can fulfill project needs. However, there may be instances where a collaborative agreement exists between one primary stakeholders and a research team which is meant to address a specific challenge or issue for a specific partner. Due to the scope and scale of such a project, the identification and inclusion of critical partners external to the primary stakeholders is not deemed necessary. Which agencies or stakeholders in the community will enable us to meet those requirements and achieve our goals? To assist with this process, create a matrix which identifies your data needs, agencies which may have that data, and which contacts your team has within the agency, if any. Then, start emailing or calling contacts on the list and ask if they have recommendations for where else you can find relevant data or suggestions for other well-connected community partners. Keep in mind that, due to the nature of your project, you will inherently favor collaboration with some organizations and agencies over others because they are deemed more critical to meeting your project requirements. Stakeholders who are not deemed critical may feel left out of the process or less valued by your team. Such partners can still enhance the overall impact or help satisfy another goal of your project, but they might not have anything to offer along the critical path toward success. You will need to prepare a diplomatic process to reassure less critical community partners that they are still valued by your team, and keep them engaged and invested in project outcomes. Procedure Once the research team and primary stakeholder have identified a compelling need or question in a community, sit down with your team and decide on the plan of research or engagement. Ask yourselves: • What are critical requirements to meeting our compelling need or answering our question? 19 Identifying Community Partners Evaluation If you are successful identifying critical community partners, you will find the planned project activities progress relatively smoothly as partners willfully commit their resources to fulfill project needs. Over the course of a project you may discover that you failed to identify a critical partner. The project may fail to meet milestones, or obstacles which were supposed to be eliminated by engaging with a particular partner have not been avoided. In these cases, your strategy may have to change, and time, energy, and resources might need to be reallocated from one critical partner to another. Doing so requires a transparent conversation with the existing critical partner which emphasizes the goals and needs of the project, strategic project management, application of project resources to agreed-upon goals and milestones, and preservation of the interpersonal relationship between collaborators. Resources Creating and Maintaining Partnerships (ctb.ku.edu). This toolkit provides guidance for creating a partnership among different organizations to address a common goal. Healthy Food Playbook (foodcommunitybenefit.noharm. org). This resource helps users identify potential community partners if they are undertaking a food initiative, but it may still spark some ideas for partnerships even if your project doesn’t concern food. A community partner might be “critical” if they: Related Tools and Plays • Collaboration Information System • Managing the Inclusion Process • Have relevant, shareable data • Grant access to legislative bodies • Amplify project impact through diverse media • Help you overcome key workflow obstacles • Are willing to commit time, energy and resources Examples de Nisco, A., Riviezzo, A. and Rosaria Napolitano, M. (2008), The role of stakeholders in town centre management: guidelines for identification and analysis. Journal of Place Management and Development,1(2),166-176. https://doi. org/10.1108/17538330810889998 You will need to prepare a diplomatic process to reassure less critical community Bryson, J. (2004) What to do when Stakeholders matter, Public Management Review, 6(1), 21-53, doi: 10.1080/14719030410001675722 Cummings JL, & Doh JP. (2000). Identifying Who Matters: Mapping Key Players in Multiple Environments. California Management Review, 42(2):83-104. doi:10.2307/41166034 20 partners that they are still valued by your team, and keep them engaged and invested in project outcomes. Game Plan Notes 03 Building the Stakeholder Network What It Is and Why It Matters Resilience challenges are complex and the stakes can be high. With inaction, significant harm could occur, because: (1) no single individual or organization is likely to solve challenges alone, and (2) many people have a shared interest or concern in the challenges and their solutions. Convening a group of stakeholders and fashioning them into a collaborative network engaging around a shared challenge deepens the pool of knowledge, resources, and capacities available to solve any challenge and also builds buy-in, good will, and trust necessary to implement the solution. This play shows you how to build your stakeholder network in a goal-driven manner Keep Reading If... If you are working on a complex and significantly harmful challenge where many stakeholders stand to gain or lose from a solution, this play may help yield outcomes that are more effective, equitable, and legitimate. Definition and Purpose Resilience challenges are complex and, left alone, are potentially harmful across a range of groups and landscapes. This suggests that no single individual or organization is likely to solve a resilience challenge on their own and there are a wide range of actors — stakeholders — who have a common interest or concern in the challenge and its solution. This is where the idea of building a stakeholder network is important. Convening a stakeholder network around a challenge deepens the pool of knowledge, perspectives, resources, and capacities available to analyse, understand, articulate tradeoffs and generate strategies to solve the challenge. The premise of diversity is built into this network - resilience challenges always involve conflicts and tensions - and a stakeholder network is a venue to (potentially) work through these challenges, based on repeated interactions and engagement. On the far end of the process, having a strong stakeholder network with links to multiple groups who are affected by the challenge also builds the buy-in, good will, and trust necessary to navigate conflicts and implement any solution. Building a stakeholder network is not a random process — it is important to be deliberate and strategic given project goals, and then identify whose perspective and expertise is necessary to achieve those goals. Doing so increases the potential for better solutions to emerge and be implemented. 22 Building the Stakeholder Network Time and Timing 2. Go through a nominating process. Building a stakeholder network should begin after the initiating team has drafted a preliminary project narrative and scope, as stakeholders are good candidates for the core collaborative team or they may play another critical role in the project. The bulk of the work should be done in the initiation and planning stages, but you may find yourself building your stakeholder network even through project execution as circumstances evolve. There is no set amount of time needed. The process is iterative, and its speed is dictated by the trust, which may take longer to build depending on existing trust or mistrust between the team members and stakeholders (for example, between government agencies, researchers, and underserved communities). Many effective stakeholder networks start small and grow strategically. Make sure nominees are fundamentally aware of the goals and needs of the project. Consider a diversity of stakeholders, including government, business, NGOs, and a range of community leaders, and so on. Nominees may already be known to the core project team or they could self-nominate through word of mouth. Snowballing is an effective tool for the nomination process. Have one-on-one conversations with identified stakeholders and identify from the beginning who have similar (and dissimilar) views on the resilience problem or challenge. Interpersonal dynamics are important to consider in building the network - who can work together, who cannot? Applicable Conditions 3. Convene an initial meeting of the group. If you are not working on a complex challenge that significantly affects a wide range of stakeholders, forming a stakeholder network may be unnecessary. However, resilience work usually implies balancing multiple perspectives. A stakeholder network by definition is a method to engage across these viewpoints and bring together needed expertise. Though the development of a stakeholder network should be strategic and deliberate, it can also be opportunistic. Taking advantage of spontaneous relationship which may be outside the bounds of a preconceived network development plan is encouraged. Listen as they describe their goals, interests, and level of commitment around the challenge. As a group, create and set ground-rules for communication among members. Agree on a timetable for tasks and meetings. What strategies prior to and during meetings could equalize power imbalances among group members? For example, virtual meetings could rotate facilitators. Physical meetings could rotate locations and contexts (e.g. sit down, walking-talking-reporting out). Potential leaders will emerge in this process; as well as individuals who can play other roles within the network, for example connectors, devil’s advocates or consensus builders. Procedure 1. Think about the shape and breadth of the resilience challenge. The first step to building a stakeholder network is to think about the shape and breadth of the resilience challenge. Who is being affected by this challenge? In what ways? Who stands to gain or lose from potential solutions? These questions help you identify stakeholders. What are the power dynamics characterizing the situation, specifically what voices are less powerful, so should be elevated through a process of stakeholder engagement? Who may stand in the way, or who is a key champion? These questions help you prioritize stakeholders. In the research team, what are the gaps in knowledge, skills, resources, and other needs required to achieve project goals? Who can fill those gaps? From these questions, determine the “shape” of the network. 23 Building the Stakeholder Network Evaluation Defining Collaboration and Explicating the Collaborative If you have executed this play successfully, the collaboration will be characterized by a network of stakeholders that over time collectively meets the needs (e.g., knowledge, skills, resources, political capital, legitimacy) required to fulfill the goals of the project. Partnerships and connections established during collaborative efforts can sometimes become downstream metrics which are several steps removed from an initial interaction or network development, spinning off onto new projects of their own. Process (bcpsqc.ca). This document sets out six empirically-demonstrated features that make for a successful collaboration. The features can be used as evaluative criteria to assess the health of the collaboration as well as design principles for collaborative processes. Collaboration Success Wizard (sc.ctsi.org). The Collaboration Success Wizard (CSW) is a 30-minute survey which generates individual and project-level reports diagnosing strengths, potential challenges, and coping Frequency of meetings and positive communication between stakeholders are not signs of success or efficacy in and of themselves - this is a common misconception about stakeholder networks and collaboration more broadly. More connections, either through larger groups or frequent meetings, do not automatically equate to better outcomes. A stakeholder network can interact frequently, and yet make little progress toward achieving progress on partnership goals. Social network analysis or a stakeholder network ethnographic approach are methods that can be employed to evaluate the relationships between network members in terms of project outcomes. strategies for geographically-distributed collaborations, using the latest findings from collaboration science. Creating and Maintaining Coalitions and Partnerships (ctb.ku.edu). This detailed and comprehensive toolkit provides guidance for creating a partnership among different organizations to address a common goal. Involving People Most Affected by the Problem (ctb. ku.edu). Learn how to reach out to and meaningfully engage community members with firsthand experience with the problem or issue at hand. Measuring while you manage: Planning, monitoring, and evaluating knowledge networks (iisd.org). This paper details the strengths and weaknesses of common Related Tools and Plays measurement and evaluation frameworks (SWOT analysis, Results Based Management, Logical Framework Analysis, Outcome Mapping, Appreciative Inquiry and human resource • Managing the Inclusion Process • Identifying Critical Community Partners performance assessment) and proposes a simple network evaluation model which incorporates the best of each. This new framework focuses on the key questions of network effectiveness and efficiency. Social Network Analysis (betterevaluation.org). This guide is a quick, one page overview of Social Network Analysis Resources (SNA). It offers an outline of the steps involved in using this technique, a summary of some of the benefits of the technique and what it might be used for, some practical tips for using it, Defining Collaboration and Explicating the Collaborative Process (bcpsqc.ca). This document sets out six empirically-demonstrated features that make for a successful collaboration. The features can be used as evaluative criteria to assess the health of the collaboration as well as design principles for collaborative processes. 24 and lists further resources. Game Plan Notes 04 Determine Readiness to Collaborate What It Is and Why It Matters Determining readiness requires potential partners to think beyond whether they simply want to collaborate and whether they are ready to engage in the level of collaboration necessary for a healthy, mutually beneficial exchange of ideas, resources, data, and knowledge. Set expectations for self and prospective collaborators, reducing future conflict and confusion regarding roles, responsibilities, and resource management. Keep Reading If... Determining readiness to collaborate is an essential part of any collaboration. Keep reading if you have jointly identified a compelling need and shared narrative for your collaboration and are ready to consider the practical aspects of your work together, before officially committing. Definition and Purpose When forming a collaborative partnership or building a knowledge exchange between university and community actors, one of the most significant steps to take — which will form the basis for the success of the resultant relationship — is for all actors involved in the collaboration to determine their readiness to collaborate. Determining readiness requires potential partners to think beyond whether they simply want to collaborate, instead making them also consider whether they are ready to engage in the level of collaboration necessary for a healthy, mutually beneficial exchange of ideas, resources, data, and knowledge. Establishing the desire to engage in knowledge exchange and convergence of expertise is important, but determining readiness encompasses many other practical elements. One of the primary benefits of determining readiness to collaborate is that it helps a potential partner communicate not only their state of readiness, but the level of contribution they can be expected to make within a project. Being able to communicate this kind of information helps set expectations for other prospective collaborators, reducing future conflict and confusion regarding roles, responsibilities, and resource management. Time and Timing Determining readiness to collaborate should be done at the outset of a potential partnership. Depending on the familiarity of partners with project goals and objectives, as well as expectations regarding each partner’s role to play in the upcoming project, this play may take more or less time. The critical self-reflection may take no more than an afternoon, but depending on the state of relationships the communication of and agreement upon each partner’s expected resource contributions and management may take several meetings. 26 Determine Readiness to Collaborate Applicable Conditions Determining readiness to collaborate should be done at the outset of a potential partnership. Depending on the familiarity of partners with project goals and objectives, as well as expectations regarding each partner’s role to play in the upcoming project, this play may take more or less time. The critical self-reflection may take no more than an afternoon, but depending on the state of relationships the communication of and agreement upon each partner’s expected resource contributions and management may take several meetings. Procedure 1. Conduct a critical self-reflection of resource availability. The most important activity related to determining readiness to collaborate is for an organization or individual to conduct a critical self-reflection regarding the available time, energy, and resources they have for a given project. Elements of this reflection may include, but are not limited to: the ability to invest in developing a healthy, mutually beneficial relationship; the assessment and potential mitigation of past collaborative relationships; and the availability of physical and/or technical resources to support collaboration. Evaluation If collaborators have successfully determined their collaboration readiness, they have communicated the anticipated level of time, energy, and resources which they can contribute to their prospective partners. Metrics of success for the transparent communication of this information include avoidance of unnecessary conflict, efficient resource management, establishment of realistic expectations for involvement and levels of commitment, and clarity of roles and responsibilities. 2. Set and communicate expectations for resource management. Determining readiness requires not only critical self-reflection on a number of factors including those described above, but perhaps more importantly the transparent communication of the result of this reflection to potential partners. It is the communication of the results of critical self-reflection that helps potential partners understand each other’s needs and increase awareness of blind spots in the relationship, creating the foundation for a healthy partnership moving forward. Have an explicit conversation with potential partners regarding expectations of participation level, time commitment, financial, data or other contributions. Determining the extent of collaboration readiness and expectations for involvement is greatly helped by the Project Readiness Checklist included in this playbook. Related Tools and Plays • Project Readiness Checklist Examples Hinrichs, M. M., & Johnston, E. W. (2020). The creation of inclusive governance infrastructures through participatory agenda-setting. European Journal of Futures Research, 8(1), 1-15. 27 05 Creating Data Analytics and Visualizations for Shared Understanding What It Is and Why It Matters Data visualization and modeling serve important sensemaking functions when done in collaboration with community partners. However, each represents a different level of collective data interpretation and analysis. As part of their sensemaking functions, both data visualization and modeling help to focus and advance conversations with community partners, as well as identify potential data needs moving forward. Keep Reading If... If project members or participants are struggling to create shared narratives and to identify relevant research questions which can be explored through data and modeling, running this play can be very helpful. Definition and Purpose Data visualization and modeling serve important sensemaking functions when done in collaboration with community partners. Each represents a different level of collective data interpretation and analysis. Data visualization is a way for the research team to present a point in time understanding of trends evident in the data and to elicit descriptive knowledge from community partners that might help explain data trends or outliers. It is primarily descriptive and static, and useful for developing a shared understanding of the current situation. Data modeling represents a more interactive, predictive sensemaking function, wherein the research team and collaborators try to describe and model a potential current and future relationship evident in the data. Modeling thus attempts to reflect and demonstrate a dynamic relationship between data, with potential implications for future decision making, service provision or public policy. As part of their sensemaking functions, both data visualization and modeling help to focus and advance conversations with community partners, as well as identify potential data needs moving forward. At an introductory level, data visualization exercises help collaborators understand which data sources exist, are accessible, and are potentially useful in refining a community’s need or driving question. Using data visualizations to facilitate a conversation is useful because they give collaborators a concrete element to focus on, organize around, and communicate through with a relatively low barrier to entry in regard to numeracy or data literacy. Such visualizations become a focal point for the conversation to further refine a challenge area and thus generate relevant conversations regarding potential solutions. 28 Creating Data Analytics and Visualizations for Shared Understanding Data modeling adds a more complex layer to the process of collective sensemaking through the use of data interpretation and analysis with community partners. Modeling introduces a synthetic layer of projecting scenarios and potential futures onto data visualizations which may broaden collaborators’ understanding of what they know and reveal the implicit biases they bring to their ways of knowing. Modeling helps both the research team and community partners determine whether or not aspects of the data are meaningful, what aspects of the story are or are not represented, what knowledge is or is not captured, and what additional data must be obtained, validated, and integrated into the model to create a more inclusive, realistic picture. • In our recent work with the Valley of the Sun United Way we began with a series of simple data visualizations which included histograms of service provision from the Homeless Management Information System (HMIS) database. It took approximately a month for the data science team to get into and familiarize themselves with the HMIS database, albeit creating visualizations along the way to quality check their understanding of variables against community experts’. These visualizations provided a foundation upon which the data science team and community data specialists could introduce more complexity, ultimately leading to clustering algorithms to create an app based on a synthetic layer of information and interpretation which enabled the group to get into dynamic modeling. • In our collaboration with the Arizona Board of Regents (ABOR) and Achieve60AZ, we initially thought we would be using student-level data procured from the Arizona Department of Education. However, due to accessibility and data sharing issues, the basis of our data visualization and modeling became the US Census American Community Survey data. Both our research team and collaborators spent several weeks iterating data visualizations to familiarize both themselves and our ABOR collaborators on potential variables of interest as the data was organized and cleaned. In the modeling phase, additional data sets from the Bureau of Labor Statistics and Office of Economic Opportunity were introduced and connected to the Census data, requiring several more meetings to explain the complexity of the modeling and assist collaborators in reading and interpreting results. In sum, this process took almost a year to complete. Both data visualizations and models should evolve in tandem with community-driven conversations that guide their refinement, presentation and eventual application. Time and Timing Data visualization and modeling can be useful at the beginning of and throughout a collaboration to get everyone on the same page regarding the current state of affairs, or to identify relevant research questions which can be supported by further data analysis. Engaging in both data visualization and/or modeling focuses conversations, helps collaborators determine whether or not the data set(s) under consideration are helpful or relevant to examine, and in the process makes explicit what might be more tacit knowledge and perspectives held by diverse collaborators. How long the process takes depends largely on the existing data literacy and numeracy of collaborators, as well as their existing familiarity with the data set under consideration. For example: • In our work with the Littleton School District, a member of our data team had been an elementary school educator for 16 years before joining the faculty at ASU and had interpersonal ties with Littleton administrators. His experience and familiarity with the data sets, variables, collection methods and interpretive challenges greatly reduced the amount of time that the research team needed to spend getting up to speed on how to best leverage the data shared by the schools. This insider knowledge helped the research team in creating data visualizations which would be easily understood and accessible to the principals, vice principals, and district-level administrators who attended a series of data workshops to understand the relationship between teacher effectiveness ratings, disciplinary actions, and academic student outcomes. The data visualization and modeling team needs to be involved at the earliest appropriate moment when creating a collaborative relationship with community members and setting expectations for data interpretation and analysis. The more the data scientists can hear the needs and perspectives of the community, the more they can strive to integrate those perspectives into later visualizations and models. 29 Creating Data Analytics and Visualizations for Shared Understanding Applicable Conditions If a collaborative group is having a hard time finding a common narrative or identifying a starting point from which to build a research agenda, data visualizations and/or models can provide a focusing mechanism around which collaborators can gather, discuss perspectives, and elicit tacit knowledge. However, if you begin with a group that already has a wellestablished need or challenge, and which includes data experts from the community itself who can serve as valuable translators both to the data scientists and back to the leadership within their community, they might not need the focusing mechanism of data visualizations and can get to data modeling faster. Ultimately whether or not this play is necessary and the time required to complete it depends on the existing cohesion, familiarity, and data awareness of the group. Procedure 1. Set clear expectations for the depth of explanatory power of visualizations vs. models. Data visualizations are more superficial than models in terms of their depth and explanatory power. However, they often take considerably less time, resources, and iteration than a model which integrates several different databases or agents. Different communities will have different needs and interests regarding data analysis, which will likely be mediated by data availability and accessibility, as well as data literacy on the part of collaborators and eventual end users of a model. Clear expectations regarding the goals and outcomes of data analysis and modeling should be set early on in a collaboration. This includes the level of interest and commitment necessary from all parties to maintain participation and engagement throughout the data analysis and model iteration process. 2. Present data analysis at the beginning of a meeting or workshop. The initial purpose of data visualization and modeling in a knowledge convergence workshop is to focus and advance the conversation by making tacit knowledge explicit. Because of this sensemaking function, visualizations and models need to be presented at the start of a collaborative meeting. The ensuing level of conversation needs to be specific to the issues and decisions that went into preparing the visuals and models, since for many participants understanding the process of creating visualizations and models is often of more initial interest than the final result. 30 These conversations need to be framed and grounded in terms of non-technical language, free of jargon, which is familiar to the community members. Only then can conversations proceed to more complex topics. 3. Frame analysis as an invitation for knowledge coproduction, not a presentation of results. The presentation of data visualizations and models should always be framed as an invitation for knowledge coproduction, not a presentation of results. Thus, visualizations and models are tools for knowledge validation and not meant to transmit information unidirectionally. Within the context of a data workshop or meeting, this requires appropriate time to be allotted for participants to process a graphic and complete a cycle of feedback. The data team’s engagement with community members needs to be varied to elicit multiple forms and levels of feedback, with occasional targeted grouplevel workshopping meetings in addition to direct access and individual conversations with community data specialists or those with other forms of local expertise. Creating Data Analytics and Visualizations for Shared Understanding Evaluation The main goals of collective data visualization and modeling are to start conversations, inspire new ways of approaching a challenge or asking questions, or to create a common understanding of a community perspective. A hallmark of these goals is that people are engaged in conversation and that conversation evolves over time. If you present a visualization or demonstrate a model and community members or other collaborators in the room have nothing to say, you have likely not been successful. Conversely, if the presentation of a data visual or model stops a conversation, it is important to know what was unappealing or why people chose not to engage with it so the team can improve facilitation efforts or clarity of data presentations moving forward. A successful data tool or product may arise out of this process, but the need and use-cases for such a product need to be community-driven and well defined. Use-cases must be guided by community knowledge and input, and thus success metrics and uptake will also be dictated by the community. Related Tools and Plays Project Readiness Checklist Examples Opel, Dawn, and Michael Simeone. “The Invisible Work of the Digital Humanities Lab: Preparing Graduate Students for Emergent Intellectual and Professional Work.” DHQ: Digital Humanities Quarterly 13.2 (2019). Resources Getting Started With Tableau Desktop (help.tableau. com). This section gives you the basics on getting started with building views in Tableau, the Tableau workspace, and Tableau concepts, a visualization and modeling software. RStudio Cheatsheets (rstudio.com). These cheatsheets make it easy to use some of the R creators’ favorite packages. From time to time, they will add new cheatsheets. Welcome to Shiny (shiny.rstudio.com). Shiny is an R package that makes it easy to build interactive web applications (apps) straight from R. This lesson will get you started building Shiny apps right away. PyGraphistry (github.com). PyGraphistry is a Python library to quickly load, shape, embed, and explore big graphs with the GPU-accelerated Graphistry visual graph analyzer Getting Started With Tableau Desktop (help.tableau.com). This section gives you the basics on getting started with building views in Tableau, the Tableau workspace, and Tableau concepts, a visualization and modeling software. Knowing What You Don’t Know: Choosing the Right Chart to Show Data Distributions to Non-Expert Users (dataliteracy.eita.org.br). This academic article provides guidelines on how to choose the right chart to display data distributions for non-experts. Contributions of Paulo Freire for a critical data literacy (dataliteracy.eita.org.br). Paulo Freire’s main work - the Popular Education pedagogy - influenced many educators all over the world who believed in education as a way of liberating poor oppressed people. In this paper, the authors propose adapting elements of the Paulo Freire’s Literacy Method for use in data literacy, i.e., the act of building capacities for working with data. Hinrichs, Margaret M., and Erik W. Johnston. “The creation of inclusive governance infrastructures through participatory agenda-setting.” European Journal of Futures Research 8.1 (2020): 1-15. Miller, Brian W., et al. “Co-producing simulation models to inform resource management: a case study from southwest South Dakota.” Ecosphere 8.12 (2017): e02020. 31 06 Managing the Inclusion Process to Optimize Stakeholder Commitment and Satisfaction What It Is and Why It Matters Community resilience collaborations are often voluntary environments where stakeholders are able to freely determine their level of commitment and participation. In these circumstances, relationships play a significant role in increasing the likelihood of gaining authentic commitment from stakeholders. This play offers strategies to help you navigate when, how, and to what extent to include different collaborators, resulting in greater stakeholder satisfaction with their use of time and energy, high-quality participation, and better project-level outcomes downstream. Keep Reading If... If you are in a collaboration where partners can choose their degree of involvement, this play is essential reading to increase effectiveness and efficiency. Definition and Purpose Creating knowledge and inclusive solutions for complex resilience challenges requires collaboration among and quality participation from diverse stakeholders, such as individuals, civil society, and academic and governmental institutions. When ramping up a project, it can be difficult to know when, how, and to what extent to include different collaborators. Managing the inclusion process carefully is very important in voluntary environments, such community resilience collaborations, because collaborators can freely withdraw their support and allocate their time, attention, and resources elsewhere if they judge their participation is not worthwhile. With strategies to prioritize stakeholders, demonstrate commitment, motivate quality contributions, and more, this play shows you how to maximize the value of stakeholder participation, gain authentic commitment, improve project outcomes, and develop relationships with a higher probability of future collaboration. Time and Timing Managing the inclusion process will take place throughout the lifecycle of the collaboration. It will take more or less time depending on the existence and/or development of relationships and trust. Communities that have worked together for a long period of time can onboard each other quickly. Where trust has not yet been established, managing the inclusion process can take longer. For this reason, collaborations that have a fixed, short-term deadline with a new community partner must be realistic and communicate clear expectations for participation with potential partners. Applicable Conditions This play is not as necessary if a project already has top-down hierarchies, clear power dynamics, or other means of compelling participation. If participation in the collaborative is voluntary, however, this play may greatly increase your collaboration’s effectiveness (through higher quality contributions) and efficiency (by including collaborators only as much as needed). 32 Managing the Inclusion Process to Optimize Stakeholder Commitment and Satisfaction Procedure 1. Prioritize who and in what order to include people. Begin with the primary stakeholders or funders (i.e., those who have organized or initiated the project) to create a start list of participants. Ask whose expertise they believe is needed, who they want to participate, who they would recommend that is willing to put in the time and energy to engage with the project. Co-develop a set of evaluative criteria to determine who should be included after this initial start list. Reasons for inclusion may fall into several categories which may include, but not be limited to: • which have existing accountable relationships with the primary stakeholder, • which have additional resources (data, funding) to contribute, • which groups have political influence, • which have the most opportunistic potential, • which have historically been left out of collaborative projects, or • which are available in the given time frame of the project. Within these categories, work with the primary stakeholder to prioritize potential participants. For example, in our work with the Arizona Board of Regents we sought to create a data-driven model to visualize and compare tradeoffs between different approaches to reaching 60 percent adult postsecondary attainment in Arizona by the year 2030. To understand the data available and accessible to such a project, as well as to ensure relevance of data analysis which would result from the model, the project required the inclusion of multiple stakeholders within the education, economic, and workforce development communities. We began with a start list of participants identified by ABOR as being relevant and necessary to the successful use of such a model, using many of the criteria identified above. We then engaged in snowball sampling with stakeholders, asking each to recommend one or two other groups whose voices they felt would shed additional light on the questions we were trying to answer, as well as voices which may have been excluded from recent education and workforce-related discussions. 2. Create ways for partners to demonstrate their commitment to the project. Rather than an all-or-nothing mindset to collaboration, it is useful to create clear, intermediary milestones to which participants can contribute. Such milestones create opportunities for participants to demonstrate their commitment to the project and help others gauge whether interest is continued or flagging. The review and evaluation of milestones help participants, primary stakeholders, and the research team re-evaluate continued involvement among groups, if necessary. It is at these points that the collaboration team can develop exit ramps that are “healthy,” enabling a collaborator to leave the project while also preserving relationships. The use of exit ramps also helps the core collaboration team to decide on inclusion criteria for further involvement of remaining community participants. 3. Create feedback loops to incentivize further collaboration. Creating feedback loops within a collaborative project helps participants to see evidence of their involvement as the project grows over time, realize the impact of their investment in shaping the development or use of a data-driven model or tool, and further motivate their continued involvement in the project. Feedback loops help participants see that their involvement (or lack thereof) has consequences for the overall project and may impact their future relationships with other collaborators who are involved. For example, members of our team recently worked on a project funded by the Robert Wood Johnson Foundation to understand continuity of care for people with serious mental illness in Arizona. The research team ran a number of themed workshops for the first phase of the project, to which all 50 potential community partners were invited. Within the workshops, participants critiqued the visualization of an analysis linking cost to the state’s fractured approach to treating serious mental illness, among other outcomes. At each subsequent workshop, the research team would review changes made in the previous visualization and continue to refine analysis in light of community input. In addition, the 11 community partners who attended all five workshops and could speak to the lifespan of the project were invited to participate in an in-depth evaluation of the entire collaborative research process, setting community-driven expectations and criteria for the second phase of the project. 33 Managing the Inclusion Process to Optimize Stakeholder Commitment and Satisfaction The question of who enforces the consequences of feedback loops varies according to the project. Collaborations may be managed by a facilitative leader, director of a research center, or principal investigator of a grant. Ideally, a collaborative partnership establishes and maintains shared norms of communication, transparency, accountability, and trust which are mutually reinforced for the duration of the project. 4. Encourage critical self-reflection of the collaborative process. Throughout the collaborative process, the core collaborative team (researchers and primary stakeholders) should conduct critical self-reflection of the extent to which they are managing the inclusion process. Examples Hicks, D., Larson, C., Nelson, C., Olds, D., & Johnston, E. (2008). The Influence of Collaboration on Program Outcomes: The Colorado Nurse—Family Partnership. Evaluation Review, 32(5), 453–477. https://doi. org/10.1177/0193841X08315131. Hinrichs, M. M., & Johnston, E. W. (2020). The creation of inclusive governance infrastructures through participatory agenda-setting. European Journal of Futures Research, 8(1), 1-15. Resources Collaboration Success Wizard (sc.ctsi.org). The Collaboration Success Wizard (CSW) is a 30-minute survey which generates individual and project-level reports diagnosing strengths, potential challenges, and coping strategies for geographically-distributed collaborations, using the latest findings from collaboration science. Defining Collaboration and Explicating the Collaborative Process (bcpsqc.ca). This document sets out six empirically-demonstrated features that make for a successful collaboration. The features can be used as evaluative criteria to assess the health of the collaboration as well as design principles for collaborative processes. Evaluation This play has succeeded if people see involvement as a valuable use of their time and resources, and if people have better relationships with each other after their participation. Managing the inclusion process carefully is very important in voluntary Related Tools and Plays environments, such community resilience collaborations, because collaborators can freely withdraw • Identifying Critical Community Partners • Building Your Stakeholder Network 34 their support and allocate their time, attention, and resources elsewhere if they judge their participation is not worthwhile. Game Plan Notes 07 Leading to a Productive Conclusion What It Is and Why It Matters At the conclusion of a collaborative project, the team may have many options for next steps, such as maintenance, growth, changing focus, or ending their work together. But how do you select the “right” alternative? This play outlines a procedure to help your team make a choice that makes the most effective and efficient use of your resources, in light of your goals. Keep Reading If... You should keep reading if your team has concluded its collaborative project and is considering next steps, or if signs of unhealthy collaboration have emerged. Definition and Purpose After the core collaborative team and community partners have completed the scope of work for their community resilience project, they must now consider what the next steps are. There are many options. A collaboration may continue what it was doing, grow, change focus, or it may simply end. Not all collaborations are meant to last forever (especially if they have fulfilled their purpose), and continued work together is not necessarily the best option, even if the team is performing adequately. So how do you select the “right” alternative among those many options? While the answer to that question will depend on the values held by the team, this play proposes that: (1) viable alternatives are characterized by the alignment of team motivation, community impact, legitimacy, opportunity, and collaborative capacity; and (2) the “best” alternative uses the group’s resources to optimize for certain chosen criteria, and is arrived at through group input or consensus. Running this play will help your team select the alternative that uses its resources in the most effective and efficient way possible for your goals. Time and Timing This play is ideally used in the closure phase of your project, after you have completed your project’s scope of work and (hopefully) accomplished your objectives. However, it may be necessary earlier on if you are noticing symptoms of unhealthy collaboration, detailed in the next section. Applicable Conditions When a collaboration has been formed around a single community resilience project (as opposed to ongoing service provision), this play becomes applicable to your team after the project’s scope of work has been completed. You and your team, in the course of doing the project, have inevitably learned a lot about one another, the focal challenge that motivated your participation, and the opportunities that exist to make an impact. Therefore, the conditions that undergirded your collective decision to commit your resources to a joint project will have also shifted or evolved, and it is time to determine how your collective resources can be best allocated in the next phase (if there is one). 36 Leading to a Productive Conclusion However, sometimes it is necessary to reconsider the direction or fate of the collaboration before the group’s scope of work is complete. This play may be applicable if you notice the following symptoms, drawn from Table 2 of Imperial et al. (2016, p. 142) • “Departures of key individuals viewed as a crisis • Members exit or stop committing resources • Mission drift • Excessive member turnover • A high-degree of “free-riding” • New problems command members’ attention” In either circumstance, this play will help you deploy or redeploy your collective resources more effectively and efficiently. Procedure After the scope of work is completed and the project is wrapping up, sometimes the next steps are obvious. For example, perhaps the team just didn’t work together well, and the collaboration needs to end. Or perhaps local decisionmakers loved the team’s work, and an opportunity has arisen to repeat the work in other communities, which the team is excited about. But next steps are not always so clear, such as when multiple good options are available or when the collaborative team is performing adequately, but not at a high-level. The following steps are one way to help you make a decision under such circumstances. • Maintenance: When the team is performing well, motivated, delivering value to the community, and resources are stable. • Growth: When there are resources available, opportunity for it, demand from the community, and motivation from the team. • Changing focus: When the team is effective and enjoys collaborating with one another, but the focal challenge has lost its salience due to other rising priorities. • Spinning off: When the focal challenge continues to be an issue, the team is no longer motivated to work on it, and other groups exist who are willing and capable of taking on the challenge. • Ending collaboration: When the challenge is resolved, when the collaborative is underperforming, or when better uses group members’ resources exist. At this stage, you should have some notion of which options are better or worse, but you need to get input from the broader team to make a more definitive decision. 2. Get initial feedback on alternatives from the broader collaborative team. Get feedback from other team members and critical community partners to gauge their feelings on the alternatives. This can be done through a group meeting, one-on-one meetings, or even through a web-based survey if you are dealing with a larger team. 1. Develop alternatives for the future of the collaboration. With the leadership or the core collaborative team, develop alternatives for the future of the collaboration. Common options include maintenance, growth, changing focus, spinning off, or ending collaboration (“Maintaining a Coalition”; Imperial et al 2016). Each alternative, very briefly summarized below from Community Tool Box (see “Maintaining a Coalition” in the Resources section), may be appropriate in different circumstances. Additional insights can be found in Imperial et al. (2016). 37 Project Name Project Value. Chance of Success Resources Challenge salience • Is the challenge causing significant harm, or will it cause significant harm if nothing is done? • Is the challenge a priority to the members of the collaborative team? • Is the challenge recognized as important by the community, relative Favorable conditions • Do current or foreseeable conditions heighten the team’s chances of successfully solving the focal challenge? • Can the team capitalize on these What resources are required for this course of action? conditions? to other priorities? Inspiring vision • Do benefiting stakeholders stand to benefit significantly? • Is the vision inspiring and aligned • with collaborators’ individual and organizational visions? Is the vision desired by legitimate stakeholders in the community? Threats • Do current or foreseeable conditions threaten the team’s chances of successfully solving the focal • • challenge? Is the team capable of coping with the threats? Will continued collaboration still be worth it if these threats come Do you have the resources required, or can your team mobilize them? Are there foreseeable threats to the participation of key team members? to pass? Need and the Impact of the collaborative project • Is the need the project set out to fulfill salient to the collaborative team? • Is the need salient to stakeholder groups / the community? • Will the collaborative project be effective in fulfilling the need? • Does this project fill a gap that Effectiveness of the collaboration • Has the collaborative team completed work together in the past with a high degree of quality? • Were project objectives achieved? • Are there others willing and able to take on this project? Were stakeholders satisfied? existing efforts satisfy? Achievement of individual goals • Is the project aligned with collaborators’ individual and / or organizational goals? • Will the project advance those goals? Collaboration quality • Did the team enjoy working together? • Are members experiencing stress or burnout? • Did team members consider the collaborative process legitimate, fair, and well-managed? Notes 38 Can those resources be better deployed? e.g., to a project with more value, probability of success, or both? Leading to a Productive Conclusion 3. Evaluate and distinguish the viable alternatives An alternative is considered “viable” when it is characterized by team motivation, community impact, legitimacy, opportunity, collaborative capacity, and sufficient resources. When these factors converge, the team and project are in a stable configuration that is likely to achieve its objectives, deliver value to stakeholders, and encourage highquality contributions. When they are out of alignment, the collaborative project runs a higher risk of unhealthy collaborative process, lower performance, loss of members to other opportunities, and project failure. The table in this play aggregates insights from KER interviews, Imperial et al. (2016), and Community Toolbox to assess alignment of these factors by answering each of the listed items with a “Yes”, “No”, or “Maybe / Uncertain”. Viable alternatives are the ones that are mostly “Yes”. Use your knowledge and the initial feedback you received from the broader collaborative team to complete a table for each alternative. 6. Communicate your decision to the team and move forward Communicate the decision to the full team, including your reasoning and their input. This ensures that members feel heard and understood, and that helps to give the decision legitimacy. If you choose to end your collaboration, create a healthy end with team leadership and move on. Communication can be tricky when ending collaboration. When it is, it can help to blame mismatches (e.g., collaborators were interested in addressing the focal challenge at different spatial scales) rather than the other team members. This helps to preserve the relationship and leave the door open to future work together. When you do end your collaboration, consider having an informal social gathering or party. This helps preserve ties while allowing individual members to network with each other on future endeavors. 4. Convene the team and deliberate on the best alternatives. See if you can make adjustments to the alternatives for better alignment and commitment. Reassess team preferences, perhaps through a vote. 5. Make a final decision. There are various ways to do this — is the project still doing good in the world? With the same resources, could you have a greater impact in some other endeavor? We propose a decision-making model based on maximizing value and impact based upon available resources, because, in collaborations based on voluntary commitments, these opportunity costs can undermine the collaboration down the line. With the resources required for the project, is the team having the biggest impact it could have? How the decision is made depends on the way power is distributed in the team, but achieving consensus among the team leadership and critical community partners is a good practice when everyone is voluntarily committing themselves to the project. Related Tools and Plays • Managing the Inclusion Process Examples Imperial, M. T., Johnston, E., Pruett-Jones, M., Leong, K., & Thomsen, J. (2016). Sustaining the useful life of network governance: life cycles and developmental challenges. Frontiers in Ecology and the Environment, 14(3), 135–144. https://doi.org/10.1002/fee.1249 Resources Maintaining a coalition (ctb.ku.edu). This resource, referenced for this play, contains a great deal of detail on various alternatives for maintaining, transforming, or ending your collaboration, as well as a wealth of other resources useful to community-based work. 39 Game Plan Notes 40 41 TOOLS 01 Humility Audit: Conducting a Critical Self-Assessment Before Doing Community-Engaged Work What It Is and Why It Matters The Humility Audit is a tool that enables university researchers to conduct a critical selfassessment of the biases and assumptions they bring to community-engaged work. This tool permits faculty to identify their assumptions and orientations in regard to communityengaged work, and to engage in critical conversations regarding those assumptions before beginning a project. Keep Reading If... At the outset of a collaboration, particularly if engaging with a new community partner. Definition and Purpose The Humility Audit is a tool that enables university researchers to conduct a critical self-assessment of the biases and assumptions they bring to community-engaged work. This tool permits faculty to identify their assumptions and orientations in regard to communityengaged work, and to engage in critical conversations regarding those assumptions before beginning a project. Its development included representation from local governments and community groups who identified structures which most enable or constrain their collaboration with academic institutions. It is built upon an analysis of best practices emerging from co-design workshops. Time and Timing 20 minutes for the Audit, additional time to discuss results with team members. Applicable Conditions Anyone who is working with a community partner, particularly for the first time, should consider having every member of their team fill out the Humility Audit. Procedure The Humility Audit has been developed by https://partner.openingpathways.org/ and features an online questionnaire that prospective collaborators can fill out. The audit features the following kinds of questions: 44 • What are your methods of choice when engaging with communities? • What role do you commonly play now in community-engaged projects - acquiring funding, connecting social networks, contributing expertise, applying technical or data science? • What role do you envision yourself playing in this project? • What role do you envision your community partner playing in this project? Humility Audit: Conducting a Critical Self-Assessment Before Doing Community-Engaged Work • • • What are your current, pending, and dormant projects can include writing or side consulting, anything else that will also be occupying your time? Would you forego any of your existing projects to devote time and effort to the prospective project at hand? Which one(s)? To what extent does your current institution or organization support the community-facing work you are considering taking on? • Do you or someone you know have firsthand experience working with people from the potential community partners? • What outcomes from this potential partnership do you think are most important to your potential partner? • What would success look like, from your perspective? • What specific products do you hope to produce with this work, or what processes do you hope to change as a result of your work? • How might you be different at the end of this prospective project or partnership? • What do you think your potential partner(s) know that you do not? • What would you like to learn, or what skills would you like to grow, in working with your community collaborators? • How might your differences (time availability, resources, background skills, perceived or structural power) influence the collaboration you are considering? Related Tools and Plays • Determine Readiness to Collaborate • Including Critical Community Partners • Managing the Inclusion Process Resources Opening Pathways (partner.openingpathways.org). This web resource has quiz to help researchers assess their readiness to work with patients as well as narratives from humbled researchers who have engaged in research with patients. After taking the Humility Audit, prospective collaborators should share and discuss results to make sure everyone is on the same page, and identify possible discrepancies in community-engaged approaches before a project starts and people have committed resources. Like any other self-reporting measure, the Humility Audit is only as reflective as the person taking it. Evaluation After using this tool, the research team and potential community partners should feel greater clarity of on the roles and expectations for one another. 45 02 PQRS Approach to Science Communication What It Is and Why It Matters Researchers do not always communicate projects or research well, and this can have impacts on their prospects finding funding or enrolling a new collaborator,or even convey significance to existing sponsors or partners. Through a step-by-step process, the PQRS tool ensures researchers are prepared to make the most out of conversations about their projects, and perhaps even help you focus on research design in the first place. Keep Reading If... This tool is needed if people react with bored or confused looks when you talk about your research or if you are struggling to pinpoint for yourself, what you really aim to do. Definition and Purpose Whether they are trying to convince a funder, enroll a new collaborator, or explain to their mothers what they do, many researchers struggle to communicate what their research is and why it matters. The PQRS (Problem, Question, Research Purpose, So What?) Approach to Science Communication is a tool that helps researchers communicate their work concisely and effectively. Benefits of the PQRS approach include more funding, more collaborators, and generally improved outcomes from conversations for scientists describing their research. Time and Timing This tool is useful at any time you need to communicate succinctly about your what your research is and why it matters — seeking out funders, collaborators, etc., defining potential impact. Applicable Conditions Anybody who needs to communicate their research for any reason can benefit from using the PQRS Approach to Science Communication to clarify their thoughts. Procedure PQRS stands for Problem, Question, Research Purpose, and So What? In the text that follows, we will review what these mean and how to develop them yourself. 46 PQRS Approach to Science Communication 1. Problem 2. Question The problem sets the stage for your audience and gives them a sense of the drama and mystery that drives your work. There are a number of ways to make resilience problems clear and compelling. After defining the challenge, you will make your audience curious about answers by crafting thought-provoking questions your research responds to. And make sure you use an actual question mark! Compose 3-5 succinct questions your work raises. If you have trouble getting started, try starting off questions using the following question prompts: Who? What? When? Why? Where? How? To what extent? What is the relationship between...? First, state the harm. For example, “there are over 25,000 people experiencing homelessness in Maricopa County”. Then, put it in motion — “there are over 25,000 people experiencing homelessness in Maricopa County, and the number is growing year-by-year”. If you can provide an anticipated consequence in a nonintervention scenario, even better — “it is anticipated that, if nothing is done, everyone in Maricopa County will be homeless by 2030”. Example: How do people use their hearing to adapt when they are losing their vision? 3. Research Purpose Next articulate the forces that are exacerbating the problem or working against solutions. To build upon the previous example — “the number of people experiencing homelessness in Maricopa County is rising because affordable housing is more difficult to come by as the costs of housing continue to increase” (notice we also put this fact in motion). In short, what is the problem and what is the conflict or controversy about that issue? In one or two sentences, concisely describe the problem that inspires your activity. You may use the following questions to help: Next, you will describe how your research contributes to answering the question. Research usually contributes in one of three ways: 1. Advancing existing research (i.e., standing on the shoulders of giants) 2. Resolving contradiction / controversy (i.e., we are going to see which of these are correct) 3. Developing a new line of inquiry, when you are trying something completely new. This is uncommon — you are usually building on something that has been done before, but sometimes this means you are combining existing parts in new ways. Your recipe is the innovation. • What is the unsatisfactory situation? • What is problematic? • What is changing? How? • What is the conflict or controversy about? You can use the following words to help you describe your research purpose. • What factors are confounding the issue? How does this relate? Advance Existing Research: Advance; promote; grow; develop; extend Example: Millions of people with incurable and uncorrectable vision loss struggle to achieve mobility and independence. Yet treatments and techniques available to help them are not based upon a clear understanding of how individuals with different types, onsets, and severities of vision loss must be treated differently because of their differential abilities to take advantage of auditory information. Resolving Conflict or Controversy: Compare; contrast; determine; resolve; settle; decide Developing a New Line of Inquiry: Model; create; design; invent; generate; formulate; converge 47 PQRS Approach to Science Communication Example: We will design, test, and implement a novel research initiative to measure the relative contribution of visual and auditory information when visually impaired individuals make judgments about collision. We will study how this relative contribution changes with increasing vision loss. 4. So What? Finally, make it clear to your audience why they should care. The following exercise may help: 1. 2. With a partner, explain why you are asking the question you chose. Your partner should then ask again, why? Please answer: 2020 KER Resilience Fellows Presentations YouTube Playlist. The 20 Fellows in this playlist used the PQRS exercise to frame their work. Resources PQRS Video Lecture. This is a 10-minute video on the PQRS method described in this text. Partner asks why a third time. Answer. 3. Partner should ask once more, why? Provide another response. 4. A fifth time, the partner asks why, for a final reply from you. 5. Make note of the trajectory of your answers to why. These may form a deepening understanding of the relevance of your research. Example: (S) Results will be used to advance a more nuanced approach to rehabilitation and development of new assistive technologies sensitive to the different realities of the visually impaired. Evaluation You and your audience understand what you do and why it matters. Related Tools and Plays • Determine Readiness to Collaborate • Project Readiness Checklist 48 Examples The Art of Scientific Storytelling: Transform Your Research Manuscript using a Step-by-Step Formula by Rafael E. Luna (amazon.com). This book shows you how to put your Title and Abstract into a story, along with the rest of your manuscript. AAAS Communication Toolkit (aaas.org). Excellent public engagement with science builds on a foundation of clear, concise communication. This toolkit provides guidance and tips to improve your communication skills. Game Plan Notes 03 Finding Connections with Collaboration Information Systems What It Is and Why It Matters Novel data systems and information tools are available on many campuses to track research and productivity. These can be repurposed or new tools created to allow university researchers, program leaders, and other community-engaged professionals to report and search community-embedded activities. One example, Collaboratory, gathers information from across campus into a centralized, publicly-accessible platform to facilitate awareness and connections among actors who are interested in similar public problems and working in the same locations. Keep Reading If... Collaboration Information Systems can be very useful when seeking out new connections or to learn more about past partnership opportunities. This makes it easier at the stage when team members are conducting stakeholder identification and outreach in the initiation phase of a project. At project closure, reporting your project back to collaboration systems can also increase your visibility to others interested in future collaborations. Definition and Purpose Universities can be large, decentralized institutions undertaking many different research activities with various communities. Community actors are also manifold, undertaking numerous activities to the benefit of their communities. If university and community actors are aware of each other and their activities, they can more easily connect and collaborate to co-design, co-develop and collaborate on community challenges. One way that information can be systematically gathered and shared is through a tool such as the Collaboratory (http://links.asu.edu/collaboratory) at Arizona State University. This platform is a space where faculty, students, and staff can report and find socially-embedded activities. The platform allows ASU faculty, staff, and external community partners to search and find out who at ASU is doing what and around what issues (particularly Sustainable Development Goals), focusing on which target populations, in what locations around the world, and with whom. In addition to its core purpose, the platform offers further benefits, such as the ability for individuals and units to download reports on their engagement activities for evaluation or other purposes. This is one example of how universities can systematize information about their collaboration connections. 50 Finding Connections with Collaboration Information Systems Using a collaboration information system may yield the following core benefits: • • • Build your personal community-engaged scholarship profile by documenting artifacts of your engaged teaching and research. Find, connect with and learn from other faculty, staff across the university who are addressing the same social issues, working in the same communities or with the same partner organizations. Strengthen and support ASU’s ability to showcase our commitment to be socially embedded in reports, accreditations and institutional recognitions. • Elevate the important role of our partner organizations in creating social impact. • Contribute to conversations that develop communitycampus partnerships. An even wider range of stakeholders may find their own use for data and information that come from such systems. For example, funders may seek out researchers working on particular problems, or a dean may want to use the platform to understand all the ways in which they are engaging the community to use it for funding, admissions, or recruitment. Time and Timing During the initiation and planning phases of your project, when you are seeking out stakeholders, critical partners, or other collaborators, look for collaboration information systems on your campus and in your community to speed up the process or provide connections you would have otherwise overlooked. During the closure phase of the project, you may document your collaboration back to organizers, making it easily searchable for potential future collaborations. Applicable Conditions Best practices learned from ASU’s Collaboratory can help inform this process. You should submit an activity to a Collaboration Information System if it: 1. Relates to a public priority or issue. 2. Involves one or more external community groups, organizations, agencies, and neighborhoods. 3. Yields mutual benefits are articulated by both ASU and the external community group, organization, agency, or neighborhood. 4. Includes an exchange of knowledge or expertise with the external community group, organization, agency, or neighborhood. 5. Is considered community engagement or a public service. You should search a Collaboration Information System if you want to learn about ongoing or previously transpired community-embedded activities in your area of interest as well as the associated faculty and staff, community organizations, or academic units. This can help you find potential collaborators, partners, or stakeholders! Depending on the tool and the community, additional targeted categories can be highlighted to meet mutual goals. For example, the search function of ASU’s Collaboratory is particularly robust in finding activities aligned with Sustainable Development Goals (SDGs). Procedure To explore how a Collaboration Information System works, try out Collaboratory. Simply: 1. Set up a profile as a guest (for non-ASU users) 2. Create a new activity 3. Input basic information, collaborators, focus areas and populations, relationship to scholarship, and goals 4. Submit 5. Seek out others and other activities 51 Finding Connections with Collaboration Information Systems One of the biggest challenges of many Collaboration Information Systems like Collaboratory is that they often rely on self-reporting from faculty, staff, and students,who are already very busy. Universities and academic units who wish to institute a Collaboration Information System like Collaboratory need to devise context-appropriate ways to incentivize use, especially in the early adoption stages. For example, an academic unit may choose to reward communityengaged scholarship and use a Collaboration Information System as the means for monitoring who is doing what in the community. Any of the benefits mentioned above are useful for beginning conversations with potential users on why a Collaboration Information System should be employed. Related Tools and Plays • Building Your Stakeholder Network • Identifying Critical Community Partners 52 Game Plan Notes 04 Community Resilience Project Planning Playboard What It Is and Why It Matters Projects can change or enhance the status quo toward greater community resilience. This tool is a way to stimulate thinking about community resilience project design, who will benefit from the project, what results (outcomes) the project can achieve in the 6-12 months after implementation, how the project might contribute to deeper change over the longer-term, and how short-term results can be measured. Keep Reading If... The Playboard provides a structure for articulating and extending thinking about your project design and what it will accomplish. It connects what project collaborators will be doing (activities) with the outcomes they anticipate as a result of implementing these activities. Definition and Purpose In the planning phase, it is common for project collaborators and partners to have a lot of ideas swirling around in their heads and scribbled on notepads. The Playboard is a way of structuring and linking these thoughts. It has two major components: a set of questions for thinking about community resilience project design and an associated set of questions around defining impacts and outcomes contributing to community resilience. Working through the Playboard will allow project collaborators to discuss and articulate ideas around project design while defining the critical changes they think will occur through implementation in the short-term and how these are linked to longer-term changes and impacts. The outcomes and outcome measurement strategies defined through the Playboard are meant to be used and refined throughout project implementation. Time and Timing The Playboard is intended for use during project planning after the project collaborators have defined the project focus. It asks collaborators to think about and articulate who will benefit from the project and how they will benefit in order to center projects on potential outcomes or changes toward greater community resilience rather than simply on activities. Applicable Conditions This is a tool that individuals and groups of collaborators can use. The process of defining outcomes can be important for building trust and consensus among collaborators and stakeholders as well as an opportunity for multiple perspectives to be heard. 54 55 Thinking about Community Resilience Thinking about Outcomes Contributing to Community Resilience Short-term outcome: Short-term outcome: Longer-term outcome: Longer-term outcome: Short-term outcome: How will you know if an outcome happened? Example: KER’s work on mobile homes and extreme heat resulted in these short-term outcomes: Increase in scientific knowledge about heat exposure in mobile homes; Increase in public health awareness about the vulnerability of mobile home residents to extreme heat. VALUE & CONTRIBUTION What value will this project bring? How will it contribute to community resilience? What conditions for resilience might be enhanced or changed? SHORT-TERM OUTCOMES What changes do you expect to see in the next 6-12 months as a result of your project? What difference will it make to those communities, organizations, or groups of individuals you identified above as benefitting from your project? Think about changes in knowledge, attitudes, skills, and small behaviors. WHO BENEFITS Who will directly benefit from or be impacted by your project? Which communities, organizations, or groups of individuals will be impacted? SHORT-TERM OUTCOME MEASUREMENT How will you and your community partners know if these short-term outcomes have happened; what are the indicators of these changes? How can you measure whether an outcome happened; what data might you collect, and how will you collect it? Impact(s): Longer-term outcome: LONGER-TERM OUTCOMES What longer term changes do you hope to see put in motion by your initial projects? These will be outside your project timeline but are important to consider. Think about changes in behaviors, actions, decisions, and practices. IMPACT What would it look like in this community in the future if these resilience challenges were fully addressed? Think about changes in policies and systems. These will outside your project timeline and direct control but dream a little! See the PQRS exercise. IDENTIFICATION How was this challenge, opportunity, need, or barrier identified? Who was involved, and what was discovered? PROBLEM STATEMENT What community resilience challenge, opportunity, need, or barrier will this project address? Roles: How will you measure whether it happened? Activity: Activity: Activity: STRATEGIES & ACTIVITIES What will you be doing in your project over the next 6-12 months to promote or stimulate the attainment of your short-term outcomes and contribute to community resilience? Stakeholders/Partners: See tools for working with stakeholders and partners. STAKEHOLDER INVOLVEMENT What stakeholders or community partners will be involved in this project? What will they do? Planning Playboard for Your Community Resilience Project Community Resilience Project Planning Playboard Procedure • Partner and stakeholder involvement: People – community stakeholders, partners, and collaborators – are at the center of community resilience. There are several plays and tools in this playbook that detail how to build stakeholder networks, identify community partners, and collaborate with partners. While collaborations are grow during project implementation, it is critical at the outset to think through what these might look like. • Other: This Project Planning Playboard only lists a handful of things to think about when planning and designing a community resilience project. What else is important for your project? 1. Thinking about community resilience. The top portion of the Playboard asks collaborators to think about the design of their community resilience project. This entails the following: • Purpose statement: This is an opportunity to refine the focus of the project toward community resilience challenges, opportunities, needs, and barriers. It will help project collaborators answer the question of “why” this project is important. For many projects, a purpose statement may be the same as a problem statement, which is covered in the PQRS tool. Other projects may focus on leveraging community assets to enhance community resilience rather than focusing on vulnerabilities. 2. Thinking about outcomes contributing to community resilience. • Identification: Effective projects come from somewhere, and the story of a project’s genesis is important to convey to funders and other stakeholders to gain support for the effort. The top portion of the Playboard grounds project collaborators in the rationale for the project, who it will benefit, and how partners and stakeholders might be involved. The bottom portion takes this further and asks about what changes will result from this endeavor. • Who benefits: Understanding who and/or what entity will benefit from project implementation is a critical part of project design. Often, projects benefit a group of people who are either living within a geographic area (as in a neighborhood or city), part of an organization, or belonging to a community of people with a common identity, interests, or challenges. Other projects might focus on organizational change toward improving community resilience. While some initiatives might focus on one beneficiary, projects can benefit more than one group or entity. For example, a community resilience research project might benefit individuals in a specific neighborhood as well as the scientific community in an academic field. “Outcomes” are the changes that project collaborators want to occur during and after project implementation. These can be changes to individuals, groups, organizations, communities, or systems. Outcomes can entail changes in individuals’ knowledge, skills, attitudes, and behaviors over the short-term which can lead to larger changes (longer-term outcomes and impacts) around a group or entity’s actions, decisions, practices, and policies as well as systems change. These are the reasons why collaborators engage in community resilience projects – they want to precipitate changes that led to greater community resilience. • 56 Value and contribution: In addition to defining a project’s purpose, articulating a project’s value and contribution to community resilience is important for answering “why” a project should go forward. KER has drawn on former Rockefeller Foundation president Judith Rodin’s thinking to define key characteristics for its own community resilience work. KER’s projects and initiatives address one or more of these community resilience characteristics: Aware, Diverse, Integrated, Self-Regulating, Adaptive, Equitable, Socially cohesive, and Place-Based. Project collaborators may identify other community resilience characteristics that are important for their particular endeavor. • Impact: It is common to start from the big changes we want to promote over the long-term and work backwards in time. This “backcasting” begins with identifying a desired future around the larger project purpose. While this will be outside the scope of time-bound projects, a recognition of impact(s) enables a greater understanding of the larger systems in which this project exists and the interaction of numerous efforts toward promoting these larger systems-level changes. Community Resilience Project Planning Playboard • Longer-term outcomes: These are also outside the scope of smaller, time-bound projects. Identifying what longer-term outcomes need to happen for impacts to be realized is critical for contextualizing a project and articulating a set of hypotheses about how long-term change can happen. As project collaborators move through a project, they can refine their understanding about the relationship between longer-term outcomes and impacts as well as between short-term and longerterm outcomes. Many logic models also include “outputs,” which are measures of your activities such as the number of people served by an activity, the number of events held, the number of solar panels installed or tangible project deliverables like websites, training curriculum, and policy documents. Outputs have been left out of the diagram below to allow teams to focus more directly on the changes they want to promote but are important to track for understanding how well project implementation is going. • Short-term outcomes: These are the most directlyrelated outcomes to your project implementation. These are an articulation of how a project makes a difference for those identified as potentially benefiting from it. While in some cases, it might be possible to say that the Resources project “caused” a certain outcome to happen, it is also important to consider how a project might “contribute” to a desired outcome. The systems around community resilience are complex and understanding cause and effect may not be possible in all instances. • Strategies and activities: Strategies and activities are the “doing” of projects and what excites project collaborators the most. These are what set short-term outcomes in motion. As the text above suggests, there are relationships between activities, short-term outcomes, longer-term outcomes, and impacts. We can express these relationships through a simple model, called a logic model, as in the diagram below where the arrows are expressing a hypothesis that the preceding item contributes in some way to the realization of the following item. Logic models illustrate how project collaborators believe a project will work; how resources (funding, human resources, technology, supplies, infrastructure, etc.) will be used to deliver activities that will change conditions (produce outcomes and impacts). We have not included an identification of resources on the Playboard because this is just a tool for jumpstarting your team’s thinking process. You and your team can add this as you refine your project design. Many logic models also include “outputs,” which are measures of your activities such as the number of people served by an activity, the number of events held, the number of solar panels installed or tangible project deliverables like websites, training curriculum, and policy documents. Outputs have been left out of the diagram below to allow teams to focus more directly on the changes they want to promote but are important to track for understanding how well project implementation is going. Activities Short-term outcomes Longer-term outcomes Impacts You can also draw arrows to show relationships between specific activities and short-term outcomes, specific shortterm outcomes and longer-term outcomes, or specific longer-term outcomes and impacts by asking your team to think through a set of if … then questions. If this activity happens, then what outcome is expected; if this outcome is realized, then what outcome might follow. Identifying if/then relationships allows teams to strengthen their understanding of how outcomes may be realized and is a good test of whether this sequence makes sense or not. If project activities have no link to desired outcomes, then the team should reconsider their project design. 57 Community Resilience Project Planning Playboard Identifying outcomes and working through a simple logic model can be an important process at the beginning of a project to build trust among project collaborators, surface expectations, give voice to multiple stakeholder perspectives, solidify the rationale for the project, and build a collective understanding of what it can accomplish in the short term. In person, collaborators can brainstorm outcomes and relationships among outcomes using sticky notes. Virtually, tools like Google Jamboards can serve the same purpose. In the Resources section below there are publications and websites that you can refer to while constructing a logic model. Many funders ask grant applicants to prepare logic models as part of their proposals, so it good to understand what these are and how to construct them. Evaluation As a team implements their project, they should continue Measuring short-term outcomes starts with thinking about how you would know if an outcome occurred or not. These are also called indicators, indications of how well an outcome has been met. For example, indicators of an outcome “Community-based organizations increase their collaboration on heat-mitigation and adaptation” might include: the number of meetings held to foster collaboration and plan joint activities, the number of collaborative projects initiated, the number of organizations collaborating on strategies, and/or a change from organizations only exchanging information to organizations planning and implementing strategies together, depending on the goals of this particular project. to revisit their outcomes and logic model, if they created one. Some unexpected outcomes may emerge during implementation, and these need to be captured and recorded. While it once was common to think of a logic model as static, we now recognize that for a logic model to be useful, it needs to be considered as a living documentation of a project or program. This is particularly important for projects trying to affect change in complex and dynamic systems. The Playboard helps collaborators think about why their project should exist (purpose, identification, value and contribution), who it will benefit, where it is going (outcomes and impacts), and how it is going to get there (collaboration, strategies, and activities). Collaborators also need to measure what progress has been made on attaining the short-term outcomes. This is addressed in the box at the bottom of the Playboard. While these outcomes will be realized over several months, project teams should think through and put in place a short-term outcomes measurement strategy at the beginning of their projects. This allows teams to continuously reflect on their progress and make adjustments during implements that can improve project implementation. After defining indicators, teams need to consider how to measure whether an outcome happened or not, the box in the bottom right-hand corner of the Community Resilience Planning Playboard. This is operationalizing the indicators. Available time and resources will shape outcome measurement, and it is important in the planning stage to think about how to make best use of these. When considering data collection methods, weighing the costs and benefits of specific methods, such as surveys, is important. The Better Evaluation website has a long list of data collection methods. Taking the time to understand what kinds of data and data collection strategies already exist within your community or among your collaborators is also a good first step. 58 Community Resilience Project Planning Playboard While some academic researchers focus on rigorous measurement through complex research designs, this is not the only means of measuring outcomes. There are many ways of collecting outcome and output data, including keeping careful records about project events and activities. Participatory methods of collecting and interpreting data, such as through PhotoVoice, might be more suitable than traditional social science research methods in some community contexts. Outcome measurement involves not only grappling with methods for data collection but also thinking about the context and power dynamics of data collection. “If evidence matters, we must care how it gets made.”2 Equity is a pillar of community resilience, and an equity focus extends to how we define and understand the success of community resilience projects. Resources Community Tool Box: A wide-range of toolkits and resources for working in and with communities, including analyzing problems and goals, developing an intervention, and assessing community needs and resources. W.K. Kellogg Foundation Logic Model Guide Innovation Network Logic Model Workbook Better Evaluation: Methods and Processes to Collect and/or Retrieve Data Why Am I Always Being Researched 2 Chicago Beyond (2018) Why Am I Always Being Researched. Equity Series, Volume One. Chicago: Chicago Beyond. Accessed at: https://chicagobeyond.org/researchequity/ 59 05 Project Readiness Checklist for Collaborative Data-Based Visualization or Modeling What It Is and Why It Matters The Project Readiness Checklist is an inventory for potential collaborators to examine the extent to which they are ready to begin a data-based visualization or modeling project. The primary categories of the checklist interrogate the extent to which a research team or community partner have developed project objectives, assessed data needs and availability, created a team roster, identified an audience or end user of a tool or product, and composed anticipated deliverables for that audience. Taking these steps helps project implementation proceed smoothly. Keep Reading If... If you are ready to begin considering the practical details of project design and planning in a data-based visualization or modeling project, this play is very helpful in identifying what needs to be considered before moving forward into project execution. Definition and Purpose The Project Readiness Checklist is an inventory for prospective collaborators to examine the extent to which they are ready to begin a data-based visualization or modeling project. The primary categories of the checklist interrogate the extent to which a research team or community partner have developed project objectives, assessed data needs and availability, created a team roster, identified an audience or end user of a tool or product, and composed anticipated deliverables for that audience. Taking these steps will ensure that project implementation proceeds smoothly. Time and Timing Plan for roughly thirty minutes to complete the checklist, plus time to compare and discuss results with team members. Applicable Conditions The checklist is a valuable tool for projects which are just beginning, or those which may hit a wall or feel stalled out. The checklist can help reveal weak points or gaps in readiness or preparation which can reinvigorate a new direction of effort or research in a project. 60 Project Readiness Checklist for Collaborative Data - Based Visualization or Modeling Procedure The following checklist is meant to help potential collaborators examine the state of their current projects, as well as orient DT members to project needs and requirements. Checklist items include several questions to consider when evaluating a project’s current status or potential for development. Answer the questions under each main category. Project Define objectives Develop research questions Identify relevant conversations Define scope and level of interactivity Create project roadmap Identify risks Data Team Identify data needed to support research questions PI Determine data availability Visioneer Determine data accessibility Develop data sharing & storage agreement Clean & organize data to support research question Audience Who to include in co-creation process SME(s) Key stakeholders Project Manager Data scientist Developer Mathematical modeler Community relationship manager End users Expert test cases Clarify obligations to funding organization Develop communication plan & presentation strategy for audience Deliverables Create visuals to support conversations around identified research questions Develop facilitation method to guide user experience of data visualization Communicate project to relevant audiences Create post-project development plan Designer Project Define project objectives • What decision is in need of support? • Is there a current policy issue that requires understanding? • • • What do you want to be able to do as a result of completing this project? What is the connection between the project and the funding organization’s mission? What would success look like for this project? Develop research questions • What do you want to know more about? • What are your hypotheses? • What do you want to ask of the data? Identify relevant conversations and literature • Which scholarly, political, or social conversations do you want to join or shape? • How are key concepts already defined or talked about in these conversations? • How do you want to challenge, support, or refine these conversations? Define scope • Which part(s) of the research question do you want to address? • What are key milestones, reports, or deliverables? • What is outside the scope of the current project, or should be saved for future projects? 61 Project Readiness Checklist for Collaborative Data - Based Visualization or Modeling Project (continued) Define level of data interactivity • Is the deliverable a static data visualization, mock-up of a future funded model, dashboard with data filters, or a multi-screen, interactive simulation? • Is the visualization meant to stand alone for public use or does it require direct facilitation for optimal use? Create timeline • What are short-and long-term deadlines for the project? • Will the project be rolled out in multiple stages which build on each other, or have one primary deadline and deliverable? Develop data sharing and storage agreement • Where will the data be stored? • Are there any relevant regulations or IRB constraints on the sharing or storing of the data? Clean and organize data to support research question • Is the data being organized in a way which supports the project objectives? • What are key variables and statistical analyses of interest in regard to the guiding research question? Team Principal Investigator (PI) • Create project roadmap • What research, data, development, design, and community engagement is required to complete the project? Identify risks • What are the biggest threats to project success? • People, resources, time, data? • What are plans to mitigate these threats? Data Identify data needed to support research questions • What data and information is necessary to address topics and decisions of interest? • Where will the data come from? Determine data availability • Does the data need to address the research question exist? If so, where? • If not, how can it be collected? Determine data accessibility • If data exist, are they publicly available or not? If not, how can you gain access? 62 The PI is the primary research lead for the project or has been designated as such on a funded grant proposal. The PI determines the research question and helps coordinate team members’ efforts around executing the project. Subject matter experts (SME) • SMEs can be faculty members, practitioners, or community members who have deep expertise in the subject matter under consideration for the research question. Visioneer • A visioneer contributes complex systems-level thinking to projects and helps the team envision the current and future connections between different conversations, fields of research, and communities of practice. Project manager • Project managers coordinate the efforts of the internal departments—software development, data science, facilitation—with those of the SMEs and PI to meet deadlines and ensure successful execution of project deliverables. Project Readiness Checklist for Collaborative Data - Based Visualization or Modeling Team (continued) Data scientist • Data scientists analyze and interpret data using descriptive and inferential statistics. They may also be responsible for cleaning and organizing data into a format which is appropriate for visualization. Developer • Developers translate data into interactive models and simulations. Mathematical modeler • Modelers create the underlying mathematical logic and equations that tie multiple data sets together, as well as connect multiple kinds of analyses into cohesive visuals. Community relationship manager • Management of relationships with partner organizations, stakeholders, and community collaborators is a key component of model co-creation and eventual use. A community relationship manager acts as a liaison to include community interests and maintain inclusive communication and collaboration throughout the course of the project. Graphic designer • Designers work with developers, community relationship managers, the PI, project manager, and other team members to create aesthetic, user-centered data visualizations. Audience Key stakeholders • Determine key political and community stakeholders to engage in the participatory modeling process, and the extent to which each will be involved. Expert test cases • Once the model is in design stages or operational, who are expert users you can invite to test and assess its usability, function, and identify weaknesses? Clarify funding obligations • Who are key representatives from the funding organization who should be involved in either the participatory modeling process, user testing, or end user assessment of project outcomes? Deliverables Create visualization to support conversations around identified research questions • Identify the extent of interactivity necessary to achieve project goals. Develop facilitation method to guide user experience of data visualization • How will the model be introduced to its intended audience? • Who will lead and facilitate engagements between the model and users? Develop communication plan and presentation strategy for audience • How can you communicate the value of the model to communities of interest? Create post-project development plan • What do future versions or add-ons of the project include? • What was outside the scope of this version but has potential for future development? End users • Identify the eventual end user of the model or data visualization—the general public, a school or department, a particular organization, a community group? 63 Project Readiness Checklist for Collaborative Data - Based Visualization or Modeling Evaluation You have used this play well if you and your team are clear on research objectives, data needs and availability, team roster, audience or end user of a tool or product, and anticipated deliverables for that audience. Related Tools and Plays • Determine Readiness to Collaborate • Managing the Inclusion Process Resources Collaboration Success Wizard (sc.ctsi.org). The Collaboration Success Wizard (CSW) is a 30-minute survey which generates individual and project-level reports diagnosing strengths, potential challenges, and coping strategies for geographically-distributed collaborations, using the latest findings from collaboration science. 64 The Game in Motion Founded in 2016, Achieve60AZ is a community-based initiative to increase educational attainment for all Arizonans, guided by the belief that a more highly trained and educated population will attract more business, boost economic well-being, and increase social gains. As of 2019, the Achieve60AZ Alliance was comprised of 40 local Arizona governments, as well as over 75 community, business, philanthropic, and education organizations. The central goal of this effort is to reach 60 percent adult postsecondary attainment in the state of Arizona by the year 2030. The Arizona Board of Regents (ABOR) is a leading member of the Achieve60AZ Alliance. As the governing board of the state’s three public universities, ABOR seeks to provide leadership and a unifying voice on key higher education issues, as well as influence public policy through advocacy and initiatives founded on evidence-based research (www.azregents.edu). In November 2017, ABOR formed a collaborative partnership with faculty affiliated with the Decision Theater to create a data-driven model and interactive visual display to explore and quantify tradeoffs of different pathways to achieving 60 percent adult postsecondary attainment in Arizona. The research team was charged with examining education and learning as a dynamic, interdependent, nonlinear process and not just as an outcome variable. We needed to understand how stakeholders in education communities conceptualized education’s public value, framed its implications for communities’ health, prioritized different actors (teachers, students, parents, policymakers) within the education system, discussed challenges related to data collection and transparency, and brought to light any other issues regarding Arizona education about which we were unaware. Thus, we engaged in an iterative process of education data analysis, model development, stakeholder engagement, integration of stakeholder feedback, and updating the user interface. 65 Game in Motion The Identify Critical Community Partners The first opportunity to understand existing conversations regarding postsecondary attainment among education stakeholders came in December 2017, when the Arizona Commission for Postsecondary Education hosted a two-day conference titled “Developing Arizona’s Human Capital: Acceleration to 60.” We attended the conference to listen to the ways in which thought leaders conceptualized and described the state of Arizona’s education system, as well as capture perceived barriers and solutions to educational attainment. The conference also provided an important network opportunity for the authors to identify key stakeholders who could provide subject matter expertise regarding the state’s education system and history of existing interventions geared toward increasing postsecondary attainment. Though the first several stakeholder engagements were with attendees of the December conference, we engaged in snowball sampling to elicit additional viewpoints and areas of expertise regarding Arizona’s education system, how the education stakeholders currently conceptualized barriers to, as well as benefits of, attainment, and what data existed to support a model which explored alternative pathways to attainment in Arizona. The Build Stakeholder Network Over the next 14 months, the research team facilitated 32 knowledge convergence workshops with approximately 75 stakeholders from education communities in the ASU Decision Theater Drum. The goals of these engagements were to 1) facilitate stakeholders’ interaction with the evolving model, 2) elicit their feedback on the relevance and usability of the data, display, and analysis, and 3) observe how the model contributed to the depth and complexity of conversations between stakeholders. The iterative process of model development and stakeholder engagement required constant coordination and communication between the research team and the software developers, data scientists, and graphics designers within DT. Thus, in addition to the 32 stakeholder engagements, we met every other week with our ABOR collaborators and DT software developers to clarify expectations and make sure stakeholder interests were reflected in ongoing model development and visualization. The participatory agenda setting process unfolded for over a year between November 2017 and February 2019. 66 Game in Motion The Co-Creating Research Questions Throughout the course of our participatory agenda setting process, two questions dominated stakeholder engagements. The first was, “What is the impact of increased attainment on Arizona’s economy and workforce?” Stakeholders did not want to conceive of education for the sake of education. They wanted to understand the impact of education on economic well-being and the ability of people to find occupations which could provide income stability far into the future. The second question was, “Which 60%?” In the effort to get from 35 to 60 percent educational attainment, did it matter who comprised the 25 percent necessary to reach the state goal? Stakeholders wanted to be able to examine the relative impact of increased attainment for one sub-population or geographic region versus another and explore alternative economic and workforce futures as a result. Figure 3 shows the final version of the Achieve60AZ model, wherein users select sub-populations of interest (Screen 1), examine each sub-population’s descriptive statistics compared to the state average (Screen 2), compare geographic locations of each sub-population (Screen 3), increase attainment for each sub-population and see the relative impact on the state goal (Screen 4), compare economic outcomes (Screen 5) and workforce alignment (Screen 6) as a result of the attainment increase, and explore differences in attainment-occupation pathways for each (Screen 7). Though the immediate end user of the model is ABOR, other educational stakeholders are welcome to engage with the model at DT in a facilitated environment before it is made available to the public online. The Cluster Analysis & Data Visualization We conducted conduct cluster analysis on publicly available 2016 Public Use Microdata Sample (PUMS) data from the US Census Bureau to assess how accurately (or not) the average household is reflected in statistically significant sub-populations, as well as any differentials in educational attainment, median household income, language spoken at home, and a range of other behavioral variables. The cluster analysis assessed in-group similarity and between-group difference between Arizona households according to a set of variables which the data science team deemed to be welldistributed and credible for inclusion in analysis. Four clusters emerged: households with children; households with no children and single heads of house; households with no children which did not speak English at home; and wealthy, English-speaking households with no children. While we acknowledged that there is wide diversity of households within the first cluster, which includes all households with children, this analysis made visible a discrepancy regarding who was designing the Arizona education system and who was using it. 67 Game in Motion For example, additional analysis conducted using the PUMS data revealed that households with children, who are primary consumers of public K-12 educational services, only comprise 36 percent of all Arizona households. According to this data set, only one-third of adults in households with children have a postsecondary degree (an associate’s degree or higher), and the median age of adults in these households is 37 years. This is in stark contrast to the 64 percent of households without children, 36 percent of which have a postsecondary degree, with an adult median age of 56 years. Thus, for every household with children in the K-12 Arizona education system, there are almost twice as many who can advocate and vote for education-related policy and provision that do not have children who will experience the outcome of such policies. When considering that households with children in the K-12 education system are generally younger, less educated, and have lower median annual income than households without children, there is significant potential for the voices of households which may be most affected by educational provision policies to be silenced or marginalized. Across the geographic regions of the state determined by Public Use Microdata Areas (PUMAs), there are biases as well. Southwestern Arizona and Pinal County, two regions with the highest percentages of households with children (73 and 69 percent, respectively), are also among the three regions with lowest postsecondary attainment among adults aged 25-39. On the other end of the spectrum, Tempe and Scottsdale are two of the regions with lowest percentages of households with children (35 and 36, respectively), with over half of all adults aged 25-39 with some postsecondary attainment. Managing the Inclusion Process The primary goal of this project was to create a multifaceted, complex systems level understanding of the impact of postsecondary attainment on the state of Arizona. We did not hold subject matter expertise on education policy and interventions and so elicited this expertise from various community groups and conferences. Community groups were included one at a time and the research team facilitated their engagement with the model, asking questions about their perception of the impact of postsecondary attainment and the background behind previous public policy interventions. Given the sensitive and politicized nature of education in the state of Arizona, as well as the nascent stages of model development, we chose this approach so that each community group could tell its own story without equivocation, interruption, or argumentation from a perceived competing group, as well as feel that they were able to tell their story and see it captured by the research team. Adapted from Hinrichs, M. M., & Johnston, E. W. (2020). The creation of inclusive governance infrastructures through participatory agenda-setting. European Journal of Futures Research, 8(1), 1-15. 68 Glossary CONCEPT augmented governance DEFINITION The idea of augmented governance has three components: (1) using cluster analysis to identify different subpopulations, their perspectives, and their policy preferences, and using that to see differences between representatives’ policy KEY REFERENCE Solís, P., Li, W., et al. 2020. Trustworthy GeoAI for Resilience: A Multi-sector Conceptual Framework and Research Agenda [White Paper]. Arizona State University Knowledge Exchange for Resilience. Available from bit.ly/trustworthygeoai. choices and the policy preferences of those whom they represent, effectively generating an AI-proxy of the policy preferences of different subgroups for representation in decision-making (2) use helper-AI to propose optimal policies when the decision making process has stakeholders with multiple views (3) when different subpopulations view a problem differently and motivated by different values, using recommender-like systems to customize messaging (i.e., public health risk) boundary objects “Boundary objects are objects that are both plastic enough to adapt to local needs and constraints of the several parties employing them, yet robust enough to maintain a common identity across sites” (Star 1989, p. 46). They enable a group of actors with different perspectives and types of information to aggregate their knowledge and solve problems together (Star 1989). Star, S. L. (1989). The structure of illstructured solutions: Boundary objects and heterogeneous distributed problem solving. In Distributed artificial intelligence (pp. 37-54). Morgan Kaufmann. Leigh Star, S. (2010). This is not a boundary object: Reflections on the origin of a concept. Science, Technology, & Human Values, 35(5), 601-617. 69 Glossary CONCEPT DEFINITION KEY REFERENCE boundary organization “Boundary organizations...[meet] three criteria: first, they provide the opportunity and sometimes the incentives for the creation and use of boundary objects and standardized packages; second, they involve the participation of actors from both sides of the boundary, as well as professionals who serve a mediating role; third, they exist at the frontier of the two relatively different social worlds of politics and science, but they have distinct lines of accountability to each” (Guston 2001, p. 400-401) Guston, D. H. (2001). Boundary organizations in environmental policy and science: an introduction. broader impacts The benefits a research project yields to society beyond advancing knowledge. Examples of broader impacts include: “full participation of women, persons with disabilities, and underrepresented minorities in science, technology, engineering, and mathematics (STEM); improved STEM education and educator development at any level; increased public scientific literacy and public engagement with science and technology; improved well-being of individuals in society; development of a diverse, globally competitive STEM workforce; increased partnerships between academia, industry, and others; improved national security; increased economic competitiveness of the U.S.; use of science and technology to inform public policy; and enhanced infrastructure for research and education” (National Science Foundation, n.d.) National Science Foundation. (n.d.) Broader Impacts: Improving Society. National Science Foundation. https://www.nsf.gov/od/oia/ special/broaderimpacts/ built environment “The built environment includes manmade building and infrastructure stocks that constitute the physical, natural, economic, social and cultural capital.” (Hassler & Kohler 2014, p. 120) Hassler, U., & Kohler, N. (2014). Resilience in the built environment. Building Research and Information : the International Journal of Research, Development and Demonstration, 42(2), 119–129. https://doi.org/10.1080/096 13218.2014.873593 70 Glossary CONCEPT community based participatory research community resilience DEFINITION KEY REFERENCE “collaborative approach to research, [CBPR] equitably involves all partners in the research process and recognizes the unique strengths that each brings. CBPR begins with a research topic of importance to the community with the aim of combining knowledge and action for social change to improve community health and eliminate health disparities. (Minkler & Wallerstein, 2003, p. 4)” Nina B. Wallerstein, & Bonnie Duran. (2006). Using Community-Based Participatory Research to Address Health Disparities. Health Promotion Practice, 7(3), 312–323. https://doi.org/10.1177/1524839906289376 “The sustained ability of a community to withstand and recover from adversity (e.g., economic stress, influenza pandemic, manmade or natural disasters)” (Chandra et al 2011, p. 1) Chandra, A., Acosta, J., Stern, S., UscherPines, L., Williams, M., Yeung, D., Garnett, J., and Meredith, L. (2011). Building Community Resilience to Disasters: A Way Forward to Enhance National Health Security. Santa Monica, CA; Arlington, VA; Pittsburgh, PA: RAND Corporation. Retrieved May 6, 2021, from http://www.jstor.org/stable/10.7249/ tr915dhhs Minkler, M., & Wallerstein, N. (Eds.). (2003). Community based participatory research in health. San Francisco: Jossey-Bass. Patel, S., Rogers, M., Amlôt, R., & Rubin, G. (2017). What Do We Mean by “Community Resilience”? A Systematic Literature Review of How It Is Defined in the Literature. PLoS Currents, 9. https://doi.org/10.1371/currents. dis.db775aff25efc5ac4f0660ad9c9f7db2 complex systems Systems of networked, heterogenous parts that evolve and adapt both in response to each other and the external environment. Holland, J. H. (2014). Complexity: A Very Short Introduction (1st ed.). United Kingdom: Oxford University Press. Holland, J. H. (2014). Complexity: A Very Short Introduction (1st ed.). United Kingdom: Oxford University Press. Fraccascia, L., Giannoccaro, I., & Albino, V. (2018). Resilience of Complex Systems: State of the Art and Directions for Future Research. Complexity (New York, N.Y.), 2018, 1–44. https://doi.org/10.1155/2018/3421529 71 Glossary CONCEPT DEFINITION KEY REFERENCE convergence “Convergence research is a means of solving vexing research problems, in particular, complex problems focusing on societal needs. It entails integrating knowledge, methods, and expertise from different disciplines and forming novel frameworks to catalyze scientific discovery and innovation” (National Science Foundation 2020) National Science Foundation. (2020). What is Convergence? https://www.nsf.gov/od/oia/ convergence/index.jsp data liberation What we term as a process “to liberate data” privileges the aim to discover and uncover the value of data that may be hidden within organizations. These data may be undervalued, inaccessible, be unstructured or contain a messy structure, or exist untouched due to time constraints. https://bit.ly/liberatedata endogenous / exogenous Attributes of a system, such as a city, that emerge from within (endogenous) versus those that are external (exogenous) attributes imposed on the system. Petersen, T. (2001). Endogeneity: Methodology. In Smelser, N. J. & Baltes, P. B. (Eds.), International Encyclopedia of the Social and Behavioral Sciences (pp. 4511-4513). Pergamon. https://www.sciencedirect.com/ science/article/pii/B0080430767007786 Lleras, C. (2005). Path Analysis. In K. Kemph-Leonard (ed.), Encyclopedia of Social Measurement (pp. 25-30). Elsevier. https://doi.org/10.1016/B0-12-3693985/00483-7 equity, distributional “[Fair] distribution of goods, services, and opportunities” (Meerow et al. 2019, p. 796) Meerow, S., Pajouhesh, P., & Miller, T. (2019). Social equity in urban resilience planning. Local Environment, 24(9), 793–808. https:// doi.org/10.1080/13549839.2019.1645103 equity, procedural “[Fair] participation in decision-making processes” (Meerow et al. 2019, p. 796) Meerow, S., Pajouhesh, P., & Miller, T. (2019). Social equity in urban resilience planning. Local Environment, 24(9), 793–808. https:// doi.org/10.1080/13549839.2019.1645103 72 Glossary CONCEPT equity, recognitional friction DEFINITION KEY REFERENCE “Recognitional equity entails: (1) acknowledging community members’ different intersecting identities (e.g. race, gender, class, and age), (2) recognizing that these identities are shaped by historical injustices and can shape individual vulnerability to shocks and stresses, ability to access resources, and capacity to participate in decision-making, and (3) fostering respect for different groups.” (Meerow et al. 2019, p. 797) Meerow, S., Pajouhesh, P., & Miller, T. (2019). Social equity in urban resilience planning. Local Environment, 24(9), 793–808. https:// doi.org/10.1080/13549839.2019.1645103 “Friction of distance is a core principle of Geography that states that movement incurs some form of cost, in the form of physical effort, energy, time, and/or the expenditure of other resources, and that these costs are proportional to the distance traveled” (“Friction of distance”, 2021, para. 1) Friction of distance. (2021, June 3). In Wikipedia. https://en.wikipedia.org/wiki/ Friction_of_distance global futures A framework for developing scenarios of possible realities at a later point in time, given different levels of action by local, regional and worldwide actors; especially where the impacts of those scenarios may be experienced as consequences on a planetary scale; and where collective mitigation, adaptation and resilience are prioritized in order to inform current approaches to universal shocks and stresses like climate change. Pieterse, J. N. (2000). Theme Section: Global Futures - Introduction, 7(2), 314-316, doi: 10.1080/096922900346983 information interventions A type of behavioral intervention where information is used to develop awareness and attitudes which promote a target behavior. Information interventions are often used to develop pro-environmental attitudes leading to pro-environmental behavior, such as recycling (Geng et al. 2016) Geng, J., Long, R., & Chen, H. (2016). Impact of information intervention on travel mode choice of urban residents with different goal frames: A controlled trial in Xuzhou, China. Transportation Research. Part A, Policy and Practice, 91, 134–147. https://doi. org/10.1016/j.tra.2016.06.031 73 Glossary CONCEPT DEFINITION KEY REFERENCE knowledge exchange “Knowledge exchange are processes that generate, share, and/or use knowledge through various methods appropriate to the context, purpose, and participants involved. Knowledge exchange includes concepts such as sharing, generation, co-production, co-management, and brokerage of knowledge” (Fazey et al. 2013, p. 19) Fazey, I., Evely, A. C., Reed, M. S., Stringer, L. C., Kruijsen, J., White, P. C., Newsham, A., Lixian, J., Cortazzi, M., Phillipson, J., Blackstock, K., Entwistle, N., Sheate, W., Armstrong, F., Blackmore, C., Fazey, J., Ingram, J., Gregson, J., Lowe... & Trevitt, C. (2013). Knowledge exchange: a review and research agenda for environmental management. Environmental Conservation, 40(1), 19-36. leakage “A specific type of spillover in which an environmental policy indirectly triggers impacts that go against its aims, thus reducing the overall benefit of the intervention” (Bastos Lima et al. 2019. p. 2) ”A specific type of spillover in which an environmental policy indirectly triggers impacts that go against its aims, thus reducing the overall benefit of the intervention” (Bastos Lima et al. 2019. p. 2) Bastos Lima, M. G., Persson, U., & Meyfroidt, P. (2019). Leakage and boosting effects in environmental governance: a framework for analysis. Environmental Research Letters, 14(10), 105006–. https://doi. org/10.1088/1748-9326/ab4551 mobility “The potential for movement and the ability to get from one place to another using one or more modes of transport to meet daily needs” (Eltis 2019) Eltis, A. (2019, May 28). Mobility. Eltis. https:// www.eltis.org/glossary/mobility modularity “A system made up of a collection of modules or functional components that change and evolve to some extent in an autonomous manner. Modules can be components of a subsystem or on another scale, subsystems can function as modules within a larger complex social-ecological system” (Belin et al 2013, p. 3) Beilin, R., Reichelt, N. T., King, B. J., Long, A., & Cam, S. (2013). Transition landscapes and social networks: examining on-ground community resilience and its implications for policy settings in multi-scalar systems. Ecology and Society, 18(2). natural capital “Natural capital is made up of resources and ecosystem services from the natural world” (Magis 2010, p. 406) Magis, K. (2010). Community Resilience: An Indicator of Social Sustainability. Society & Natural Resources, 23(5), 401-416, doi: 10.1080/08941920903305674 74 Cresswell, T. (2006). On the move : mobility in the modern Western world. Routledge. Glossary CONCEPT DEFINITION KEY REFERENCE participatory modeling “Participatory modeling – the involvement of stakeholders in the modeling process – can support various objectives, such as stimulating learning processes or promoting mutual understanding of stakeholders” (Halbe et al. 2020, p. 60) Halbe, J., Holtz, G., & Ruutu, S. (2020). Participatory modeling for transition governance: Linking methods to process phases. Environmental Innovation and Societal Transitions, 35, 60–76. https://doi. org/10.1016/j.eist.2020.01.008 placemaking “Placemaking is the process of creating quality places that people want to live, work, play and learn in” (Wyckoff 2014, p. 2) Wyckoff, M. A. (2014). Definition of placemaking: four different types. Planning & Zoning News, 32(3), 1. postcolonial computing “an alternative sensibility to the process of [technological] design and analysis” which “asserts a series of questions and concerns inspired by the conditions of postcoloniality”, including the “history of global dynamics of power, wealth, economic strength, and political influence [which] shape contemporary cultural encounters” (Irani et al. 2010, p. 1311) Irani, L., Vertesi, J., Dourish, P., Philip, K., & Grinter, R. (2010). Postcolonial computing: a lens on design and development. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 2, 1311–1320. https://doi. org/10.1145/1753326.1753522 redundancy “The extent to which elements, systems, or other units of analysis exist that are substitutable, i.e., capable of satisfying functional requirements in the event of disruption, degradation, or loss of functionality” (Bruneau et al 2013, p. 737) Bruneau, M., Chang, S. E., Eguchi, R. T., Lee, G. C., O’Rourke, T. D., Reinhorn, A. M., Shinozuka, M., Tierney, K., Wallace, W., & Von Winterfeldt, D. (2003). A framework to quantitatively assess and enhance the seismic resilience of communities. Earthquake spectra, 19(4), 733-752. resilience dividend “The difference in the outcomes between the scenario with a resilience approach and without. Beyond quantifying the direct returns to the immediate goal, the model helps highlight the co-benefits—societal as well as financial—of a project that decision-makers may otherwise overlook.” (Rodin 2017) Rodin, J. (2017, Feb 27). Valuing the Resilience Dividend. The Rockefeller Foundation. https://www. rockefellerfoundation.org/blog/valuingresilience-dividend/ 75 Glossary CONCEPT DEFINITION KEY REFERENCE risk “In the fields of disaster risk management and increasingly climate change adaptation, risk is commonly defined as the ‘probability of harmful consequences’ (UNDP, 2010, p. 1), arising from the interaction of exposure, vulnerability, and hazard (IPCC, 2012).” (Mochizuki et al. 2018, p. 364) Mochizuki, J., Keating, A., Liu, W., HochrainerStigler, S., & Mechler, R. (2018). An overdue alignment of risk and resilience? A conceptual contribution to community resilience. Disasters, 42(2), 361–391. https://doi. org/10.1111/disa.12239 safe-to-fail “Built systems designed to lose function in controlled ways; thus, different types of failure consequence are experienced as incurred by prioritized decisions, even when safety thresholds are exceeded by unpredicted risks. Under a safe-to-fail approach, a given system can fall into at least three different states: functioning, limited functioning/ incurred failure, and full failure with chosen consequences.” (Kim et al. 2019, p. 711) Kim, Y., Chester, M. V., Eisenberg, D. A., & Redman, C. L. (2019). The Infrastructure Trolley Problem: Positioning Safe-to-fail Infrastructure for Climate Change Adaptation. Earth’s Future, 7(7), 704–717. https://doi. org/10.1029/2019EF001208 scale Geographic or spatial units at which particular phenomena can be observed or understood. For example, the neighborhood, city, region, nation, or globe could be conceptualized as different spatial scales. Howitt, R. (2000). Nests, webs and constructs: contested concepts of scale in political geography. A companion to political geography. Blackwell, Oxford, 138157. http://carlosreynoso.com.ar/archivos/ territorio/Howitt-Contested-concepts-ofscale.pdf self-regulating “An entity can regulate itself in ways that Rodin, J. (2014). The Resilience Dividend: enable it to deal with anomalous situations and disruptions without extreme malfunction or catastrophic collapse. Cascading disruptions do not result when the entity suffers a severe dysfunction” (Rodin 2014) Being Strong in a World Where Things Go Wrong. United States: PublicAffairs. 76 Glossary CONCEPT shock or stress DEFINITION Shocks are “external short-term deviations from long-term trends that have substantial negative effects on people’s current state of well-being, level of assets, livelihoods, or safety, or their ability to withstand future shocks” (Choularton et al. 2015, p. 15) Stressors are “long-term trends or pressures that undermine the stability of a system and increase vulnerability within it” (Choularton et al. 2015, p. 15) KEY REFERENCE Choularton, R., Frankenberger, T., Kurtz J. & Nelson, S. 2015. Measuring Shocks and Stressors as Part of Resilience Measurement. Resilience Measurement Technical Working Group. Technical Series No. 5. Rome: Food Security Information Network. Available at: https://www.fsinplatform.org/sites/ default/files/paragraphs/documents/FSIN_ TechnicalSeries_5.pdf smart cities “A smart city is a well defined geographical area, in which high technologies such as ICT, logistic, energy production, and so on, cooperate to create benefits for citizens in terms of well being, inclusion and participation, environmental quality, intelligent development; it is governed by a well defined pool of subjects, able to state the rules and policy for the city government and development” (Dameri 2013, p. 2549). It is important to note that the smart city is a contested concept and that this definition is not authoritative. Dameri, R. P. (2013). Searching for smart city definition: a comprehensive proposal. International Journal of computers & technology, 11(5), 2544-2551. social capital “Resources embedded in one’s social networks, resources that can be accessed or mobilized through ties in the networks” (Aldrich & Meyer 2015, p. 256). There are three types of social capital — bonding (strong emotional bonds between family and close friends), bridging (weaker ties between distinct groups), and linking (ties to people in power) — and these are shown to be helpful for community resilience. (Aldrich & Meyer 2015) Aldrich, D., & Meyer, M. (2015). Social Capital and Community Resilience. The American Behavioral Scientist (Beverly Hills), 59(2), 254–269. https://doi. org/10.1177/0002764214550299 Bourdieu, P. (1985). The forms of capital. In J. G. Richardson (Ed.), Handbook of theory and research for the sociology of education (pp. 241-258). New York, NY: Greenwood Bourdieu, P. (1985). The forms of capital. In J. G. Richardson (Ed.), Handbook of theory and research for the sociology of education (pp. 241-258). New York, NY: Greenwood 77 Glossary CONCEPT DEFINITION KEY REFERENCE social cohesion “The ongoing process of developing wellbeing, sense of belonging, and voluntary social participation of the members of society, while developing communities that tolerate and promote a multiplicity of values and cultures, and granting at the same time equal rights and opportunities in society.” (Fonseca et al. 2019, p. 246) Fonseca, X., Lukosch, S., & Brazier, F. (2019). Social cohesion revisited: a new definition and how to characterize it. Innovation (Abingdon, England), 32(2), 231–253. https:// doi.org/10.1080/13511610.2018.1497480 social embeddedness Mutually beneficial partnerships between the university and community Arizona State University. (n.d.). ASU MIssion & Goals. Arizona State University Office of the President. https://president.asu.edu/asumission-goals#design-aspirations social justice “Full participation in society and the balancing of benefits and burdens by all citizens, resulting in equitable living and a just ordering of society. Its attributes included: (1) fairness; (2) equity in the distribution of power, resources, and processes that affect the sufficiency of the social determinants of health; (3) just institutions, systems, structures, policies, and processes; (4) equity in human development, rights, and sustainability; and (5) sufficiency of wellbeing” (Buetter-Schmidt & Lobo 2012, p. 948) Buettner-Schmidt, K., & Lobo, M. L. (2012). Social justice: A concept analysis. Journal of advanced nursing, 68(4), 948-958. social network analysis “Social network analysis (SNA) is the process of investigating social structures through the use of networks and graph theory.[1] It characterizes networked structures in terms of nodes (individual actors, people, or things within the network) and the ties, edges, or links (relationships or interactions) that connect them” (“Social network analysis”, 2021, para. 1) Social network analysis. (2021, June 3). In Wikipedia. https://en.wikipedia.org/wiki/ Social_network_analysis 78 Glossary CONCEPT DEFINITION KEY REFERENCE strong versus weak ties “The strength of a tie is a (probably linear) combination of the amount of time, the emotional intensity, the intimacy (mutual confiding), and the reciprocal services which characterize the tie” (Granovetter 1973, p. 1361) Granovetter, M. S. (1973). The Strength of Weak Ties. The American Journal of Sociology, 78(1), 1360–1380. systems change “An intentional process designed to alter the status quo by shifting the function or structure of an identified system with purposeful interventions...Systems change aims to bring about lasting change by Abercrombie, R., Harries, E., & Wharton, R. (2015, June). Systems change: a guide to what it is and how to do it. NPC. https://www. thinknpc.org/resource-hub/systems-changea-guide-to-what-it-is-and-how-to-do-it/ altering underlying structures and supporting mechanisms which make the system operate in a particular way” (Abercrombie et al. 2015, p. 9) transparency “Legal, political and institutional structures that make information about the intemal characteristics of government and society available to actors both inside and outside of the domestic political system” (Finel and Lord, 1999, p. 316) Foster-Fishman, P., Nowell, B. and Yang, H. (2007) ‘Putting the system back into systems change; a framework for understanding and changing organizational and community systems’. American Journal of Community Psychology (39): pp.197–215. Finel, B., & Lord, K. (1999). The Surprising Logic of Transparency. International Studies Quarterly, 43(2), 315–339. 79 Additional Resources A Manager’s Guide to Resolving Conflict in Collaborative Networks (maxwell.syr.edu). Written by two scholars of public administration and service, Rosemary O’Leary and Lisa Blomgren Bingham, this is an practical, thorough, and overall excellent guide to managing and resolving conflict in collaborative networks. AAAS Communication Toolkit (aaas.org). Excellent public engagement with science builds on a foundation of clear, concise communication. This section provides guidance and tips to improve your communication skills. Action Priority Matrix (www.mindtools.com). These simple diagrams help you choose the activities you should prioritize and the ones you should avoid, if you want to make the most of your time and opportunities. Collaboration Success Wizard (sc.ctsi.org). The Collaboration Success Wizard (CSW) is a 30-minute survey which generates individual and project-level reports diagnosing strengths, potential challenges, and coping strategies for geographically-distributed collaborations, using the latest findings from collaboration science. Contributions of Paulo Freire for a critical data literacy (dataliteracy.eita.org.br). Paulo Freire’s main work - the Popular Education pedagogy - influenced many educators all over the world who believed in education as a way of liberating poor oppressed people. In this paper, we propose adapting elements of the Paulo Freire’s Literacy Method for use in data literacy, i.e., the act of building capacities for working with data. Creating and Maintaining Partnerships (ctb.ku.edu). This toolkit provides guidance for creating a partnership among different organizations to address a common goal. Defining Collaboration and Explicating the Collaborative Process (bcpsqc.ca). This document sets out six empirically-demonstrated features that make for a successful collaboration. The features can be used as evaluative criteria to assess the health of the collaboration as well as design principles for collaborative processes. Ethical Decisionmaking Protocol (worldofwork.io). The foursquare protocol is a four stage process. It helps individuals and leaders make ethical decisions in the workplace. The stages are: gather facts, understand how previous ethical decisions were reached, look for similarities to previous ethical situations and assess self-interest or bias. Find your Elected Decisionmaker Map (myreps.datamade.us). This is a tool for looking up who your elected representatives are across the United States. Based on your address, we can find all the federal, state, county and local officials who represent you in government. 80 Additional Resources Getting Started With Tableau Desktop (help.tableau.com). This section gives you the basics on getting started with building views in Tableau, the Tableau workspace, and Tableau concepts, a visualization and modeling software. Group Decision-Making Practices (uwaterloo.ca/centre-for-teaching-excellence). There are a variety of ways to make decisions as a group; this seven-step decision-making model offers an effective structure for choosing an appropriate course of action for a particular task or project. It can also be an effective method for dealing with a problem or interpersonal conflict that arises within the group. Healthy Food Playbook (foodcommunitybenefit.noharm.org). This resource helps you identify potential community partners if you are undertaking a food initiative, but it may still spark some ideas for partnerships even if your project doesn’t concern food. Involving People Most Affected by the Problem (ctb.ku.edu). Learn how to reach out to and meaningfully engage community members with firsthand experience with the problem or issue at hand. Knowing What You Don’t Know: Choosing the Right Chart to Show Data Distributions to NonExpert Users (dataliteracy.eita.org.br). This is an academic article with guidelines on how to choose the right chart to display data distributions to non-experts. Measuring while you manage: Planning, monitoring, and evaluating knowledge networks (iisd.org). This paper details the strengths and weaknesses of common measurement and evaluation frameworks (SWOT analysis, Results Based Management, Logical Framework Analysis, Outcome Mapping, Appreciative Inquiry and human resource performance assessment) and proposes a simple network evaluation model which incorporates the best of each. This new framework focuses on the key questions of network effectiveness and efficiency. Opening Pathways (partner.openingpathways.org). This web resource has quiz to help researchers assess their readiness to work with patients as well as narratives from humbled researchers who have engaged in research with patients. PQRS Video Lecture (bit.ly). This is a 10-minute video on the PQRS method described in this text. PyGraphistry (github.com). PyGraphistry is a Python library to quickly load, shape, embed, and explore big graphs with the GPU-accelerated Graphistry visual graph analyzer. RStudio Cheatsheets (rstudio.com). The cheatsheets below make it easy to use some of our favorite packages. From time to time, we will add new cheatsheets. If you’d like us to drop you an email when we do, click the button below. Social Network Analysis (betterevaluation.org). This guide is a quick, one page overview of Social Network Analysis (SNA). It offers an outline of the steps involved in using this technique, a summary of some of the benefits of the technique and what it might be used for, some practical tips for using it, and lists further resources. 81 Additional Resources SWOT Analysis (www.mindtools.com). SWOT stands for Strengths, Weaknesses, Opportunities, and Threats, and so a SWOT Analysis is a technique for assessing these four aspects of your project. The Art of Scientific Storytelling: Transform Your Research Manuscript using a Step-by-Step Formula by Rafael E. Luna (amazon.com). How do we get someone to read our publications? This book shows you how to put your Title and Abstract into a story, along with the rest of your manuscript. Upstream: The Quest to Solve Problems Before They Happen by Dan Heath. This book delivers practical solutions for preventing problems rather than reacting to them. Welcome to Shiny (shiny.rstudio.com). Shiny is an R package that makes it easy to build interactive web applications (apps) straight from R. This lesson will get you started building Shiny apps right away. 82