Methodology - Digital Promise

Methodology

Through funding from the Walmart Foundation, Digital Promise sought to understand how data sharing could create a data-driven learning ecosystem for frontline workers that could promote access to personalized learning opportunities and skill building in order to support workers’ economic advancement. To answer this broad-reaching question, we identified three general categories of questions which drove our work:

  • Who are the stakeholders in this field and what data do they collect?
    • Why do they collect the data? How do they manage, use and report the data?
    • What technologies are used in the ecosystem today?
    • How aware are individual workers/learners of the data that is collected about them?
    • What data could be collected that is not currently? What data would help stakeholders provide more effective services to adult learners/frontline workers?
    • What are the motivations and goals of the various stakeholders, and how do they intersect?
  • What does collaboration and data sharing currently look like in this ecosystem?
    • How do stakeholders interact and/or collaborate with one another, with workers, and with relevant local, state, and federal agencies?
    • What data do they share and for what purposes?
    • What data would make those collaborations more beneficial?
  • What are the perceived benefits and barriers to sharing data?
    • What pain points could be solved by sharing data across stakeholders, for the stakeholder and for the worker/learner?
    • What barriers exist in the current ecosystem that make it challenging to collect, analyze, and share data? What barriers make it difficult to collaborate across the various stakeholders? How do these barriers vary by stakeholder?
    • What motivates those participating in data sharing to do so? What lessons have they learned to make this work more manageable? What perceived benefits could be shared with others to encourage participation in data sharing?

Between May and October, 2018, the Digital Promise research team conducted 43 interviews and focus groups with 44 individuals across the multitude of sectors that interact with frontline workers, including government agents, education and training providers, technology vendors, workforce development boards, and employers. Interviewees were primarily identified by our advisory board, though, in some instances, interviewees referred us to others in the frontline worker ecosystem who could contribute to the project. Interviews were typically 60 minutes in length and most commonly conducted using Zoom Conference, though interviews for the Lake Tahoe Case Study were conducted in person.

After conducting an extensive literature review in April 2018, researchers developed three interview protocols tailored to the three major stakeholder categories: providers, employers, and those speaking from a systems level perspective, such as staff at state governments or foundations. Each protocol was custom tailored to the specificity of the interviewee; for example, while there was a standard protocol for those we spoke to from Human Resource departments, the company size, breadth of and history with employee training, and current efforts drove the questions emphasized in the interviews. We asked each person we interviewed to describe their current organization’s use of data, their perceived benefits and challenges to data sharing, and recommendations to improve the system to be data-driven and more personalized for individual frontline workers/learners.

Given the exploratory nature of the study, we conducted a thematic analysis of the qualitative data. While many qualitative analysis methodologies are tied to particular epistemological or theoretical perspectives, thematic analysis is a method that enables reliable, yet flexible interpretation of the data, thus allowing us to more openly learn from the insights of those interviewed (Braun & Clarke, 2006; Clarke & Braun, 2013; Maguire & Delahunt, 2017). As qualitative researchers Braun & Clarke (2006) explain, thematic analysis is intended to identify themes, or patterns, in the data that are important to address the research question, which requires analyzing beyond the themes of the research questions in order to effectively interpret and make sense of the data.

Our analysis followed Braun & Clarke’s (2006) six-phase framework, beginning with becoming familiar with the data in order to generate our initial codes. Each interview was coded by two researchers at Digital Promise for reliability. Interviews were transcribed through Rev.com. We analyzed interview transcripts using Dedoose, a cross-platform software package for analyzing qualitative data that allows researchers to code text, record memos, and analyze emergent themes.

After each interview or focus group transcript was coded by two researchers, codes were examined to search for themes (Braun & Clarke, 2006). Many of the initial codes lent themselves to convergent themes, such as perceived barriers to effective data use in the current ecosystem, perceived benefits to data use and sharing, and levers for change. Themes were largely descriptive of the patterns that emerged from the interviews; interestingly, we learned that the current ecosystem rarely, if ever, collects outcome data and the infrastructure typically blocks effective analysis of data, showing us that data interoperability is leaps ahead of where this conversation needs to begin. All codes and excerpts fit into one or more of the themes:

Theme Coding Analysis

* applied when a code fits into multiple themes

Theme: Interviewee Background information

Incorporated Codes

  • Frontline workers
  • Role & Program description
  • Services

Theme: Today’s Landscape

Incorporated Codes

  • Data Reported*
  • Data Collection Initiative
  • Data Collection Storage/System
  • Data Collection Process
  • Data Ownership*
  • Data Collected*
  • Data Collected: Skills progression/Learner Analytics
  • Data Collected: Learner Information
  • Data Collected: Employment Status & History
  • Data Collected: Administrative Data
  • Data Collected: Education Records & History
  • Data Collected: Wellness
  • Data Collected: Life Support
  • Data Collected: Outcomes*
  • Data Collected: Other
  • State Longitudinal Data Systems
  • Data Sharing
  • Data Sharing: Frequency
  • Data Sharing: Data Type
  • Data Sharing: Driver
  • Data Sharing: Processes
  • Technology
  • Technology: Instructional Instrument
  • Technology: Data*
  • Technology: Management & Tracking*
  • Technology: Connecting Services*
  • Technology: Other
  • Data Interoperability*
  • PK12 Comparison

Theme: Perceived Barriers to Effective Data Use in Current System

Incorporated Codes

  • Data Ownership*
  • Data Collected*
  • Data Collected: Outcomes*
  • Desired Data*
  • Challenges: Lack of Resources*
  • Challenges: Data Collection, Use & Tracking
  • Challenges: Funding & Sustainability*
  • Challenges: Other*

Theme: Perceived Barriers to Collaboration

Incorporated Codes

  • Challenges: Current System*
  • Challenges: Priorities
  • Challenges: Unclear Understanding
  • Challenges: Data Sharing & Goals
  • Challenges: Silos & Fragmentation*
  • Challenges: Funding & Sustainability*
  • Challenges: Logistics & Planning*
  • Collaboration*

Theme: Perceived Barriers to Data Sharing

Incorporated Codes

  • Data Reported*
  • Data Ownership*
  • Challenges: Current System*
  • Challenges: Lack of Resources*
  • Challenges: Data Sharing & Goals*
  • Challenges: Silos & Fragmentation*
  • Challenges: Funding & Sustainability*
  • Challenges: Logistics & Planning*
  • Challenges: Policies & Regulations
  • Challenges: Leadership & Data Ownership
  • Challenges: Other*

Theme: Motivations to Use and Share Data

Incorporated Codes

  • Why Collect Data
  • Collect Data: Impact & Improvement
  • Collect Data: Requirements
  • Collect Data: Other
  • Desired Data*
  • Collaboration*

Theme: Perceived Benefits to Data Use and Sharing

Incorporated Codes

  • Data Sharing: Success Stories
  • Benefits of Data Use, Data Sharing & Data Interoperability
  • Technology: Data*
  • Technology: Transparency
  • Technology: Management & Tracking*
  • Technology: Connecting Services*
  • Ideal System
  • Data Interoperability*
  • Recommendations & Best Practices
  • Recommendations & Best Practices: Streamline Collaboration
  • Recommendations & Best Practices: Learner Empowerment
  • Recommendations & Best Practices: Data Interoperability & Sharing*
  • Recommendations & Best Practices: Other

Theme: Levers for Change

Incorporated Codes

  • Recommendations & Best Practices: Funding
  • Recommendations & Best Practices: Identify Gaps
  • Recommendations & Best Practices: Data Interoperability & Sharing*
  • Recommendations & Best Practices: Awareness & Excitement

Next, researchers reviewed the qualitative data to make sense of the themes. We read the data associated with each theme to ensure that it had been appropriately interpreted and used this structure to define our themes and understand the story the qualitative data told (Braun & Clarke, 2006). Below is a thematic map demonstrating the relationship and implications of each theme (see Figure 1).

Diagram showing that you can bypass barriers to data sharing Figure 1. Thematic Map

The research team used this thematic map to develop the report. First, the report describes the need for and benefits of data interoperability in the frontline worker space. It then lays the foundation of today’s landscape and describes the perceived challenges we heard consistently throughout the interviews that make it difficult to achieve data interoperability. The report concludes with recommendations based on the levers of change described most consistently throughout the interviews.

Sign Up For Updates! Sign Up For Updates

Sign up for updates!

×