Transcript

00:00:01.52
>> You're listening to "A Date With Data" with your host Amy Bitterman.

00:00:07.34
>> Hey, it's Amy, and I'm so excited to be hosting "A Date With Data." I'll be chatting with state and district special education staff who, just like you, are dealing with IDEA data every day.

00:00:19.50
>> "A Date With Data" is brought to you by the IDEA Data Center.

00:00:24.70
>> Welcome to "A Date With Data." On this episode, I am joined by Heather Dunphy, who is the lead education programs specialist from the Arizona Department of Education, and we also have with us one of my wonderful colleagues from IDC, Tamara Nimkoff. Heather is going to be talking to us about how Arizona has been conducting their nonresponse bias analysis that all states are required to complete for SPP/APR Indicators 8 and 14, and Tamara is also going to be here to highlight an IDC tool and talk about some other support that IDC can provide states related to the nonresponse bias analysis so welcome to both of you.

00:01:08.06
>> Thank you.

00:01:09.19
>> Thank you.

00:01:10.50
>> So first off, I was hoping if each of you could just introduce yourselves briefly, say a little bit about your role and what you do. Heather, do you want to go first?

00:01:20.13
>> Sure, my name is Heather Dunphy, and I'm a lead education programs specialist at the Department of Education in Arizona. I work a lot with significant disproportionality and LEA determinations, and I also coordinate most of our federal submissions, including the state performance plan annual performance report, and I'm really happy to be here today.

00:01:42.19
>> Great, happy to have you. Tamara, do you want to talk a little about what you do on IDC?

00:01:47.26
>> Sure, thanks, Amy. So I've been with IDC as a state liaison for mostly a [Indistinct] specialist for several years now. My work has been really focused around the analysis and use of data, previously on the state systemic improvement plans, on using data on structured data meeting and over the recent years more focused on supporting states around their data collections or sampling as well as in the areas of representativeness and nonresponse bias, on the topic of today's chat.

00:02:26.75
>> Great, thank you both. So, Heather, can you start us off by kind of walking us through Arizona's data-quality journey related to nonresponse bias analysis? What does that look like?

00:02:39.97
>> Sure, so I began with the agency about 2 1/2 years ago. The FFY2020 SPP/APR was the first federal document that I was responsible for coordinating, and that was the first year that the words nonresponse bias appeared in the APR, and the question in the APR, it asked, describe the analysis of the response rate including any nonresponse bias that was identified. So I was familiar with how to analyze response rates, but I was unclear what exactly nonresponse bias was and certainly how we were going to analyze it.

>> So in 2021 we did our best to analyze the nonresponse bias to the extent that we could, so for Indicator 8, for example, parent involvement survey, what we did was, we divided our survey window into three periods: the beginning period, the middle and the end, and the idea was that the responses might differ from people who answered this survey early compared to those who answered the survey late, and then we examined those responses that came in from parents at the end of the data-collection period as a proxy for nonresponders.

>> And then we then compared those responders to the ones that came in during the beginning and the middle of the data collection period, so this method gave us some insight into whether or not the results might be biased.

>> And then the other strategy we used at that time for Indicator 8, we were looking at our responses by subgroup and to see if they were representative in respect to certain demographic areas such as race and ethnicity, and then we looked at the rate of agreeableness with Indicator 8 by race and ethnicity, and we were just kind of visually trying to see if there was any nonresponse bias.

>> If we received more survey responses from one particular race, ethnicity, and their level of agreement was different than the others, then there might be nonresponse bias. So this provided a good estimate for measuring nonresponse bias at that time. Those are kind of the tools that we had at that time.

00:04:58.87
>> Gotcha, okay, what about for 14? Were you doing something similar??

00:05:02.10
>> Yes, we were doing the same for 14 that we were doing for, or similar, to Indicator 8, yes.

00:05:09.91
>> Great, so you had kind of your start and had a sense of what kind of makes sense. And then what led you to engage with IDC in terms of supporting you around the nonresponse bias analysis?

00:05:40.62
>> Mm-hmm, right. Well, yes, in the spring of 2023, it was last spring, I had heard about a new tool for analyzing nonresponse bias that was in the testing stage, and they were looking for states to try it out.

00:05:57.72
>> I had heard that the tool was built to assist states in addressing the requirements related to response rates, representativeness and nonresponse bias and to kind of ease that burden of analyzing survey data for Indicators 8 and 14, and anything that's going to ease the burden of any work, I'm all for it.

00:06:19.85
>> So myself and one of our Indicator 14 specialists in April went to Rockville, Maryland, for 2 full days of learning about the NRBA App.

00:06:43.00
>> Oh, absolutely, yeah, at the workshop it was Tamara and Ben. They walked us through the various ways to use the app. There are several tests that we can run. They showed us how to set up our data set in the appropriate columns. It needs to be set up in a certain way, and as soon as you have your data set up in a certain way, really the app does most of the work...

00:08:08.48
>> Yeah, absolutely... The impetus of the tool was really aligned with our direct support of states' capacity to meet those SPP/APR data quality requirements for Indicators 8 and 14...

00:11:45.35
>> When users upload... their data into the app for a session... no data are actually passing across the Web, so their data remains secure within their local computer.

00:12:06.17
>> Absolutely... it guides the user through setting up the session... and then they can choose from a whole series of analysis options... response rate, representativeness and nonresponse bias...

00:13:37.16
>> And the tool is also powerful enough that it gives the user options for looking at... statistical adjustments to reduce nonresponse bias...

00:14:22.64
>> That's right... All of the analyses... are done through these preprogrammed analyses that are part of the application...

00:18:43.18
>> So, Heather, tell us about your experience using the tool...

00:19:00.24
>> ...when I returned to Arizona, I tried using the tool on my own... I reached out to Tamara... She helped me understand how to interpret those results. She made sure that my data set was organized correctly...

00:21:08.25
>> ...That gave us a lot of confidence that we didn't have nonresponse bias in respect to race and ethnicity for Indicator 8... and for Indicator 14... instead of 19 percent, it would be lower... about 17 percent.

00:22:48.91
>> ...from what I've seen it is really, really neat and saves a lot of work.

00:23:17.68
>> ...one thing that did help me to understand the tool better is, I made a smaller data set of fake data...

00:24:28.87
>> ...reach out to IDC... make sure that your data set is set up correctly... and... help interpreting them...

00:25:09.45
>> ...information about it... is available on the IDC website along with those supporting documents... contact your IDC state liaison...

00:27:22.56
>> ...thank you both so much... so appreciative that you came on and have shared your story...

00:27:44.71
>> To access podcast resources, submit questions related to today's episode... connect with us via the podcast page on the IDC website at ideadata.org.