Group Efforts, Part Six: Meet IDC’s Indicator 4, 9, and 10 Data Quality Peer Group

00:00:01.52 >> You're listening to "A Date with Data" with your host, Amy Bitterman.

00:00:07.34 >> Hey, it's Amy, and I'm so excited to be hosting "A Date with Data." I'll be chatting with state and district special education staff who, just like you, are dealing with IDEA data every day.

00:00:19.50 >> "A Date with Data" is brought to you by the IDEA Data Center.

00:00:24.49 >> Hello. This is Amy with the IDEA Data Center, and you're listening to "A Date with Data," a show for those who love IDEA data. As part of a series of episodes about IDC's Data Quality Peer Groups, today we are featuring the Indicator 4, 9 and 10 Peer Group. These groups are facilitated by IDC TA providers to bring state role groups together to discuss and collaborate on the data quality issues and needs of greatest importance across the states. To tell us about this group, I am joined by the group's facilitators, Rachel Wilkinson and Miki Imura. Welcome and thank you both so much for being on.

00:01:05.37 >> Thank you so much for having us.

00:01:07.59 >> Of course. All right. To get us started, Miki, can you tell us a little bit about the Peer Group and generally who participates?

00:01:16.78 >> Sure. So we focus on all different components related to Indicators 4, 9 and 10, from the methodologies to monitoring activities to the correction of noncompliance. So it's a place where the states get together to talk about timely questions, and then it also intended to be collaborative and an opportunity for us to learn from each other. State participants pretty actively participate and share their experiences, strategies, tools, anything else related to these three indicators. And who participates?

00:01:59.35 >> Yeah.

00:02:00.55 >> So we have three general bucket of people who we target, kind of. We think that SEA staff who are responsible for compiling and calculating the Indicator 4, 9 and 10 data would benefit from this peer group. So this is Part B Data Managers, SPP/APR coordinators, and some other type of data analyst positions. Also SEA staff responsible for conducting monitoring activities related to Indicators 4, 9 and 10 are also welcomed, so indicator leads, SPP/APR coordinators, and other monitoring staff, and SEA staff responsible for reporting and writing up Indicators 4, 9 and 10 for the SPP/APR submission are also welcomed. So these roles can be indicator leads, SPP/APR coordinators. What we want really is a collaboration between these different roles because so many different roles touch Indicators 4, 9 and 10. So we want to create an environment where not just data managers or not just monitors but all people who touch Indicators 4, 9 and 10 can get together and discuss these indicators.

00:03:23.52 >> Yes. So from ... You typically have probably multiple people sometimes from each state. Is that true?  

00:03:29.45 >> Oh, definitely.

00:03:30.52 >> Yeah.

00:03:30.65 >> And we love it when we have the monitor and the data manager and the SPP/APR coordinator from one state. It just creates such good conversations.

00:03:42.09 >> Yes, absolutely, and that chance to hear what each other might be doing or responsible for that they weren't maybe aware of or didn't know these other aspects of the indicators and get an opportunity to listen and collaborate, like you said.

00:03:57.27 >> Yeah, absolutely. And also, even if a state brings ... one person from a state participates, they get to hear from another role from another state.

00:04:09.15 >> Right.

00:04:09.72 >> And so that's always very positive.

00:04:12.10 >> Yeah, that's a great benefit. Thanks, Miki.

00:04:15.50 >> Mm-hmm.

00:04:16.25 >> Rachel, can you describe what the structure or the format of these calls are like?

00:04:21.90 >> Sure. So we try to keep our sessions pretty interactive because what we want is obviously the audience to engage with the content, knowing that our group is the youngest group. So we've only had, I think, four calls thus far this year. So right now we're in more of a laying-the-foundation stage. So it might be more heavily us talking through content, but what we try to do is break it up with quizzes or encouraging people to respond to questions or their own personal experiences in the chat. And then we've had a lot of people who are willing to unmute and talk because obviously we know that peer groups are important because it gives people an opportunity to talk to their peers. Miki and I can tell people what the requirements are, and we maybe be able to give some suggestions based on the work we've done with other states, but it's really states learning from each other that's the most important because they can relate to the structures and confines and things that are feasible and not feasible in a state agency.  

00:05:29.40 >> Mm-hmm. 

00:05:30.29 >> So now what we're going to try to start doing with that base set after our first few calls is really scaffold up where the staff from different states can actually share the work they're doing. So we want people to share about things like how they've engaged stakeholders which is a really challenging thing to do with these indicators because they're so nuanced. So we want the opportunity for people to hear from their peers, so that's sort of been the structure and format we've had thus far and where we want to go.

00:05:59.68 >> Great. And I know also from being involved in these groups over the years, too, the sharing of the resources that each state has developed and worked on is something else that a lot of states I know get out of being part of these groups. So is that ... I'm assuming that's something that will be part of and has been as well.  

00:06:20.45 >> Yes, absolutely, and I think one of the key areas is going to be those reviews to determine noncompliance.

00:06:28.23 >> Mm-hmm.

00:06:28.49 >> So Indicators 4, 9 and 10 give you the data side of things that Miki mentioned, but then they also have this general supervision, verifying that there's noncompliance or not noncompliance, and I think that piece is where it gets really tricky for a lot of people. So we're hopeful and thus far have actually had one state share their resources. They actually presented with us at the Interactive Institutes as well about how they conduct those reviews after the data. So I'm hopeful that we'll be able to share more of those resources and have states share their resources as well.

00:07:08.24 >> Great. Miki, what are some of the topics that you all cover in your group?

00:07:15.74 >> Yeah, Rachel touched on it a little bit, too, but ... So we are a quite new group. We started in February 2024, so it has ... It's been less than 6 months so far. Okay, so the topics we've covered so far, oh, so the first session, February 2024, this is where we introduced the peer group. We did the indicator overview, and we also did a participant survey to explore what the participants are trying to get at from this peer group.

00:07:50.52 >> Mm-hmm.  

00:07:51.17 >> And so we asked a bunch of questions and took a poll of, which indicators are the highest priority for you? Which components of indicators are priorities for you, like data monitoring, reporting information? And then what things would you like to focus on over the next few months?

00:08:11.84 >> Mm-hmm.

00:08:13.37 >> And then we learned that 70 percent of participants, Indicator 4 was the priority to their states.  

00:08:23.31 >> Hmm, interesting.

00:08:24.28 >> And everyone but one said the revising methodologies and deeper dives into indicators and their requirements would be helpful. So this kind of sparked our conversation about, "Okay, we need to dive into Indicator 4 now." And so, let's see, so that was February. And so March, we did a really deeper dive on the Indicator 4 methodology, and it was really fun. Rachel and I thought about the indicator requirements in terms of a building. So we have the outside walls, drywalls and furniture. So we thought about outside walls as things that we don't have control over. So these are things that are set by the laws and regulations. For example, for Indicator 4, all states have to choose between option one or option two.

00:09:19.68 >> Mm-hmm. They don't have an option ...

00:09:20.98 >> And so these are like ... Hmm?

00:09:22.13 >> No option three. It's only those two.

00:09:24.42 >> So, yeah, no option three, no making up your own options.

00:09:27.60 >> Right.

00:09:27.81 >> And so these are the ... Something that states must do and choose from. So that's the outside wall, whereas the drywalls are things that it's really hard to change but states have some control over. So these are things like the calculation methods for Indicators 9 and 10, or maybe cell size, minimum cell size, minimum n-size. These are things that states should seek stakeholder input, so it's not as easy to change but it's doable, and states have control over it, right?

00:10:09.61 >> Yep.

00:10:10.02 >> And the furniture pieces are things that there are not so many do this, do that from the regulations, so states can pretty much ... I don't know. Guide their own practices so things are ... These things are like how to conduct the Policy, Procedure, Practice Review. So, yeah, so for Indicators 9 and 10, it may be the ... determining whether the disproportionate representation is the result of inappropriate identification can be a furniture piece because states can do that, make that determination. And they can't make it any way they want, but they have the leeway in how to do this.

00:10:56.05 >> Mm-hmm.

00:10:56.87 >> "What is the outside wall? What's the drywall? What's the furniture?" kind of conversations for Indicators 4, 9 and 10. And then we actually were very excited for July Data Quality Peer Group, that we're going to have one state present on how they have done the stakeholder input session for their Indicator 4 methodology changes, and I think that many states will benefit from hearing how this state have conducted the stakeholder input session.  

00:11:35.68 >> Mm-hmm.

00:11:36.21 >> So, and I think that this ... Seeking stakeholder input for Indicator 4 methodology changes is something that's on many states' minds. So ...

00:11:50.42 >> Yes.

00:11:50.98 >> ... hearing about it is a good idea.

00:11:53.29 >> Yeah, that's great. I really like that analogy of the outer walls and the inner walls and the furniture.  

00:12:00.23 >> Yeah, that's cute, isn't it?

00:12:01.48 >> It is. It's an interesting way to think about it, and it kind of sticks in your mind. So, Rachel, what are some of the common themes that you all are hearing from states during your calls? 

00:12:11.87 >> Well, Miki touched on it a little bit, that I think was great to mention, that Indicator 4's methodology has been a lot of states' focus, especially because this is something that OSEP has issued comments on during clarification periods about the methodology that's being used, whether enough LEAs are being considered in analysis or excluded due to minimum cell and n-sizes, whether or not the thresholds that are being used to determine significant discrepancy are appropriate or high or low. So this is something that's certainly been a focus of OSEP and then, in turn, has been a focus of states. So those are some ... That would be one of the major themes that we have had on our calls has been reconciling that type of feedback and how to design a methodology for 4 that sort of meets OSEP's requirements but then is also informed by stakeholders which really gets to Miki's point there as well about, how do you intentionally communicate the information about these super complex ideas that are meaningful to the stakeholders who may be parents of students with disabilities, individuals with disabilities, community leaders who are not familiar with risk ratios versus rate ratios versus n and cell sizes and thresholds and all of that stuff? Because ultimately for them, it's much more a student-centric focus, so having that information about ways to interact with these stakeholders and get meaningful feedback will also be incredibly helpful for making changes to Indicator 4 methodology. So that's certainly something as a theme we have heard from some of the states who have joined our calls, is the need to better communicate it and suggestions on how to do that.

00:14:09.21 >> Mm-hmm.

00:14:10.75 >> And then I think another topic that, of course, is very popular because, to Miki's point, it's the furniture which means it's sort of vague, sort of undefined but also required, and so it's challenging, is really digging into whether there are noncompliant practices in place that are resulting in either a disproportionate representations, inappropriate identification or noncompliance as a result of different behavioral components and regulations and requirements for Indicator 4. So those are some of the biggest themes we've had, is how are people actually conducting these reviews? If they're using Policies, Procedures, and Practices Reviews for Indicators 4 as well as 9 and 10, which they can, what does that actually look like? Is that something that the state does on their side of things? Is that's something that's informed by a self-assessment that districts submit? How is the self-assessment being reviewed? What is noncompliance in these reviews? Is it one question that indicates noncompliance out of 50? Does that automatically mean you have noncompliance in an LEA? What does that threshold look like? So that's also some of the themes that we've heard which ties a little bit into then the corrections of noncompliance process. Those are a bit more straightforward for Indicator, say, 11, where the finding of noncompliance at the student record level is that they didn't have their evaluation completed within the set timeline for the state or the federal timeline, depending on which they use. And then the systemic correction of noncompliance at the regulatory requirement level is pulling new evaluations that were done after strategies were implemented. But 4, 9 and 10 is very different. You may not even have student-level data that you're looking at in your analysis because that's not prescribed anywhere. So what that review process looks like to achieve verification of correction of noncompliance looks very different and can be challenging. So I would say those are kind of the major themes we've had thus far in our initial meetings.

00:16:24.77 >> Wow. Well, it sounds like for not being around very long, you all have really tackled some very heavy and complicated topics. So that's exciting. All right. Miki, what are tips or recommendations related to Indicators 4, 9 and 10 data that you can share that other states might benefit from knowing about?

00:16:46.32 >> Well, a thing I'd like to say is that every state is unique, and so if your state has specific needs, then you should just articulate how you're meeting your state's specific needs. And there are many parameters to Indicators 4, 9 and 10, so some more in states' control, others not in states' control at all. So I think one tip is that we should use what's under states' control wisely and try to serve the needs of the states' constituents, especially students with disabilities. Is that good enough?

00:17:30.13 >> Yep.

00:17:30.53 >> That was great.

00:17:31.71 >> Anything else, Rachel?

00:17:34.74 >> No, I think that's one of the best tips to give is, ultimately you, as a state, need to be aware of your own context and hear from what you're getting from the field and from your stakeholders to actually build the system that you feel is appropriate. And if it isn't necessarily going to meet all the criteria, or it's still resulting with you getting feedback from OSEP during clarifications about excluding too many LEAs or exceeding a certain threshold, as long as you have a concrete process in place and can clearly articulate to OSEP and the public why you're doing what you're doing, and it's sound and rational and based in fact, I think that is certainly something to commend and not avoid doing just to sort of avoid any comments during clarification. So we're urging states to be really mindful of that and to do what's right for them as long as it still operates within the requirements of the law.

00:18:40.16 >> Okay. Great. Well, thank you, both, so much for sharing all of this great information and letting us know what's going on with this peer group. And if folks are interested in learning more about this peer group or other peer groups or want to join this peer group, please reach out to your IDC State Liaison, or you can send an email to ideadata@westat.com. And, again, thank you both for being on.

00:19:07.80 >> Yeah, thank you for having us.

00:19:09.95 >> Thanks, Amy.  00:19:12.21 >> To access podcast resources, submit questions related to today's episode, or if you have ideas for future topics, we'd love to hear from you. The links are in the episode content, or connect with us via the podcast page on the IDC website at ideadata.org